Walls adjacent to Hacı Bayram Mosque in Ulus are a remnant of Temple of Augustus. Another temple built in name of moon god Men and fertility goddess Kybele was located in here in the 2nd century B.C. Temple of Augustus was built in name of Roman Emperor Augustus by King Pulamenes, son of last Galatia emperor Amintos in years between 25 and 20 B.C. The reason it survived in a good condition today is the result of being used in each period. It was converted into a church by Christians, then it was converted into a mosque. It was used as a madrasah after the construction of Hacı Bayram Mosque.
One of the interesting features of the temple is the Latin and Greek inscriptions describing the works Augustus did. These occupations Augustus wrote himself were hanged in his tomb after his death; the ones in the Temple of Augustus are the copies of these.
The mosque next to the temple was built in name of famous sufi and poet Hacı Bayram-ı Veli (1352-1429), his mausoleum is adjacent to the south wall. The mosque, was built entirely in brick in years between 1427 and 1428.