The written history of Brežice Castle dates back to the mid-13th century, when the castrum Rain first appears in written sources, but it was undoubtedly built earlier. The original castle building was most likely a wooden fortress that served administrative and defensive purposes for the Salzburg Archbishopric, which had owned the territory in the triangle between the Sava and Sotla rivers since the 11th century.

The original castle fortress housed an armed garrison, and the court and mint were located here. Towards the end of the 12th century, Brežice became the administrative and economic center of the Salzburg estate in Posavje, which was managed by the ministerial lords of Brežice. The castle, built on a small hill above the bank of the riverbed, was named after its location Rain. The Slovenian name, which was also adopted by the urban settlement that developed north of the castle on the river bank – on the small hill – Brežice, has the same meaning. In 1479, the Hungarians, led by Matthias Corvinus, better known as King Matthias, occupied part of today's Posavje with a town and castle.

After 12 years of Hungarian rule, the Habsburgs acquired this area through a peace treaty and the town of Brežice became a provincial princely town. In the early 16th century, when the then Slovenian territory was covered by a bloody all-Slovenian peasant revolt in 1515, both the castle and the town partially burned down. Around 9,000 rebellious peasants gathered near Brežice, who captured the knight Marko and his soldiers in the castle and killed him along with all the nobility. The repair of the castle dragged on throughout the years when the Turks frequently invaded this territory until 1528, when it burned down again.

At that time, the Habsburgs began to build a defensive belt against the Turks, the so-called Military Border, where the heavily fortified fortress was to serve as a rear post. On 22 January 1529, the German Emperor Ferdinand approved the construction of a new Renaissance fortress along with the strengthening and expansion of the city walls. The castle fortress was built by Italian architects, Julius Dispatio from Merano near Roveretto (South Tyrol), and in 1554/55 the famous Renaissance architects, brothers Andrej and Domenico del Allio, are mentioned as architects. Archival sources say that between 1530 and 1550 a perimeter fortress with four corner towers and connecting walls was built, between 1567 and 1579 the eastern and western castle wings, and between 1586 and 1590 or 1601 the northern castle wing and the arcaded corridors with Tuscan columns reaching into the floors of the western wing.

Above the main Renaissance portal is the coat of arms of Franc Gall v. Gallenstein with a unicorn and the year 1601, marking the end of the construction that had begun long before. Originally, the castle was protected by a moat, over which a drawbridge led to the castle gate. In 1573, it was heavily fortified, so that Croatian-Slovenian peasant rebels led by Ilija Gregorič did not capture it. In the 17th century, the castle passed from the Gall family into the hands of the Frankopan family. After the death of Countess Julijana Frankopan, her heirs sold it in 1694 to Count Ignac Maria Attems, who transformed the former fortress into a residential mansion. The largest scale of the reconstruction and artistic renovation in the spirit of the Baroque was carried out in Brežice Castle between 1694 and 1732. First, they rebuilt the eastern wing and arranged the two-story Knight's Hall, which was painted by F. K. Remb in 1702–3, and connected the western and eastern wings with a terrace. Around 1720, they rearranged the access to the floors of the western wing with a representative two-flight staircase and commissioned the painting of the staircase and chapel, which were painted by J. C. Waginger and F. I. Flurer.