Hrastovec Castle is located in the settlement of Hrastovec in the Municipality of Lenart. It was named after the tree species, oak, as it was dominant in the surrounding forests in the past. The settlement of Hrastovec was first mentioned in written sources in 1265. The mention of Hrastovec can be found in the land register of the provincial prince. It shows that the provincial prince received one sow per year from each village belonging to the Maribor provincial court, to which the settlement of Hrastovec also belonged. In the same year (1265), Hrastovec Castle was first mentioned, which consisted of two towers and was surrounded by a deep moat. Ten years earlier, in 1255, Ulrik from Hrastovec (Vlricus de Hage) was mentioned as a witness in a document of Frederick the Younger of Ptuj. Hrastovec is mentioned again in 1294, when Adelhajda of Hrastovec is mentioned as a benefactor of the Cistercian monastery Rein near Graz. After Ulrik, the next owner of Hrastovec was his brother Gunter of Hrastovec (1346 – 1376). The very beginnings of the fortification on the site of the castle could date back to the second half of the 10th century. At that time, Hungarian campaigns and raids across central Slovenia ceased. In any case, only a small manor stood on this site until the 15th century, as Hrastovec is first mentioned as a castle and lordship in 1457 (veste und gschloss Gutenhag). The first known owners of the Hrastovec lordship were the Hrastovec family, from whom the ownership and management of the lordship was transferred to the Herberstein family. However, since Günter Herberstein died without heirs in 1421, the new managers of the Hrastovec lordship became the Holeneški family. Under Henry of Holeneški, the castle was first referred to as a fortress (known as Gutenhag) in 1443. The Holeneškis acquired the right of blood justice for the castle and its subjects in Močna, Ruperče and Jablanice. They obtained this right in 1447, when Andrej of Holeneški was its lord. Andrej of Holeneški obtained market rights from the ruler Frederick III. in the same year (1447). With this, Lenart gained the right to seven livestock fairs per year and a weekly market day. The market judge, who was elected by all full-fledged market residents and confirmed by the Hrastov castle lord, is mentioned in Lenart since 1332.
The story in the novel by Ožbalt Ilaunig tells of the unhappy love between the lord of Hrastovce, Friderik Herberstein, and Agata Nurnberger from the Štralek castle. According to Frederick's mother, Agatha was too lowly to marry him, which is why he opposed the marriage between them. Nevertheless, Agatha and Frederick were secretly married in the Jacob's Church, which was located in Lormanj. The mother's revenge was threatening, because in Frederick's absence, she had Agatha accused of witchcraft, imprisoned and sentenced to beheading and burning her body at the stake. Frederick Herberstein had a black cross erected at the place where this was supposed to happen.