Kunšperk Castle was built in the early 12th century, when its owner was supposedly a free lord Otto Ehrnegg. Documents show that in 1173, the bishop of Krk, Henry, had the original castle demolished due to a dispute with the Ehrneggs, who originated from Velikovec in Carinthia. The new castle was built between 1167 and 1174 and was managed until the end of the 14th century by the Kunšperk lords, relatives of the Ptuj lords. The Kunšperk lords were ministeriales of the Krk bishops, who had acquired this property from Countess Hema Breža-Selška through the Krk monastery. Otto, who was often accompanied by the Styrian margraves, died shortly after 1185. The bishop of Krk then entrusted the defense to his alleged nephew Otto, brother of Frederick of Ptuj. Otto I was also one of the most distinguished knights at the famous tournament in Breže in 1224, when he fought with Ulrik of Liechtenstein himself (/.../ dar kom von Kungesperc ein helt /.../). He is mentioned three times in the poem Vrowen dienest – Aventiure von dem turnay ze Frisach. Otto II was succeeded as head of the family after 1250 by his son of the same name, Otto III. After his death around 1270, his sons Otto IV and Frederick ruled together in the first years. After Frederick's death around 1323, Hans took over the business, who died shortly before 1388, and after his death, his son Andrej took over the estate, who died in 1395. With him, the Kunšperski family became extinct in the male line.
At the beginning of the 13th century, the Kunšperski and Ptuj families acquired a very extensive allodial estate in the Upper Savinja Valley, which they divided equally. It is obvious that they were joint heirs of some Carinthian lords. The Ptuj family received Kacenštajn and the territory of Žamberk, while the Kunšper family received the territory of the later Hekenberg, Turn and Velenje.
The Kunšper family and their knights ruled the castle until 1395, when the knight Andrej handed it over to Herman I of Celje. Immediately after the formal granting of Kunšperk to the Celje family, Andrej's sister Agnes, married for the second time to Henry of Ehrenfels, complained and claimed the castle for herself as an heiress. Kunšperk was a hereditary fief, which could also be inherited by women. Therefore, the bishop had to grant her Kunšperk in 1397, whether she wanted to or not. She probably then immediately came to an agreement with the Celje family and ceded the castle to them, since no potential heir of Andrej's after 1395 was mentioned in it.