In the north-western direction from the current Komorní Hrádek there used to be the smaller castle of Hrádek Čajchanov, located on a promontory steeply falling into the Sázava valley and founded probably towards the end of the 13th century. Its scarce remnants have been preserved in a location approximately 200 metres from the castle. In about 1400 it belonged to Jan Zoule of Ostředek who, together with his cavalcade, took part in a minor skirmish in Bohemia at the time when King Václav I. was captured by the Noblemen’s Union. In 1404 Zoule’s cavalcade was captured by the land protection army and Hrádek was conquered and reduced to rubble. The estate, where the small chateau used to stand, was given as a gift of the King to Vice-Chamberlain Mikuláš.
Vyšehrad‘s Burgrave Racek of Dvojice was given the King’s permission to built a new castle in 1412. His castle was called Veselé. Racek used it as his seat until 1416 when, being the King‘s messenger, he was killed by minors in Kutná Hora. Racek’s widow later passed on her right to local possessions to Vilém Kostka of Postupice. He was followed by his eldest son Bohuše who purchased the nearby castle of Dubá. In 1516 he was the owner of Hrádek and also of the possessions from the previous monastery. In 1525 Hrádek was taken over by Jaroslav of Šelmberk who held the position of the most senior chamberlain in the Czech kingdom and, according to his position, the castle acquired its name Komorní Hrádek - Chamberlain‘s Castle.

An extensive Renaissance reconstruction and extension of the castle comes from the era of guardianship after Valdštejn’s death when the east wing, with a main staircase and a hall, was built. All four wings of the chateau had two storeys and new vaults were added; front facades were decorated with graffito envelope decoration. Another "developer” of Hrádek was Adam of Valdštejn who held the important posts of the most senior burgrave and as governor. He had the medieval ramparts in front of the western and partly also in front of the eastern front walls pulled down and the draw-bridge removed. Further, he also had a new chateau with three one floor high wings and with a prismatic tower in front of the main entrance built. The moat in front of the western front wall was spanned over with a stone bridge and a new Renaissance garden was established in the place of the original south moat. Adam’s son Jan Viktorin carried out an expensive internal refurbishment of the chateau in about 1660. Several ground floor rooms received a mannerist stucco decoration and have the highest preservation value in the whole of this building.

Baroque reconstruction in Hrádek took place in the era of František Josef of Valdštejn who acquired the estate in 1713. A large chapel was built in the north wing of the old castle. A hall with an elevated ceiling and a rich stucco decoration was created in the main wing of the new chateau. Other rooms on the first floor were decorated in a similar way. Visitors can see figural motives, ornamental decorations, ceiling and wall paintings - all of them in the style of High Baroque.