The castle was built in 1720, initially as a classic country manor house, the summer residence of the Esterházy family. Originally a baroque building, it was rebuilt in the 19th century in the neoclassical style. The castle is surrounded by a beautiful English park. Count Ján Károly Esterházy (1775 - 1834) was known for his positive attitude towards culture, for example by supporting the Budapest. 

National Museum and contributed to the establishment of the Hungarian Academy of Sciences. In 1818, he invited Franz Schubert, now a world-famous composer, to teach music to his daughters Maria and Caroline. Schubert spent a few months here in the service of the family, later returning after six years in 1824. According to the information available, his stay was short (May-December), but it was a significant and inspiring period for his work. It is more than likely that his composition Fantasia in F minor was inspired by his secret desire for the young Countess Caroline.

Another person who made the life of the Esterházy family more interesting - or rather sweeter - with his stay was the Viennese confectioner Franz Sacher, who became world famous for his speciality cakes. Historical documents confirm that between 1842 and 1843 he worked in Zselíz as head chef to the count's family and his son. Eduard, who later founded the prestigious hotel chain, was born here in February 1843.

Near the castle is the so-called Owl Mansion, which today serves as the Municipal Museum, a showcase of local history. The neoclassical building was built by the aforementioned János Károly Esterházy for his family teachers: the façade is decorated with a statue of an owl, a symbol of wisdom. On the first floor of the house is the F. Schubert Memorial Room, which, with period furniture and preserved personal objects, shows the everyday life of the composer during his stay.