The building that now houses the museum and other city facilities was built up in successive layers.
"The deed of acquisition of 5 Vendémiaire Year 11 relates to a house and garden, part of which is planted with groves, located in the street called du petit Versailles, the said house consisting of a large main building, courtyard, backyard, sheds, stables, loft for hay and wood, and quarters for the janitor”. At that time, the building consisted of a dwelling between courtyard and garden, the oldest parts of which date from the 15th century, and a common wing (the north wing) added at the end of the 18th century.

In the 19th century, the Concordat bishops planned, extended and fitted out the building. The classic arrangement on courtyard and garden is the result of successive accommodations:
• In 1813 a project aims to double the main building to form a central avant-corps. Its description evokes the constitution of a south wing symmetrical to the north wing to accommodate a vestibule, a library, a chapel and a library. The avant-corps that we see today in the courtyard is completed; the chapel is the subject of numerous estimates and descriptions until 1831.
• In 1843, the land along rue Joachim Murat was purchased to allow the extension of the main body by the architect Charles Hector Malo, to the south on the garden side, which rests on a base devoted to the orangery. A bedroom and a study are planned on the ground floor while a guest apartment is created upstairs accessible by the elliptical staircase. The dwelling has a slightly projecting central avant-corps on the garden between two symmetrical wings.

The northern extension on the garden side is later (between 1868 and 1896) and abandons both inside and outside the neoclassical style that characterizes the earlier constructions. Its volume, visible from the courtyard, dominates the entire museum.
The neo-Romanesque portal was made at the end of the 19th century.

When the building was made available to the museum in 1929, it appeared to be in good condition. The museum then occupies the central body. The north wing and its entrance pavilion, the orangery and the south pavilion are available to other local institutions.

After the war, renovation work undertaken in the museum continued until 1962-63.
In 1970, the arrival of works by Henri Martin at the museum initiated a new phase of work. The room of the triptych "Le Monument aux Morts", lit from above, is then leaned against the North wing. The museum bears the name of Henri Martin since that date.
As for the museum institution, it was created in 1833 to collect the collection "of the productions of the three kingdoms of nature and of all the objects of antiquity whose study could be used for general history or historical peculiar to Quercy". First departmental, the museum then became municipal in 1841. The Cahors Henri Martin museum today preserves around 10,000 objects and documents, interesting archaeology, extra-European ethnography, history, history of Art and contemporary art.