Castanet dʼEscarts was originally the site of a very ancient human occupation. Tools found on a Magdalenian site attest to this. The standing stones known at Cambayrac are proof of Neolithic occupation. 
From the High Middle Ages, sarcophagi remain, one of which can be seen near the church in Castanet, discovered on the site of the current Café Castagné, close to the old fort. As early as the 12th century, the De Castanet family owned the seigneury...

The 2200-hectare commune now comprises two parishes: Castanet and its annex Combayrac and Cuzoul. Although small, the commune is no less lively, and its inhabitants are very active through numerous cultural and leisure associations.

The commune is divided between the Segala to the east and the limestone plateaux to the west. The Segala side has the highest point in the department (504m) and is partly covered in chestnut groves, which have given their name to the commune.