It is thought this castle was built by the count of Aumale, Baldwin de Bethune during the 12th century. It is suspected it was used as a residence for 150 years by lords and earls of importance. The castle would have had an outer and inner bailey and ruined sections of curtain wall and an additional tower have been found on nearby land.
The remains of the tower you can see today most likely contained stores on the ground level, a hall on the 1st floor with possible additional rooms and a 2nd floor, possibly only for defence.
The castle is recorded as being owned by Simon de Montfort, sixth earl of Leicester, in 1238. Earl Montfort led a baronial rebellion against Henry III and was killed at the Battle of Evesham in 1265, his estates were confiscated by the Crown.
The castle was conferred by Henry III to his half-brother, William de Valence, to reward him for his support through the rebellion. Surviving Valance household accounts survive and record that between May 1296 and September 1297 the widow of William de Valence, Joan, spent a month at this castle.
Williamís son, Aymer de Valence inherited the castle in 1307 and stayed here in June 1315. This is the last real record of the use of the castle.