Facing the castle, on the western side of the pill, stand the considerable remains of Monkton Priory, a Benediction house founded by Earl William Marshal as a cell to the abbey of Seez or Sayes in Normandy, but under Henry VI.
He went in 1652 to Sayes Court at Deptford, a house which Sir Richard Browne had held on a lease from the crown.
In 1694 he left Sayes Court to live at Wotton with his brother, whose heir he had become, and whom he actually succeeded in 1699.
Evelyn's house at Sayes Court had been let to Captain, afterwards Admiral John Benbow, who was not a "polite" tenant.
A mansion known as Sayes Court, taken down in 1729, was the residence of the duke of Sussex in the reign of Elizabeth; it was occupied in the following century by John Evelyn, author of Sylva, and by Peter the Great during his residence in England in 1698.