In 1786 he was appointed vicar of Kingston-on-Thames, and in 1788 rector of Bemerton, Wiltshire.
During a long residence at Bemerton Coxe was mainly occupied in literary work.
John Norris of Bemerton also strongly influenced him by his Essay on the Ideal World (1701-1704).
In 1630 Charles I., at the instance of the earl of Pembroke, whose kinsman Herbert was, presented him to the living of Fugglestone with Bemerton, near Salisbury, and he was ordained priest in September.
The story of the poet's life at Bemerton, as told by Walton, is one of the most exquisite pictures in literary biography.
Walton illustrates Herbert's kindness to the poor by many touching anecdotes, but he had not been three years in Bemerton when he succumbed to consumption.
Nothing in his own church at Bemerton was too commonplace to serve as a starting-point for the epigrammatic expression of his piety.