With the young prince, the future king, Bute's intimacy was equally marked; he became his constant companion and confidant, and used his influence to inspire him with animosity against the Whigs and with the high notions of the sovereign's powers and duties found in Bolingbroke's Patriot King and Blackstone's Commentaries.
He still for a short time retained influence with the king, and intended to employ George Grenville (whom he recommended as his successor) as his agent; but the latter insisted on possessing the king's whole confidence, and on the failure of Bute in August 1763 to procure his dismissal and to substitute a ministry led by Pitt and the duke of Bedford, Grenville demanded and obtained Bute's withdrawal from the court.
He still corresponded with the king, and returned again to London next year, but in May 1765, after the duke of Cumberland's failure to form an administration, Grenville exacted the promise from the king, which appears to have been kept faithfully, that Bute should have no share and should give no advice whatever in public business, and obtained the dismissal of Bute's brother from his post of lord privy seal in Scotland.
The remainder of Bute's life has little public interest.
His work of greatest value, his Annals, still remains in MS. in Lord Bute's and Lord de la Warr's collections (Hist.