Map showing Bedfordshire seats of and locations connected with the Grey family, Barons Grey of Ruthin, Duke of Kent and Earls of Kent.

The family of Grey of Ruthin was just one cadet branch of the influential Grey family founded by w:Henry de Grey (1155-1219) of Grays Thurrock, Essex. As Marcher Lords, the main seat of this branch of the family was Ruthin Castle, Denbighshire, North Wales. The maternal grandmother of Edmund Grey, 1st Earl of Kent, 4th Baron Grey of Ruthin (1416-1490), was Elizabeth Plantagenet, a daughter of John of Gaunt, 1st Duke of Lancaster (the 3rd surviving son of King Edward III) and wife of John Holland, 1st Duke of Exeter (half-brother of King Richard II), the third son of Thomas Holland, 1st Earl of Kent, by his wife Joan of Kent, "The Fair Maid of Kent", the daughter of Edmund of Woodstock, 1st Earl of Kent, a son of King Edward I. The lordship of Ruthin was sold to King Henry VII by Richard Grey, 3rd Earl of Kent, 6th Baron Grey of Ruthin (1481-1524), who lost much of the family lands through gambling.

The principal seat since the time of the 1st Earl was the manor of Wrest in Bedfordshire, England (140 miles south-east of Ruthin) held since 1284 by the Grey family. Historically it was within the parish of Flitton, where in 1606 the 6th Earl built the Grey Mausoleum as an expansion to the parish church. Four generations after the extinction of the male line (following the death in 1740 of Henry Grey, 1st Duke of Kent, 12th Earl of Kent, who rebuilt the house at Wrest on a grand scale) the chapel of ease at Silsoe, adjacent to the house at Wrest, was rebuilt and transformed into a church. The work was instigated by Thomas Robinson (who became Thomas de Grey, 2nd Earl de Grey (1781-1859) and was the brother of Prime Minister Frederick Robinson, Lord Goderich) a talented amateur architect (the heir of the 1st Duke via two female lines) who inherited the title Earl de Grey created for his maternal aunt in 1816. He rebuilt the house at Wrest in the French style, as it survives today.

Early members of the Grey family of Ruthin were buried in Warden Abbey (6 miles N-E of Wrest), including George Grey, 2nd Earl of Kent (d.1505) and his first wife Anne Woodville (a sister in law of King Edward IV) and their son the 3rd Earl (d.1524). Later family members were buried in the Grey Mausoleum at Flitton. Letters of various members of the family were dated at Ampthill, where the 2nd Earl died. A junior branch was seated at Cople and Kempston, and in Cople Church survives the monumental brass of Thomas Grey (c.1458-1507), a great grandson of the 3rd Baron, by his second wife Joan de Astley.