Inigo Revisits: Charles and Romilly Saumarez Smith’s art-filled 18th-century townhouse