The land on which the villa was built originally belonged to the Abbey of Grottaferrata
, which donated it in 1563 to Annibal Caro, who commissioned a small villa where he spent the last years of his life, translating the Aeneid. (In 1896, Prince Leopoldo Torlonia placed a memorial stone to remember this event.)
The villa's grand Baroque terraced gardens and fountains provided subjects for watercolors by the American painter John Singer Sargent and more others painters.
In 1954, the Duke Andrea Torlonia made an exchange of real property with the mayor Micara of Frascati between the "Gardens" of villa Torlonia and the "Quadrato estate": now the gardens are a public park.