There is a descriptive board in front of these caves which says, in effect, that the caves are the remains of a double-storeyed monastery of the 10th century, Parmar Period. They have been named after the ascetic brother of King Vikramaditiya of Ujjain who used them as his hermitage. Bhartrihari abdicated the throne in favour of his brother and devoted himself to spirituality. He is also known for the exquisite meter of his Sanskrit verses. Centuries later , by the 14th and 15th century, these caves had become an important centre of the monastic Naths. Local belief associates these caves with Goraknath and Matsyendranath: both were influential leaders of the sect.
From the car-park, visitors have to descend a long flight of steps to a flat stretch of ground on the banks of the Shipra River. From here, queue rails lead to a few steps accessing another terrace to the left of which is a cave shrine with an image of Bhartrihari. It faces a shivling across the terrace. More steps then pierce & wall running along one side of this terrace, leading to a third terrace from where one can enter the east cave on the right and the west cave on the left.