Muwatta Imam Malik Book of Hajj, Hadith 107

Yahya related to me from Malik that he heard Ibn Shihab say that he had heard one of the people of knowledge say that the Hijr was only enclosed so that people would go beyond it as they were making tawaf, and their tawaf would therefore encompass the original House.

وَحَدَّثَنِي عَنْ مَالِكٍ، أَنَّهُ سَمِعَ ابْنَ شِهَابٍ، يَقُولُ سَمِعْتُ بَعْضَ، عُلَمَائِنَا يَقُولُ مَا حُجِرَ الْحِجْرُ فَطَافَ النَّاسُ مِنْ وَرَائِهِ إِلاَّ إِرَادَةَ أَنْ يَسْتَوْعِبَ النَّاسُ الطَّوَافَ بِالْبَيْتِ كُلِّهِ .