Muwatta Imam Malik Book of The Two 'Ids, Hadith 14

Yahya related to me that Malik said, "The sunna concerning the time of prayer on the ids of Fitr and Adha - and there is no disagreement amongst us about it - is that the imam leaves his house and as soon as he has reached the place of prayer the prayer falls due."
Yahya said that Malik was asked whether a man who prayed with the imam could leave before the khutba, and he said, "He should not leave until the imam leaves."