The meaning of the verse is one who hastens to leave within two days of the days of al-Tashreeq, which are the eleventh, twelfth and thirteenth. So hastening to leave means leaving on the twelfth. Perhaps the questioner assumed that that first day was the day of Eid, but this is incorrect.
Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) said:
I would like to alert our brother pilgrims to this mistake, because many pilgrims assume that what is meant by the verse “Whoever hastens to leave in two days” [al-Baqarah 2:203 – interpretation of the meaning] is leaving on the eleventh, so they count the two days as the day of Eid and the eleventh of Dhu’l-Hijjah. But this is not the case, rather this is a misunderstanding, because Allaah says (interpretation of the meaning):
“And remember Allaah during the appointed Days. But whosoever hastens to leave in two days, there is no sin on him”
The appointed days are the days of al-Tashreeq, and the first of the days of al-Tashreeq is the eleventh. Based on this, the phrase “whoever hastens to leave in two days” means in two days of the days of al-Tashreeq, which is the twelfth. So people should make sure that they understand this matter correctly, so that they will not make mistakes.
Fataawa Arkaan al-Islam, p. 566.