Praise be to Allah.
If there was an agreement between you and the agent that you would pay the extra cost of 40,000 in return for staying in double occupancy rooms for the duration of your trip to the holy places, then the agent is obliged to uphold that agreement, because Allah, may He be exalted, says (interpretation of the meaning): “O you who believe! Fulfill (your) obligations [or contracts]” [al-Maa’idah 5:1]. And the Prophet (blessings and peace of Allah be upon him) said: “The Muslims are bound by their conditions.” Narrated by Abu Dawood (3594); classed as saheeh by al-Albaani in Saheeh Abi Dawood.
It is not permissible for him to demand extra payments or to claim that that money was only for accommodation in Madinah, so long as the agreement covered the entire trip.
What matters in this case is the wording of the agreement between you and him, not whether he took money from someone else or let him off. Each person should act according to his contract and agreement, and the agent or travel agency may let some pilgrims off and not others.
If you are certain that the agreement covered the duration of the entire trip, i.e., it covered your stay in both Madinah and Makkah, then you do not have to pay anything more than what you already paid, and it does not matter that you agreed to pay the additional amount and promised to do that.
And Allah knows best.