What the man gives to his fiancée before the marriage contract may be part of the mahr or it may be a gift; that may be known by means of clear statements or on the basis of what is customary. The wedding dress may be part of the mahr or it may be a gift.
The ruling depends on this distinction.
If the dress was part of the mahr and is known to be such, either because of a clear statement to that effect or on the basis of what is customary in your country, then it should be returned to the man if the engagement is cancelled, regardless of whether it was cancelled by him or by you, because no part of the mahr belongs to the woman until the marriage contract is done, and if it was given to the fiancée, then it was a trust to be looked after by her until the marriage contract was done.
If the dress was a gift, then there is a difference of scholarly opinion concerning the ruling on it. The more correct opinion is that if the engagement was cancelled by the man, he has no right to take back his gifts and to ask them to be returned. If it was cancelled by the woman, then he does have the right to take his gifts back, because they were not purely gifts; rather they were gifts given for something in return, namely marriage. So if they did not give their daughter to him in marriage, it is permissible for him to take back his gifts. This is the view of the Maalikis according to one opinion, and it is the view favoured by Shaykh al-Islam Ibn Taymiyah (may Allah have mercy on him), who attributed it to Imam Ahmad (may Allah have mercy on him).
Based on that, if the dress was a gift and the annulment of the engagement was because of the fiancé, because he failed to fulfil his commitments, then you do not have to return it to him.
If the annulment was on your part although he wanted to go ahead with the marriage and was prepared to fulfil his commitments, then you have to return the gifts that he has asked to be returned, including the wedding dress.
Both parties should cooperate to bring the matter to a rational and wise conclusion, and seek reward (from Allah) for what they have lost. Financial loss usually affects both parties. If the woman takes back whatever she gave him of gifts and the like, she will also be affected by the depreciation in their value and may not be able to sell them.
Both parties should have the hope of better things in the future after learning from this experience.
We ask Allah to help and guide you both.
And Allah knows best.