The Sunnah is for the pilgrim to remain in Mina during the day, because that is what the Prophet (blessings and peace of Allah be upon him) did. He may go out from there to Makkah or Jeddah and the like, especially if that is because he has no place to stay in Mina.
Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) was asked: Does going out during the days of at-tashreeq to places near Makkah, such as Jeddah for example, affect one’s Hajj?
He replied: It does not affect one’s Hajj, but what is preferable is to remain in Mina night and day, as the Prophet (blessings and peace of Allah be upon him) remained there night and day.
End quote from Majmoo‘ Fataawa ash-Shaykh Ibn ‘Uthaymeen (23/241- 243)
See also the answer to question no. 36244
Tawaaf al-wadaa‘ (the tawaaf of farewell) can only be done after one has completed all the rituals, i.e., after the days of Mina and stoning the Jamaraat. It is not permissible and is not valid to do it before that. So whoever does tawaaf al-wadaa‘ on the tenth or eleventh day (of Dhu’l-Hijjah), that is not valid.
It is permissible to delay tawaaf al-ifaadah until the time of tawaaf al-wadaa‘, as has been explained previously in the answer to question no. 36870.
Shaykh Muhammad ibn Ibraaheem (may Allah have mercy on him) said:
As for the one whose house is in Jeddah, if he does tawaaf al-ifaadah before finishing the stoning of the Jamaraat, and he intends in his tawaaf that it should be both tawaaf al-ifaadah and tawaaf al-wadaa‘, this does not count for tawaaf al-wadaa‘, because he has not yet completed the actions of Hajj.
But if this tawaaf is for ifaadah after he has completed the stoning of the Jamaraat, and he intends it as such and did not perform tawaaf al-wadaa‘ after that, and did not stay in Makkah after that – rather he left straightaway – then it will suffice for tawaaf al-wadaa‘ too.
End quote from Fataawa ash-Shaykh Ibn Ibraaheem (6/108)
To sum up: what you did as tawaaf al-wadaa‘ was not valid, and your departure for Jeddah after doing the rituals without a tawaaf is something for which you must offer a compensatory sacrifice, which is a sheep to be slaughtered in the Haraam and its meat distributed among the poor there.
Your wife also has to sacrifice a sheep, so long as she was not menstruating at the time of departure, because tawaaf al-wadaa‘ is waived in the case of a woman who is menstruating, because of the report narrated by al-Bukhaari (1755) and Muslim (1328) from Ibn ‘Abbaas (may Allah be pleased with him) who said: The people were commanded that the last thing they should do is to circumambulate the House, but an exception was made for menstruating women.
Your tawaaf al-wadaa‘ is not valid as you did it, and the compensatory sacrifice cannot be waived by doing it in your case, because you left Makkah without doing tawaaf al-wadaa‘.
Shaykh Ibn Baaz (may Allah have mercy on him) was asked: We are residents of Jeddah, and we came last year for Hajj. We completed all the rituals except tawaaf al-wadaa‘, which we delayed until the end of Dhu’l-Hijjah, after the crowds grew less. Is our Hajj valid?
If a person does Hajj and delays tawaaf al-wadaa‘, his Hajj is valid, but he must do tawaaf al-wadaa‘ when he decides to leave Makkah. If he lives outside Makkah, such as the people of Jeddah and the people of at-Taa’if, Madinah and so on, then he should not leave until he bids farewell to the Ka‘bah with a tawaaf of seven circuits only, and there is no sa‘i, because there is no sa‘i in wadaa‘ (farewell); rather it is tawaaf only. If he leaves without doing tawaaf al-wadaa‘, then he must offer a compensatory sacrifice according to the majority of scholars, to be slaughtered in Makkah and its meat distributed to the poor and needy. But his Hajj is valid as mentioned above. This is the view of the majority of scholars. To sum up, tawaaf al-wadaa‘ is an obligatory ritual, according to the most correct scholarly view. It is proven from Ibn ‘Abbaas (may Allah be pleased with him) that he said: Whoever omits a ritual or forget it, let him offer a sacrifice. This ritual was omitted deliberately, so the person must offer a sacrifice, to be slaughtered in Makkah and the meat distributed to the poor and needy. The fact that he wants to come back to Makkah later on to do it, does not mean that this obligation could be waived. This is the favoured view, and this is the most correct view in my opinion. And Allah knows best.
End quote from Majmoo‘ Fataawa Ibn Baaz (17/397)
And Allah knows best.