Praise be to Allah.
If an adulterer or anyone else who has committed a sin repents sincerely, Allaah will accept his repentance and expiate for his bad deeds, as is indicated by a great deal of evidence in the Qur’aan and Sunnah. For example, Allaah says (interpretation of the meaning):
“Say: O ‘Ibaadi (My slaves) who have transgressed against themselves (by committing evil deeds and sins)! Despair not of the Mercy of Allaah, verily, Allaah forgives all sins. Truly, He is Oft-Forgiving, Most Merciful”
Indeed, if his repentance is good, his bad deeds may be replaced with good, by the immense grace and mercy of Allaah, as Allaah says (interpretation of the meaning):
“And those who invoke not any other ilaah (god) along with Allaah, nor kill such person as Allaah has forbidden, except for just cause, nor commit illegal sexual intercourse and whoever does this shall receive the punishment.
The torment will be doubled to him on the Day of Resurrection, and he will abide therein in disgrace;
Except those who repent and believe (in Islamic Monotheism), and do righteous deeds; for those, Allaah will change their sins into good deeds, and Allaah is Oft-Forgiving, Most Merciful”
What is implied by the forgiveness of Allaah and His acceptance of repentance is that He will not punish him.
But if a person persists in committing zina and does not repent from it, then it is proven that such a person will suffer different kinds of punishment, in this world, in the grave and in the Hereafter. We do not find any text that states that he will be deprived of al-hoor al-‘ayn, but some of the scholars have reached this conclusion by analogy with the proven warning concerning one who dies without repenting from drinking wine, who will not drink it in the Hereafter; and one who does not repent from wearing silk in this world, who will not wear it in the Hereafter.
Ibn al-Qayyim (may Allaah have mercy on him) listed the punishments that will befall the one who commits zina, if he does not repent:
“He exposes himself to missing out on the enjoyment of al-hoor al-‘ayn in the blessed abode of the Paradise of ‘Aden (the Garden of Eden). For Allaah will punish the one who wears silk in this world by depriving him of it on the Day of Resurrection; and He will punish the one who drinks wine in this world by depriving him of it on the Day of Resurrection; so by the same token He will punish the one who enjoyed haraam intimacy in this world. Indeed, everything of haraam things that a person enjoyed in this world, he will be deprived of its legitimate counterpart in the Hereafter.”
(Rawdat al-Muhibbeen by Ibn al-Qayyim, 365-368)
With regard to the hadeeth which says that the Prophet (peace and blessings of Allaah be upon him) said “Whoever commits adultery, the same will be done to him (i.e., his womenfolk), even within the walls of his house”, this is a fabricated (mawdoo’) hadeeth, as was stated by al-Suyooti and Ibn ‘Arraaq, and by al-Albaani in al-Silsilah al-Da’eefah, 2/155.
Based on that, there is no point to your objection. Even if the hadeeth were saheeh, it could be interpreted in a sound manner, which is that a man who commits zina and persists in it is one of the immoral and corrupt, and this corruption will have an effect on his family, because it will influence his family and make them corrupt as well. If the head of the household is careless about his own attitude, he will be even more careless about his family, and he will not discipline them to follow their religion properly. As a result, it is not unlikely that they will fall into the same sins as he did, because their faith is weak. In fact there are many stories from real life which point to such cases. This is one of the punishments which are hastened in this world, by means of which Allaah punishes those who violate the honour of the Muslims, so Allaah causes their honour to be violated in turn, as a fitting punishment. Allaah does whatsoever He wills, in accordance with the utmost wisdom and perfect justice, and He is never unjust towards anyone. He is not to be questioned as to what He does, and He is the Most Wise, All-Knowing.