Praise be to Allah.
If it is proven that your brother’s husband is as you describe him and that he is a homosexual and does the action of the people of Loot, then it is not permissible – in this case – for your husband to admit him to his house, let alone allow him to meet his sons. It is also not permissible for him to ask you to show respect to him, because he is not one of those who are deserving of respect; rather he is one of those who are deserving of rebuke and shunning, and he deserves to be shown hostility because of what he is, not to be respected, because what he is doing is a grave evil, immorality, and act of disobedience. The major sin that he is doing – if it is proven – is one of the most horrid and repulsive of major sins. Hence those who do it are deserving of the greatest punishment such as landslides, stoning and drowning such as no nation of disbelievers were punished with before or since.
Ibn al-Qayyim narrated – in Zaad al-Ma‘aad (5/40) – from his shaykh, Ibn Taymiyah (may Allah have mercy on him), that the Sahaabah (companions of the Prophet) were unanimously agreed that the one who does the action of the people of Loot is to be executed; they only differed as to the manner of execution.
Hence we agree with you in your stance about him entering your house, and we also agree with you about the kind of treatment that he deserves. So do not hesitate to prevent him from entering the house in the first place, let alone staying there. What your husband has to do is to take the decision himself; rather he should have taken a decision before you did, because Islam has given him the responsibility of taking care of his family, looking after them and advising them. His giving permission to his homosexual brother to enter his house and meet his sons, and asking his wife to show respect to him – all of that is contrary to the obligations that Allah has given him of protecting his family and taking care of them. It is a kind of haraam (unlawful) betrayal of his flock for whom Allah has made him responsible.
But if it so happens that this homosexual has repented sincerely before he enters your house or stays there, and he shows you that he has truly repented and mended his ways, then in that case there is nothing wrong with him entering the house, provided that the wife does not appear before him or mix with him or spend time alone with him; these actions are basically forbidden even if the person is pious. He is still the “in-law” whom Islam warns about. We have explained in more than one answer the seriousness of this problem, how widespread it is and how people are heedless concerning it. So in the case of a person like this, it is essential to deal with him, even after his repentance, with the greatest caution and to block all ways that may lead to evil and temptation.
Let your husband also read the answer to question no. 147901 in which there is discussion of what the brother’s responsibilities are towards his homosexual brother.
And Allah knows best.