Praise be to Allaah.
There is no harm in you and your wife living in one house with the rest of the family members you mentioned. All that you have to be careful about is that your wife does not go out in front of your brother or be alone with him in the house, because the Prophet (peace and blessings of Allaah be upon him) forbade non-mahrams (unrelated men) to enter upon women. He said: "Beware of entering upon women." One of the Sahaabah said to him, "O Messenger of Allaah, what about the brother-in-law?" He said: "The brother-in-law is death!" (Reported by al-Bukhaari, Fath al-Baari, 9/330).
Al-Nawawi, may Allaah have mercy on him, said: "What is referred to in the hadeeth is all the husband’s (male) relatives apart from his father and sons. People customarily take the matter of a man being alone with his brother’s wife as being of little consequence; to indicate the seriousness of the matter, it was likened to death. Indeed, one should be more cautious about the brother-in-law than about a stranger. The phrase "the brother-in-law is death" may have a number of meanings:
All this stems from Islam’s keenness to protect households, to avoid evil and to preserve marriages in the best possible way. You will find more details under question #217 .
May Allaah bless our Prophet Muhammad.
Sheikh Muhammed Salih Al-Munajjid