Firstly: I would like to remind you of what the Messenger (peace and blessings of Allaah be upon him) said: “I have not left behind me any fitnah (temptation) that is more harmful to men than women.” (Narrated by al-Bukhaari, 4808; Muslim, 2740, from the hadeeth of Usaamah ibn Zayd).
Secondly: It is permissible for a woman to be in her father’s house when there is a non-mahram present, so long as there is no fear of haraam mixing or of the fitnah (temptation) that may happen as a result of that.
But if there is no guarantee that there will be no mixing and no fitnah, then that is not permissible. It is even more haraam if the woman does not observe proper Islamic hijaab.
Thirdly: a boy of four years of age is not considered to be a mahram because the role of a mahram is to guard and protect the woman, and this cannot be done by a four year old child. So it makes no difference whether this boy is there or not.
Fourthly: if your wife’s father and brothers take this man’s presence lightly and do not care if he is alone with your wife or if she takes off her hijaab in front of him, then it is not permissible for you to leave your wife with them, because they are careless about protecting their daughter. The Prophet (peace and blessings of Allaah be upon him) said: “Beware of entering upon women.” A man from among the Ansaar said: “O Messenger of Allaah, what do you think of the in-law?” He said, “The in-law is death.” (Narrated by al-Bukhaari (4934) and Muslim (2172), from the hadeeth of ‘Uqbah ibn ‘Aamir). So you must protect your wife and feel protective jealousy and pride (gheerah) where she is concerned, and not expose her to temptation.
Fifthly: With regard to her going on trips, there is nothing wrong with that if the place to which she is going is not one where immoral people meet, such as beaches and other places which are very dubious, and so long as she will keep her hijaab on and stay far away from places of temptation. If you are worried that the presence of that non-mahram man will be a cause of fitnah or may make her do something that Allaah has forbidden, then you must not allow your wife to travel with them. This is the matter of praiseworthy protective jealousy (gheerah) that is loved by Allaah and His Messenger. If a man feels protective jealousy towards his family he will stop them from committing immoral actions and anything that may lead to that.
You should note that most of the disasters that befall families come from friends or relatives, so try your best to protect your wife from that.
And Allaah knows best.