His sister is studying in a mixed university; should he spend on her?

Praise be to Allah.

The sister’s husband is a non-mahram for a woman, like all other “strangers”, so it is not permissible for her to show any part of her body in front of him and it is not permissible for him to look at her or shake hands with her or be alone with her. What you have mentioned about her sometimes being alone with him is something that is undoubtedly haraam, and it is a cause of temptation and evil, because the Prophet (blessings and peace of Allah be upon him) said: “Do not be alone with a woman unless there is a mahram with her.” Narrated by al-Bukhaari (1862) and Muslim (1241). And he said: “No man is alone with a woman but the third one present is the Shaytaan.” Narrated by al-Tirmidhi (1171); classed as saheeh by al-Albaani in Saheeh al-Tirmidhi. 

Al-Nawawi (may Allah have mercy on him) narrated in Sharh Muslim (14/153) that there was consensus among the scholars that it is haraam for a man to be alone with a woman who is a non-mahram. This was also mentioned by al-Haafiz in al-Fath (4/77). 

See also question no. 13261. 

Secondly: 

It is haraam to study in co-ed schools and universities because the haraam things and evils that result from that are obvious to everyone, such as looking, being alone with a member of the opposite sex, speaking softly, infatuation and things that are even worse than that. This has been discussed previously in detail; see the answer to question no. 45883 and 8827. 

Thirdly: 

What you have to do is advise your sister and explain to her that it is haraam for her to stay in her sister’s house if that will result in her being alone (with her sister’s husband) or taking lightly the matter of looking or hijab. You should also tell her that it is haraam for her to study in a co-ed university and you should try to make her study in an environment where there is no mixing, even if that means studying at home or distance learning. It is not permissible for you to help her in this haraam study with any of your wealth, because Allah says (interpretation of the meaning):

“Help you one another in Al‑Birr and At‑Taqwa (virtue, righteousness and piety); but do not help one another in sin and transgression. And fear Allaah. Verily, Allaah is Severe in punishment”

[al-Maa’idah 5:2]. 

May Allah help us and you to do that which He loves and which pleases Him. 

And Allah knows best.