Praise be to Allah.
You brother could continue his studies in the specialty mentioned, and also study Islamic sciences with one of the scholars, or after completing his studies, he could join one of the institutes or colleges of sharee’ah in order to learn about his religion and gain knowledge and understanding of sharee’ah. If he fears that continuing to study mathematics will keep him from studying sharee’ah, he can give that up and occupy himself with Islamic studies, because this is more important and more beneficial, and because he was created to worship his Lord with insight. There is no way to do this except by the help of Allaah and then by studying the sciences of sharee’ah. May Allaah makes matters easy for him and bless him in his studies.