Al-hamdu lillah (praise be to Allah). Before answering this question, it is important to warn the Muslim brothers residing in countries of the kuffar (unbelievers) of the danger in marrying unbelieving women from these countries which enforce their man-made system of laws. And much of what is ruled upon regarding possession of children is in the favor of the unbelieving mother, due to the consideration that she is a citizen of this country or perhaps any other factor, such as her role as a mother, regardless of her religion. Unfortunately, via this way many Muslim children have been lost and have been converted to Christians. There is a clear violation of a principle of sharia in this case which asserts that al-islam yaloo wa laa yulaa alaih, i.e. Islam must be always given the preference and the highest priority in consideration and nothing can come above it. Accordingly, the child must be given to the Muslim parent when there is a separation between a husband and wife and one of them is Muslim and the other kaafir (unbeliever). So a very urgent advice to the those brothers who are compelled to live in the countries of the kuffar and to submit themselves to many rulings of the jaahiliyya (era of pre-Islamic ignorance), is that they should not get themselves caught in a bind with such marriages to unbelievers, after which follows much distress.

As for the questionwhat is the ruling regarding inviting the unbeliever wife to make mubaahala--the calling of the unbelievers from ahl il-kitaab (People of the Book) to mubaahala has been revealed in the Quran, as in surat Aal Imran, verse 61 (a translation of the meaning):

If anyone disputes in this matter with you now, after what (full) knowledge has come to you, say, Come! Let us gather togetherour sons and your sons, and our women and your women, and ourselves and yourselvesthen let us earnestly pray and invoke the curse of Allah on those who are the liars. (3:61)

And mubaahala is a type of ibtihaal (supplication) and it is: the endeavor in making duaa for cursing etc, actively performed by two or more parties so that it falls upon one of the two parties. And its definition according to sharia is the gathering of people when they differ on an issue and the calling for Allahs curse on the transgressor among them. And the Muslim fathers calling of this woman to mubaahala for Allahs curse and revenge to descend upon whoever is guilty of falsehood of the two parties would be something prescribed and he can do it so that perhaps it may affect the woman.

It is essential and important advice in this aforementioned case that the father maintains a consistent link between himself and his two sons via visits and fosters friendly relations through various gifts and toys so that the children remain on good ties with their father. This will be helpful for him to win them over to the Islamic religion.

We ask Allah for protection and to guide our brothers children to Islam and to inspire his integrity in action and to guide us all to what is correct.

Wallahu alam (and Allah knows best).