The one who does not pray at all is a kaafir as you mention, whether he does not pray because he denies that prayer is obligatory or because he is lazy. This is according to the more correct of the two scholarly opinions. Some of the scholars even said that one who delays an obligatory prayer until its time is over is a kaafir.
It says in Fataawa al-Lajnah al-Daa’imah (6/30) concerning the case of a woman who delays the prayers until their time is over and encourages her grown-up and small daughters to do likewise:
“If her situation is as described, then she is an apostate who is corrupting her daughters and the daughters of her husband. She should be asked to repent, and if she repents and reforms, then praise be to Allaah. But if she persists in that, then her case should be referred to the ruler to separate her from her husband, and the hadd punishment should be carried out on her, namely execution, because of the hadeeth of Ibn ‘Abbaas (may Allaah be pleased with him) who said that the Prophet (peace and blessings of Allaah be upon him) said: “Whoever changes his religion, kill him.” This is if she delays the prayers until their time is over, such as delaying ‘Asr until the sun goes down, or delaying Fajr until the sun rises, because delaying them until the time is over with no legitimate shar’i excuse comes under the same ruling as not praying.”
Based on this, it is not permissible for you to marry this young man no matter how nice his character. What good character can be left after abandoning prayer and dealing in riba?!
So long as he has not repented from that and the signs of righteousness have not appeared in him, then you should reject his proposal of marriage. If the marriage contract has already been done then you should tell him that the contract is not valid because he does not pray and it is not permissible for a Muslim woman to marry a kaafir. But if he repents and prays regularly on time, then you must make a new marriage contract with him, because the first contract was not valid.
Do not be deceived by his words and promises. If he does not keep his word during the engagement period, then do not expect him to keep his word after that.
Your saying that you cannot leave him is a trick of the Shaytaan. Rather you can do that, by putting your trust in Allaah and seeking what is with Him, and because of your fear of falling into that which is haraam. It is not right for a kaafir to be the husband of a Muslim woman under any circumstances.
It seems from your question that the marriage contract has been done, because you say that the most hated of permissible things in the sight of Allaah is divorce, but at the end of your question you mention engagement. If the marriage contract has not yet been done, then we would remind you that the fiancé is a non-mahram for his fiancée and it is not permissible for him to be alone with her or to see any part of her, or for her to speak to him in an alluring voice or to speak to him needlessly. All that is permissible at the time of engagement is for him to see of her that which will encourage him to marry her, without being alone with her.
The best advice we can give you is to fear Allaah in private and in public, and to pray to Allaah to bless you with a good and righteous husband.
And Allaah knows best.