The Prophet (peace and blessings of Allaah be upon him) said: “Any woman who asks her husband for a divorce when there is nothing wrong, the fragrance of Paradise will be forbidden to her.” Narrated by al-Tirmidhi and classed as saheeh by al-Albaani in Saheeh al-Tirmidhi.
What is meant by something wrong is hardship which compels her to seek a separation.
Asking for divorce when there is no problem that would compel a woman to do that is haraam, and some scholars regarded it as a major sin, such as Ibn Hajar al-Haytami in al-Zawaajir.
From the hadeeth quoted above it may be understood that it is permissible for a woman to ask for a divorce if there is some hardship or harm that will be caused if the marriage continues.
Based on this, there is nothing wrong with you seeking divorce if the matter is as you describe, that your husband left you three years ago.
But it is better for you to be patient and ask your husband to try to remedy the problem that has befallen him; perhaps Allaah will heal him and set things straight. But if you feel that there is no improvement at all in the situation and you hope that if you divorce him you will be able to marry someone else, then divorce is better in that case, because Allaah says (interpretation of the meaning):
“But if they separate (by divorce), Allaah will provide abundance for everyone of them from His Bounty. And Allaah is Ever All‑Sufficient for His creatures’ needs, All‑Wise”
What you mention about him being put off and feeling stressed whenever he remembers you may be due to him having been affected by sihr (witchcraft). Hence he should use ruqyahs that are prescribed in sharee’ah, or he should go to a righteous person who adheres to the Sunnah, to perform ruqyah for him.
And Allaah knows best.