He said salaam before the end of the prayer by mistake then he started another prayer

Praise be to Allah.

If a person forgets and says the salaam before completing his prayer, then he remembers shortly afterwards, then he should go back and complete his prayer and do the prostration of forgetfulness, because of the report narrated by al-Bukhaari (1227) and Muslim (573) from Abu Hurayrah (may Allaah be pleased with him) who said: The Messenger of Allaah (peace and blessings of Allaah be upon him) prayed Zuhr or ‘Asr and he said the salaam (i.e. after two rak’ahs). Dhu’l-Yadayn said to him: Has the prayer been shortened, O Messenger of Allaah? The Prophet (peace and blessings of Allaah be upon him) said to his companions: “Is it true what he is saying?” They said: Yes. He prayed two more rak’ahs then he prostrated twice. 

If he has started to offer a naafil prayer, then he remembers that he did not complete the obligatory prayer, he should stop the naafil prayer and complete the obligatory prayer, then do the prostration of forgetfulness. 

Ibn Qudaamah said in al-Mughni (1/373): The one who says the salaam before completing his prayer by mistake, then realizes before a long time has passed or his wudoo’ is invalidated, has to do what remains of his prayer, then recite the tashahhud and say the salaam, then do the two prostrations of forgetfulness. If a long time has passed or he has invalidated his wudoo’, then he should repeat the prayer from the beginning. 

If he does not remember until he has started another prayer, then it depends. If he has only done a little of the second prayer and a long time has not passed, then he should go back to the first prayer and complete it. If a long time has passed then the first prayer is invalidated. This is the view of al-Shaafa’i. End quote. 

In Kashshaaf al-Qinaa’ (1/400) it says: If the one who said the salaam before completing the prayer does not remember until he has started another prayer, he should stop it if a short time has passed, and go back to the first prayer and complete it, so that he can attain continuity between its essential parts, then he should do the prostration of forgetfulness. End quote. 

There is no specific definition of what counts as a long time or a short time; he should refer to what is customary. 

As for making the first (deficient) obligatory prayer a voluntary prayer, that is not correct, because actions are by intentions, and you prayed it with the intention of offering an obligatory prayer, and you exited it likewise. Because you repeated the obligatory prayer after that, then you have done what was required of you. But it would have been better for you to stop the naafil prayer and go back to the obligatory prayer and complete it, then do the two prostrations of forgetfulness after saying the salaam at the end of the prayer, then pray the naafil prayer after that. 

And Allaah knows best.