Should he do the prostration of forgetfulness after doing the part of the prayer that he has to make up?

Praise be to Allah.

Yes, you have to do the prostration of forgetfulness after completing your prayer, because if one who is praying behind an imam makes a mistake, he does not have to do the prostration of forgetfulness unless he joined the prayer late. 

Al-Nawawi (may Allaah have mercy on him) said in al-Majmoo’ (4/63): 

If he makes a mistake when praying behind the imam, the imam carries his mistake, and neither of them should do the prostration of forgetfulness. There is no difference of scholarly opinion on this point. Shaykh Abu Haamid said: This is the view of all the scholars except Mak-hool, who said: The one who is praying behind an imam should do the prostration of forgetfulness for a mistake that he made. End quote. 

The evidence for that is the words of the Prophet (peace and blessings of Allaah be upon him): “The imam has been appointed to be followed, so do not differ from him.” Narrated by al-Bukhaari (722) and Muslim (414). 

If the person praying behind an imam does the prostration of forgetfulness, this means that he is differing from the imam. 

But if he joined the prayer late, then he may do the prostration of forgetfulness when he has completed his prayer, because he is completing it on his own, so then his prostration of forgetfulness does not mean that he is differing from the imam. 

Secondly: 

Imam al-Nawawi (may Allaah have mercy on him) said in al-Majmoo’ (4/64): 

If he joins the prayer late, then he makes a mistake after the imam says the salaam, the imam will not carry it for him, because he is no longer following him. End quote. 

It says in al-Mawsoo’ah al-Fiqhiyyah (8/206): 

If the person who joined the prayer late says the salaam after the imam by mistake, he should continue making up what he has missed of his prayer and do the prostration of forgetfulness. End quote. 

Shaykh al-Sa’di (may Allaah have mercy on him) said: 

If he joined the prayer late and he makes a mistake in the part of the prayer that he is making up, then he has to do the prostration of forgetfulness. End quote. 

Fataawa al-Sa’di (157). 

Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) said in Majmoo’ al-Fataawa (14/ no. 698): 

If the person praying behind the imam makes a mistake in his prayer, and he joined the prayer late, he has to do the prostration of forgetfulness if his mistake is one that requires the prostration of forgetfulness. End quote. 

That includes the person who joined the prayer late saying the salaam after the imam by mistake. In that case he has to stand up and offer the part of the prayer that he missed, then do the prostration of forgetfulness. 

And Allaah knows best.