Praise be to Allah.
The imam, the one who is praying on his own and the one who is praying behind an imam come under the same ruling with regard to doing extra rak‘ahs. If any of them them adds a rak‘ah deliberately, knowing that it is additional, his prayer is invalidated; but hardly anyone does that.
But if the addition to the prayer was by mistake, then either the worshipper (whether he is an imam, praying behind an imam or praying on his own) realizes that it is something additional during the extra rak‘ah, in which case he must sit back down immediately, otherwise his prayer will become invalid, because he deliberately added something to the prayer. He must also recite the tashahhud if he did not do so previously, and prostrate twice (the two prostrations of forgetfulness) after the salaam.
Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said: The phrase “if he realises” means: if he realises that it is something additional during the additional rak‘ah itself.
The phrase: “he should sit back down immediately” means: as soon as he realises it, and he should not delay; even if he remembers whilst bowing that this is an additional rak‘ah, he should sit down.
Some seekers of knowledge may think that the ruling on this issue is the same as the ruling on one who stands up following the first tashahhud and thinks that if he has stood up for the extra rak‘ah and started to recite, it is haram for him to sit back down. This is a mistaken notion, because one should never continue with the additional rak‘ah. As soon as he realises, he must sit back down, in order to put a stop to this additional rak‘ah, because if he continues with it when he is aware of it, he will be adding something to the prayer deliberately, which is not permissible and renders the prayer invalid.
The phrase “must also recite the tashahhud if he did not do so previously” means: when he realises that he is doing something additional and sits back down, then he should recite the tashahhud, unless he recited it before he stood up for this additional rak‘ah. But this begs the question: is it possible that anyone could get up to do another rak‘ah after reciting the (final) tashahhud?
The answer is yes, it is possible, if he recited the tashahhud in the fourth rak‘ah, then he got confused and thought that it was the second rak‘ah, so he got up for what he thought was the third rak‘ah, then after standing up he realized that this was a fifth rak‘ah and that the tashahhud he had recited was the last tashahhud. The words “he should prostrate and say the salaam” clearly mean that he should prostrate (the two prostrations of forgetfulness) before the salaam. This is the adopted view, because they do not think that the two prostrations of forgetfulness should be done after the salaam, except in the case where one has said the salaam before completing the prayer. But in all other cases, it should be done before the salaam. However the more correct view, which was favoured by Shaykh al-Islam Ibn Taymiyah, is that two the prostrations of forgetfulness for adding something to the prayer should be done after the salaam in all cases.
Question: if a person gets up for a third rak‘ah in Fajr prayer, what should he do?
Answer: he should sit back down even if that is after bowing; he should sit back down, recite the tashahhud and say the salaam, then do the two prostrations of forgetfulness and say the salaam again, according to the more correct view, which is that in this case the prostration should come after the salaam.
End quote from ash-Sharh al-Mumti‘, 3/342-343
But if the worshipper did not realise that he had done something extra until after he had finished the prayer, then in this case his prayer is valid and the two prostrations of forgetfulness should come after the salaam, for the extra actions that occurred during the prayer.
In Majmoo‘ Fataawa Ibn ‘Uthaymeen (14/31) there is a question about a man who prayed Zuhr with five rak‘ahs, and did not realise until he was reciting the tashahhud. What is the ruling in this case?
He replied: If a person does an extra rak‘ah in his prayer, and does not realize until he has finished the rak‘ah, then he must do the two prostrations of forgetfulness; these two prostrations should come after saying the salaam at the end of the prayer. The evidence for that is the fact that when the Messenger (blessings and peace of Allah be upon him) offered a prayer with five rak‘ahs, and they told him about that after the salaam, he prostrated twice. As he prostrated after the salaam, and did not say that the two prostrations for having added this rak‘ah should come before the salaam, we know that the two prostrations for having added something to the prayer should come after the salaam. This is also supported by the hadeeth of Dhu’l-Yadayn. End quote.
And Allah knows best.