If the imam forgets something during the prayer, then the Prophet (peace and blessings of Allaah be upon him) commanded the members of the congregation to remind him. He said: “I am only human like you, and I forget as you forget, so if I forget then remind me.” Narrated by al-Bukhaari, 401. So the congregation in the mosque should have said tasbeeh (“Subhaan Allaah”) so that the imam would notice and would do the second prostration.
Both the first and second prostrations are pillars or essential parts of the prayer, without which the prayer is not valid. Whoever deliberately omits one or both of them has sinned, and his prayer is invalid. Whoever forgets one or both of them has to do what he forgot when he remembers it, whether he is leading a congregation, following an imam or praying on his own. Whoever does not do that, his prayer is invalid.
Shaykh Muhammad ibn Saalih al-‘Uthaymeen (may Allaah have mercy on him) said:
The pillars or essential parts (of the prayer) are obligatory and more important than the waajibat (obligatory parts of prayer), but they differ from then in that the pillars are not waived if one forgets them whereas the waajibaat are waived in that case and may be made up for by doing the prostration of forgetfulness (sujood al-sahw), unlike the pillars. Hence whoever forgets a pillar, his prayer is not valid without it.
Al-Sharh al-Mumti’, 3/315
And he said:
The evidence that the pillars cannot be made up for by the prostration of forgetfulness is the fact that when the Prophet (peace and blessings of Allaah be upon him) said the salaam after praying two rak'ahs of Zuhr or ‘Asr, he completed it and did what he had omitted, and did the prostration of forgetfulness. This indicates that the pillars of prayer cannot be waived by doing the prostration of forgetfulness, and they must be done.
Al-Sharh al-Mumti’, 3/323
With regard to what your imam did by repeating the final rak’ah in full after being reminded, this is in accordance with one of the two scholarly opinions on this matter, which is that the one who omits a pillar or essential part of the final rak’ah and does not realize it until after saying the tasleem should do the rak’ah in full. This is the view of Imam Ahmad (may Allaah have mercy on him). See al-Mughni, 1/658.
This view was favoured by Shaykh Ibn Baaz (may Allaah have mercy on him), who was asked about an imam who forget the final prostration in ‘Asr prayer. He stood up and offered a complete rak’ah, then he said the tashahhud and the tasleem, then he did the prostration of forgetfulness. He said: This is what is prescribed in Islam. If the imam forgot a prostration and said the tasleem, then he remembered it or it was pointed out to him, he should stand up and do a full rak’ah, then say the salaam, then do the prostration of forgetfulness after saying the salaam. This is better. The same ruling applies to one who is praying alone. But if he does the prostration of forgetfulness before the salaam, there is nothing wrong with that, but doing it after the salaam is preferable.
Majmoo’ Fataawa Ibn Baaz, 11/277
The second view concerning this matter is that he does not have to do the rak'ah in full, rather he should do the pillar that he forgot and whatever comes after it. This is the view of Imam al-Shaafa’i (may Allaah have mercy on him). See al-Majmoo’, 4/33. This view was favoured by Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him). See Sharh al-Mumti’, 3/374.
The point is that the prayer of the imam, and the prayer of those who completed the prayer with him, is valid.
But for those who did not complete the prayer and did not do the prostration that was omitted, their prayer is not valid, and they have to repeat it.
And Allaah knows best.