He is not sure whether he said the opening takbeer or not

Praise be to Allah.

The opening takbeer is one of the pillars or essential parts of the prayer, which is not waived because of forgetting or not knowing about it, and it cannot be substituted with anything else. If a person remembers during his prayer that he forgot to say the opening takbeer, or he is not sure whether he said it, he has to start the prayer over again. If his doubt about that comes after the prayer, then it does not matter, because doubts after completing an act of worship do not affect it. 

Based on this, if the doubt of this worshipper who is not sure whether he said the opening takbeer or not came during the prayer, he has to stop praying and start the prayer over again. If the doubt came after the prayer, then he does not have to do anything and his prayer is valid. 

The evidence that the opening takbeer is an essential part of the prayer is the report narrated by al-Bukhaari (757) and Muslim (397) from the hadeeth of the one who prayed badly. The Prophet (peace and blessings of Allaah be upon him) said to him: "Go back and pray, for you have not prayed.” Then he said: “When you stand up to pray, say takbeer then recite whatever you know of the Qur'aan. Then bow until you are at ease in bowing, then stand up until you are standing up straight. Then prostrate until you are at ease in prostration, then sit up until you are sitting up straight, and do that throughout your prayer." 

And Abu Dawood (61), al-Tirmidhi (3) and Ibn Maajah (275) narrated that 'Ali (may Allaah be pleased with him) said: The Messenger of Allaah (peace and blessings of Allaah be upon him) said: “The key to prayer is purification, its opening is to say Allaahu akbar and its closing is to say al-salaamu’ alaykum.” Classed as saheeh by al-Albaani in Saheeh Abi Dawood. 

Al-Nawawi (may Allaah have mercy on him) said: 

The opening takbeer is one of the pillars (essential parts) of prayer without which the prayer is not valid. This is our view and the view of Maalik, Ahmad and the majority of earlier and later scholars. Ibn al-Mundhir and our companions narrated that al-Zuhri said: The prayer begins simply with the intention, without any takbeer. But Ibn al-Mundhir said: No one said that except al-Zuhri. 

Then he said: We have stated that the prayer is not valid without the opening takbeer. If the imam or the person praying behind him omits it by mistake or deliberately, his prayer has not begun, and the takbeer for bowing does not make up for it, neither does any other takbeer. This is our view and is the view of Abu Haneefah, Maalik, Ahmad, Dawood and the majority. 

End quote from al-Majmoo', 3/250 

Whoever is not sure whether he has said the opening takbeer or not should assume that he has not said it. Shaykh Ibn Baaz said: 

If a person forgets to say the opening takbeer or is not sure about that, then he has to say takbeer straight away, and follow the imam in whatever comes after he said this takbeer. If he said the takbeer after the imam has done the first rak'ah, then he should regard himself as having missed the first rak'ah, which he should make up after the imam says the salaam. If he said the takbeer in the third rak'ah, then he should regard himself as having missed two rak'ahs, which he should make up after the imam says the salaam. This applies if he is not affected by waswasah (whispers from the Shaytaan). If he is affected by waswasah, then he should regard himself as having said the opening takbeer at the beginning of the prayer, and not make up anything, in order to annoy the Shaytaan and combat his whispers. Praise be to Allaah. End quote. 

Fataawa al-Shaykh Ibn Baaz, 11/275 

If the doubt comes after finishing the prayer, then he should not pay any attention to it. 

Ibn Rajab (may Allaah have mercy on him) said: If a person is uncertain after finishing a prayer or any other act of worship as to whether he omitted a essential part of it, he should not pay any attention to that doubt. 

End quote from al-Qawaa'id, p. 340 

See also the answer to question no. 211 

And Allaah knows best.