He started to pray, then he wasn’t sure whether he formed the intention to pray or not; does he have to repeat it?

Praise be to Allah.

The intention (niyyah) is one of the conditions of prayer being valid. If a person starts to pray without having formed the intention, then his prayer is not valid.

Ibn Qudaamah (may Allah have mercy on him) said: We do not know of any difference of opinion among the scholars concerning the necessity of forming the intention to pray; prayer cannot be done without it. The basic principle concerning that is the words of Allah, may He be exalted (interpretation of the meaning): “And they were not commanded except to worship Allah, [being] sincere to Him in religion” [al-Bayyinah 98:5]. Sincerity is an action of the heart, and this refers to the intention, seeking Allah alone and no other. And the Prophet (blessings and peace of Allah be upon him) said: “Actions are but by intentions, and each one will have but that which he intended.” And the seat of the intention is the heart.

End quote from al-Mughni, 1/287

But it is very unlikely that this thing (praying without intention) could happen. The intention could accompany the opening takbeer (when one says “Allahu akbar” to start the prayer), or it may be shortly before it.

When the Muslim hears the adhaan, then he gets up, does wudoo’ and goes to the mosque, and sits down waiting for the iqaamah (call immediately preceding the prayer), then when the iqaamah is given he gets up and stands in the row and so on, then he has definitely formed the intention to offer the prayer for which the call was given and the iqaamah was recited, so how could he have any doubts about his intention after that?

Based on that, what appears to be the case is that the doubt you are feeling comes under the heading of whispers from the Shaytaan (waswaas) who wants to distract you from your prayer and prevent you from pondering and focusing properly. So you should not pay any attention to it.

For more information on dealing with waswaas, please see the answer to question no. 100268 and 62839.

It should be noted that if there is a great deal of doubt, you should not pay any attention to it, because it is whispers from the Shaytaan.

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said: If a person experiences so many doubts that he can hardly do anything without doubting it – if he does wudoo’ he doubts it, and if he prays he doubts it, and if he fasts he doubts it – then this also does not mean anything, because this is a kind of sickness. What we discussed above applies to the healthy person who is free of sickness; if a person is experiencing a great deal of doubt like this, then it is to be assumed that his mind is not stable, so no attention should be paid to these doubts.

End quote from ash-Sharh al-Mumti‘, 3/379

And Allah knows best.