With regard to the hadeeth (narration) mentioned in the question and other marfoo’ hadeeths (reports attributed to the prophet) about the good deeds of the innovator not being accepted, they are da’eef or munkar ahadeeth (i.e., weak) and are not saheeh (authentic). A detailed discussion follows.
The first hadeeth: It was narrated from Hudhayfah (may Allah be pleased with him) that the Messenger of Allah (peace and blessings of Allah be upon him) said: “Allah will not accept any fasting, prayer, charity, Hajj, ‘Umrah (major and minorpilgrimage), jihad (physical struggle against oppression), or any other obligatory or nafil (voluntary) action from a person who follows innovation (bid’ah). He comes out of Islam like a hair pulled out of dough.”
Narrated by Ibn Majah in al-Sunan (no. 49). He said: Dawood ibn Sulayman al-‘Askari narrated to us, Muhammad ibn ‘Ali Abu Hashim ibn Abi Khaddash narrated to us, Muhammad ibn Muhsin narrated to us, from Ibraheem ibn Abi ‘Ablah, from ‘Abd-Allah ibn al-Daylami, from Hudhayfah (may Allah be pleased with him).
Al-Shaykh al-Albani said in al-Silsilah al-Da’eefah (no. 1493):
It is mawdoo’ (fabricated); the fault lies with Ibn Muhsin who is a liar as Ibn Ma’een and Abu Hatim said. Al-Hafiz said in al-Taqreeb: They regarded him as a liar but al-Boosayri was more lenient concerning him in al-Zawaid (1/10). This is a da’eef (weak) hadeeth; its isnad (chain of transmission) includes Muhammad ibn Muhsin, and they are agreed that he is da’eef. The reason for this lenient view is that the narrator is unanimously regarded as da’eef but not as a liar. In that case, mentioning the consensus concerning a narrator without mentioning the reason does not give a proper idea about the narrator.
The second hadeeth: It was narrated that ‘Abd-Allah ibn ‘Abbas (may Allah be pleased with him) said: The Messenger of Allah (peace and blessings of Allah be upon him) said: “Allah refuses to accept the good deeds of one who follows innovation until he gives up that innovation.”
Narrated by Ibn Abi Hatim in al-Jarh wa’l-Ta’deel (9/439); Ibn Majah in al-Sunan (no. 50), Abu’l-Fadl al-Muqri’ in Ahadeeth fi Dhamm al-Kalam wa Ahlihi (3/111), Ibn Abi ‘Asim in al-Sunnah (no. 32), al-Khateeb in Tareekh Baghdad (13/185), via Ibn al-Jawzi in al-‘Ilal al-Mutanahiyah (1/144). All of them narrated via Bishr ibn Mansoor al-Khayyat from Abu Zayd from Abu’l-Mugheerah from ‘Abd-Allah ibn ‘Abbas (may Allah be pleased with him).
Ibn Abi Hatim said, after narrating it: Abu Zu’rah was asked about them – meaning Abu Zayd and Abu’l-Mugheerah – and he said: I do not know them, and I do not know Bishr ibn Mansoor from whom al-Ashajj narrated it. End quote.
Ibn al-Jawzi said in al-‘Ilal al-Mutanahiyah (1/145):
This is not a saheeh hadeeth (authentic report) from the Messenger of Allah (peace and blessings of Allah be upon him); its isnad includes majhool (unknown) narrators. End quote.
Al-Shaykh al-Albani said in al-Silsilah al-Da’eefah (1/1492):
It is munkar (denounced). This is a da’eef (weak) isnad, full of majhool (unknown) narrators. Abu Zur’ah said: I do not know Abu Zayd or his Shaykh, or Bishr. Al-Dhahabi said concerning the first of them: He is unknown. And he said concerning the other two: I do not know who they are. And al-Boosayri agreed with him in al-Zawaid (1/11). End quote.
But some people may be confused by what is mentioned in the hadeeth of ‘Ali ibn Abi Talib (may Allah be pleased with him), according to which the Prophet (peace and blessings of Allah be upon him) said concerning one who introduces innovation in Madeenah: “Allah will not accept any obligatory or nafil act of worship from him.” Narrated by al-Bukhari (7300) and Muslim (1370).
Something similar was narrated from a number of the Tabi’een, such as al-Hasan al-Basri, from whom it was narrated that he said: “Allah does not accept from the one who follows innovation any fast, prayer, Hajj or ‘Umrah, until he gives it up.” End quote.
Narrated by al-Ajurri in al-Sharee’ah (64); Abu Shamah in al-Ba’ith ‘ala Inkar al-Bida’ wa’l-Hawadith (p. 16); and elsewhere. A similar report was narrated from al-Awza’i, as it says in al-Bida’ wa’l-Nahi ‘anha by Ibn Waddah (27). It was narrated that al-Fudayl ibn ‘Iyad said: “No good deed is taken up to Allah for the one who follows innovation.” Narrated by al-Lalkai in Sharh Usool I’tiqad Ahl al-Sunnah (1/139).
Al-Shatibi remarked in his commentary on such reports. He said in al-I’tisam (1/108-112):
Either it means that his good deeds are not accepted from him at all, regardless of whether they are in accordance with the Sunnah (prophetic teachings) or not, or it means that the deeds in which he follows innovation in particular will not be accepted from him, to the exclusion of those in which he does not follow innovation.
With regard to the first scenario, it may be understood in three ways:
1 – it may be understood according to the apparent meaning, i.e., that for every innovator, no matter what his innovation is, none of his good deeds will be accepted when he follows that innovation, whether any particular deed comes under the heading of that bid’ah or not. This is very harsh on those who introduce innovations into Islam.
2 – it may be understood as referring to a type of bid’ah that may form the basis for all other deeds, such as if he holds the view that ahad reports are to be rejected, as most deeds prescribed in Islam are based on ahad reports.
3 – The third way of understanding it is that bid’ah in some acts of worship or other deeds may lead the one who follows that particular bid’ah to form some understanding that makes his belief in sharee’ah (Islamic teachings in general) very weak, which invalidates all his good deeds.
With regard to the second interpretation, it means that specifically their deeds in which they follow innovation are not accepted, and that is possible. This is indicated by the hadeeth: “Every action that is not in accordance with this matter of ours will be rejected.” End quote.
But the correct view concerning this issue is what was mentioned by Dr. Ibraheem al-Ruhayli in his book Mawqif Ahl al-Sunnah min Ahl al-Ahwa wa’l-Bida’ (1/292-293), where he says:
What is indicated by the apparent meaning of the texts and the words of the Salaf (pious predecessors) is that for the one who follows innovation, Allah will not accept his deeds, and it may be interpreted in the following ways:
1 – The words may be taken at face value, and that what is meant is that all the deeds of the innovator are rejected, those in which he follows innovation and those in which he does not follow innovation. This applies to the innovator who is a kafir (disbeliever), and no one else.
2 – What is meant is that only the innovated actions are rejected, whether they are pure innovation or they were prescribed in Islam but some innovations were introduced into them and spoiled them.
3 – The innovations may cause the reward for the deed to be erased by way of punishment, until it is as if it is not accepted.
4 – The texts may be understood as a rebuke for innovation and as a deterrent.
The reason why we need to find a proper understanding for these texts and the words of the salaf here is that the apparent meaning of the texts seems to contradict the basic principles of sharee’ah, which indicate that the Muslim’s deeds are accepted if they fulfil the conditions of sincerity towards Allah and following the Sunnah, regardless of any innovations, sins, etc., that the person may be doing in other deeds, because they have no effect on the acceptability of this particular good deed. End quote.
With regard to the third interpretation here, it should be understood as referring only to any bid’ah that a shar’i (religious) text indicates that this particular bid’ah erases the good deeds of the one who commits it, and not every single bid’ah falls into this category.
The scholars commented on this hadeeth and explained what is meant by the words of the Prophet (peace and blessings of Allah be upon him): “Allah will not accept any obligatory or nafil act of worship from him.”
Al-Qadi said: And it was said that what is meant is that his obligatory and nafil acts of worship will not be accepted in the sense of Allah being pleased with them, even though they may be acceptable in the sense that he will be rewarded for them. And it was said that acceptance here refers to being acceptable as expiation for sin.
Sharh al-Nawawi ‘ala Muslim (9/141).
To sum up: It is permissible to pray behind one who follows innovation so long as his bid’ah is not one that constitutes blatant kufr (disbelief). Allah will call people to account on the basis of their deeds.
“So whosoever does good equal to the weight of an atom (or a small ant) shall see it. And whosoever does evil equal to the weight of an atom (or a small ant) shall see it.”
And Allah knows best.