If a person hears the call to prayer but does not respond, his prayer (elsewhere and not in the mosque) does not count, unless he has a valid excuse. What is meant is that his prayer is imperfect and he is a sinner because he kept away from the prayer in congregation. Ibn Mas’ood (may Allaah be pleased with him) said: “No one keeps away from the prayer (in congregation) except a hypocrite who is known for his hypocrisy.” (Narrated by Muslim, 654) Not responding to the muezzin implies weakness of faith, lack of religious commitment, not caring about reward and forsaking the houses of Allaah.
With regard to the ruling on one who prays Jumu’ah only, some of the scholars are of the view that one who prays Jumu’ah only is a kaafir, because he comes under the same ruling as one who does not pray at all, because he prays only one of the thirty-five prayers required each week, so he is like one who does not pray at all. Among those who expressed this view were Shaykh ‘Abd al-‘Azeez ibn Baaz and Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on them). Some of the scholars said that he is not a kaafir but he is committing a great sin that is worse than riba (usury), adultery, stealing, drinking alcohol etc.