Praise be to Allah.
If the Muslim realizes that the kaafir has said to him al-saam alaykum, which means may death be upon you, he should respond in kind by saying wa alaykum (and also upon you).
If he realizes beyond a doubt that he has greeted him with the greeting of Islam (al-salaamu alaykum), Ibn al-Qayyim (may Allaah have mercy on him) said: if the listener realized that the Dhimmi [Jew or Christian] has said salaam alaykum and he is sure of that, should he say wa alayk al-salaam or just wa alayk?
According to the evidence and principles of shareeah, he should say wa alayk al-salaam, because this is more fair, and Allaah commands us to be just and to treat others well.
(Ahkaam Ahl al-Dhimmah, 1/425, 426).
Shaykh Muhammad al-Saalih ibn Uthaymeen (may Allaah preserve him) said:
These people who have come to us from the east and the west and are not Muslims, it is not permissible for us to initiate the greeting of salaam with them, because the Prophet (peace and blessings of Allaah be upon him) said: Do not initiate the greeting of salaam with the Jews and Christians. (Narrated by Muslim in his Saheeh).
If they greet us, then we can respond in a manner similar to that in which they greet us, because Allaah says (interpretation of the meaning):
When you are greeted with a greeting, greet in return with what is better than it, or (at least) return it equally [al-Nisaa 4:86].
When they greet us with the greeting of Islam al-salaamu alaykum either of the following two cases applies:
Either they pronounce the laam (l-sound) clearly, and say al-salaam alaykum (peace be upon you), so we may say wa alaykum al-salaam or wa alaykum
Or they do not pronounce the laam clearly, and so they say al-saamu alaykum (death be upon you), so we should say wa alaykum only. This is because the Jews used to come to the Prophet (peace and blessings of Allaah be upon him) and greet him by saying al-saam alaykum, without pronouncing the laam. Al-saam means death; i.e., they were praying against the Prophet (peace and blessings of Allaah be upon him), praying that he would die. So the Prophet (peace and blessings of Allaah be upon him) commanded us to say to them wa alaykum.
Thus, if they say al-saam alaykum, we should reply, wa alaykum, which means: and the same to you, may death be upon you. This is what is indicated by the Sunnah.
But if we initiate the greeting of salaam with them, our Prophet (peace and blessings of Allaah be upon him) forbade us to do this.
(Majmoo Fataawaa Ibn Uthaymeen, 2/97, 98).
And Allaah knows best.
For more information, see Question # 6583.