Praise be to Allaah.
It is not permissible firstly to initiate the greeting of salaam to a non-Muslim. The Prophet (peace and blessings of Allaah be upon him) said: Do not initiate the greeting of salaam to a Jew or a Christian(Narrated by Muslim, 2167).
If one of them says As-Saam alaykum meaning, may death be upon you or it is not clear whether they have said salaam, then we should respond by saying Wa alaykum (and upon you).
It was reported that Ibn Umar (may Allaah be pleased with him) said: the Messenger of Allaah (peace and blessings of Allaah be upon him) said: When the Jews greet you, they say Al-saam alaykum (may death be upon you), so respond by saying alayk (and also upon you).(Narrated by al-Bukhaari, 5902; Muslim, 2461).
If a non-Muslim greets us with the correct shari greeting (i.e., says Al-salaamu alaykum clearly), the scholars differed as to whether we have to return the greeting. The majority of scholars said that we do have to return the greeting, and this is the correct view.
Ibn al-Qayyim (may Allaah have mercy on him) said: they differed as to whether it is obligatory to return the greeting. The majority said that it is obligatory and this is the correct view. A group of scholars said that it is not obligatory to return their greeting just as it is not obligatory to return the greeting of those who follow bidah. But the correct view is the first one. The difference is that we are commanded to forsake the followers of bidah by way of rebuke and to warn others about them, which is not the case with the Ahl al-Dhimmah (Jews and Christians).
(Zaad al-Maaad, 2/425, 426)
The Muslim who is returning the greeting should respond in the manner prescribed by shareeah, giving a similar or better greeting, because of the general meaning of the Aayah (interpretation of the meaning):
When you are greeted with a greeting, greet in return with what is better than it, or (at least) return it equally. [al-Nisa 4:86]
Ibn al-Qayyim (may Allaah have mercy on him) said: if the person is certain that the Dhimmi (Jew or Christian) is clearly saying al-salaamu alaykum to him, and he has no doubts about that, should he say wa alayka al-salaam or shorten it to wa alayk? What is indicated by the evidence and principles of shareeah is that he should say wa alayka al-salaam, because this is more just, and Allaah commands us to be just and to treat others well this does not contradict any of the ahaadeeth on this topic at all, because the Prophet (peace and blessings of Allaah be upon him) commanded us to shorten the greeting to wa alayk because of the reason mentioned above, which is that they deliberately used to say al-saam alaykum instead of al-salaam alaykum, as indicated in the hadeeth narrated by Aaishah (may Allaah be pleased with her). He said, Do you not see that I say wa alaykum when they say al-saam alaykum? Then he said, If the People of the Book greet you with salaam, say, Wa alaykum.
Allaah says (interpretation of the meaning):
and when they come to you, they greet you with a greeting wherewith Allaah greets you not, and say within themselves, Why would Allâh punish us not for what we say? [al-Mujaadilah 58:8]
If this reason is not there, and the Jew or Christian says, Salaam alaykum wa rahmat-Allaah, then it is only fair to respond in kind.
(Ahkaam Ahl al-Dhimmah, 1/425, 426)
The hadeeth of Aaishah was narrated by al-Bukhaari (5901) and Muslim (2165)
See also: Majmoo Fataawa Ibn Uthaymeen, 2/97
And Allaah knows best.