Your question covers several issues:
1 – This action that you and your friends are doing, calling people to Allaah, is something that is to be appreciated at a time when few people are doing such work. We give you the glad tidings of the words of the Prophet (peace and blessings of Allaah be upon him): “Whoever points others towards doing good will have a reward like that of the one who does it.” Narrated by Muslim, 133.
If by doing this work you intend to visit others for the sake of Allaah, then this is how it will be, in sha Allah.
2 – Do not let the blame of anyone put you off from doing this work, so long as you and your brothers are calling people with proper knowledge and insight, as appears to be the case from what you say.
3 – If the people you mention are known to have sound ‘aqeedah and knowledge, and are known for their good conduct and attitude, then there is nothing wrong with cooperating with them in da’wah and going out with them, whether that is for a short time or a long time, because Allaah says (interpretation of the meaning):
“Help you one another in Al‑Birr and At‑Taqwa (virtue, righteousness and piety)”
“And who is better in speech than he who [says: ‘My Lord is Allaah (believes in His Oneness),’ and then stands firm (acts upon His Order), and] invites (men) to Allaah’s (Islamic Monotheism), and does righteous deeds, and says: ‘I am one of the Muslims’”
“Invite (mankind, O Muhammad) to the way of your Lord (i.e. Islam) with wisdom (i.e. with the Divine Revelation and the Qur’aan) and fair preaching, and argue with them in a way that is better”
And the Prophet (peace and blessings of Allaah be upon him) said to ‘Ali (may Allaah be pleased with him) when he sent him to Khaybar to call the Jews to Islam: “By Allaah, if Allaah were to guide one man through you, that would be better for you than having red camels [the best kind].” Narrated by al-Bukhaari, 3701; Muslim, 2406
The Prophet (peace and blessings of Allaah be upon him) sent daa’iyahs (callers) to Yemen and to many of the Arab tribes.
Many members of this group with whom you are going out are lacking in Islamic knowledge, and they may hold some false beliefs, but this should not prevent you from going out with them and calling people to Allaah; rather you should go out with them and explain the truth to them, and teach them what they do not know of the Sunnah, and advise them to give up the innovations that they do. Perhaps Allaah will benefit them through you.
And Allaah knows best.