If most of the man’s wealth is haraam, then it is permissible to accept his invitation.
The Prophet (peace and blessings of Allaah be upon him) accepted invitations to meals with the Jews, even though Allaah had described them as consuming riba (usury) and consuming the people’s wealth unlawfully. One of the salaf said concerning such cases: It is a gain for you and his sin is against him.
It is also permissible for you not to accept his invitation as a rebuke to him for his haraam earnings. This is preferable if that will make him give up the evil that he is doing.
And Allaah knows best.