It is as if you are saying that there is no one in the neighbourhood who can take the place of this man, and the people need him. So his teaching this group of people is an individual obligation (fard ‘ayn) for him. If he moves, we think that:
If he is in the same city, then he can come to the mosque where he was previously, even if it is on the other side of the city.
If he moves to another city, then if it is for an essential reason, there is no sin on him, such as if he travels to earn a living and he does not have enough to live on. In such a case there is no sin on him, and the community has to look for another person to take his place, or they should support this man so that he will be free to teach them.