When a Muslim gives charity and hands it over to the poor person, it is not permissible for him to ask for it to be returned to him or take it back, if the one to whom he gave it wants to return it. Whatever a person has given in charity for the sake of Allah, it is not permissible for him to take it back.
It was narrated from Ibn `Abbas (may Allah be pleased with him) that the Prophet (blessings and peace of Allah be upon him) said: “The likeness of the one who takes back his charity is that of a dog which vomits then goes back to its vomit and eats it.” Narrated by Muslim, 1622.
And it was narrated from `Abdullah ibn `Umar (may Allah be pleased with him) that `Umar ibn al-Khattab gave a horse in charity for the sake of Allah, then he found it being offered for sale. He wanted to buy it, then he went to the Prophet (blessings and peace of Allah be upon him) to consult him about that, and he said: “Do not take back your charity.” Narrated by al-Bukhari, 1489.
Ibn Hajar (may Allah have mercy on him) said: As for charity, the scholars unanimously agreed that it is not permissible to take it back after the poor person has taken possession of it. End quote from Fat-h al-Bari, 2/235.
Ibn Qudamah (may Allah have mercy on him) said: It is not permissible for the giver to take back his charity, according to the view of all the scholars.
End quote from Al-Mughni, 8/279.
II.
The only situation in which it is permissible for a person to agree to take back charity from the one to whom it was given is if it is obligatory zakah in a case where it was given to someone who was not entitled to receive it, and he tells the giver about that. In this case it is permissible for him to take it back and give it directly to someone who is entitled to receive it, and he should not take anything of it for himself.
Al-Bahuti (may Allah have mercy on him) said: If someone gives voluntary charity to a rich person, not knowing that he is rich, he should not take it back, because the purpose is to attain reward, and he will not miss out on that. This is in contrast to zakah, if he gives it to a disbeliever and the like, because the purpose of giving is to fulfil the duty of giving zakah, and if he gives it to a disbeliever, he has not fulfilled his duty, so he may take it back [and give it to someone who is entitled to receive it].
End quote from Kashshaf al-Qina`, 2/295.
This last scenario is different from what you asked about; we have mentioned it in order to give a full picture of the issue.
Based on that, what you did of agreeing to take back the charity is not permissible. If he has transferred it to you, then you must return it to him, and if he refuses to take it, then you must give it in charity as soon as you receive it, and not benefit from it in any way.
And Allah knows best.