Praise be to Allah.
Your saying to your wife, It's over between us, go to your family, is a kind of implicit divorce which does not count as such unless it is intended as such. If you do not know what is your intention was or you have forgotten, the basic principle is that no divorce has taken place.
With regard to your divorcing your wife on the second and third occasions, the ruling is as you were advised by those muftis.
You should beware of using the word of divorce lightly, because of the consequences that result from that, which may adversely affect the husband, his wife and his children. It is clear from the question that you take this serious matter lightly, because you divorced her one day and took her back, then the next day you divorced her again. This is a transgression of the sacred limits of Allaah and a person like this deserves to have his talaaq counted everytime, as ‘Umar (may Allaah be pleased with him) did when the people began to take the threefold divorce lightly.
And Allaah knows best.