If a man divorces his wife and this was the first or second talaaq, and the ‘iddah has not yet ended, he may take her back verbally, such as saying, “I take you back” or “I have kept you”; or it may be done by an action with the intention to take her back. If he has intercourse with her with the intention of taking her back, that also counts as taking her back.
The ‘iddah of a woman who menstruates is three menstrual cycles; when she becomes pure following the third menses and does ghusl, then her ‘iddah has ended.
The ‘iddah of a woman who does not menstruate because she is too young or has passed the age of menopause is three months.
The ‘iddah of a woman who is pregnant ends when her pregnancy ends.
Your saying “okay” when you saw that your wife had returned to your mother’s house when in fact you did not want to cancel the divorce, and your staying for one year without approaching her or looking at her – all of that indicates that you did not actually take her back with that word, because you did not intend to take her back thereby. Rather it was approval and agreement with her being present to serve your mother. In that case the ruling does not differ whether that happened during the ‘iddah period or afterwards, because this word does not indicate taking her back.
What happened of intercourse after that was a haraam action. You should have asked before that, especially since you felt anxious about what you had done.
Based on that, what you have to do is repent to Allah for having intercourse when you did not believe that you had taken her back. And you have to do a new marriage contract – fulfilling all the conditions of the wali (guardian), mahr and two witnesses – if both of you want that. And it is not permissible for you to come near her until the marriage contract has been done, because she is a non-mahram to you.
And Allah knows best.