What should she do as she prayed against herself asking that she not be successful and her du‘aa’ was accompanied by an oath?

Praise be to Allah.

Firstly:

 The secret habit is an abhorrent and haraam action on the part of both males and females. We have discussed the ruling on that and how to deal with it in the answers to questions no. 329 and 101539. 

Secondly: 

What you have done of praying against yourself and asking that you not be helped to succeed in obtaining high grades is something that was not appropriate, because it was praying for something bad for oneself. Falling behind in your studies is not a good thing; rather it would lead to serious negative consequences. The Prophet (blessings and peace of Allah be upon him) forbade the Muslim to pray against himself. It was narrated that Umm Salamah said: The Messenger of Allah (blessings and peace of Allah be upon him) said: “Do not pray against yourselves except for good things, for the angels says ameen to whatever you say.”

Narrated by Muslim, 920 

Muslim also narrated (3014) from Jaabir (may Allah be pleased with him) that the Prophet (blessings and peace of Allah be upon him) said: “Do not pray against yourselves, do not pray against your children, and do not pray against your wealth, lest that coincide with an hour when Allah is asked for something and He responds to you.” 

Thirdly: 

If a person swears to do an act of disobedience then he does it, what he is required to do is offer expiation for the oath he swore. This expiation is not expiation for the sin he committed; rather it is expiation for the oath that he swore. As for the sin, what he has to do is repent sincerely from it. Part of his repentance is regretting what he did in the past and resolving not to do it again. 

See the answer to question no. 103424 

Thus, if a person swears against himself with the aim of encouraging himself to do something or preventing himself from doing something, as appears to be the case in your question, then it comes under the ruling on oaths, and if he broke it then he must offer expiation for it. 

Shaykh Khaalid al-Mushayqih (may Allah preserve him) was asked: 

What is the ruling on one who says, “Insha Allah I should die if I do such and such?” or he said “May such and such befall me if I do this thing?” when he knows that he wants to go back to it and he said that in a moment of anger? Is this regarded as being the same as an oath? May Allah reward you with good. 

He replied: 

Firstly, it is forbidden for a person to pray against himself… 

Secondly, if a person prayed against himself in this manner, intending to encourage himself to do something or prevent himself from doing something, or to confirm something or to say that something was a lie, then the oath was broken, then he must offer expiation for breaking an oath (kafaarat yameen). For example, if he said, “If I do not go, may I fall ill or die,” then he did not go. In that case he has to offer expiation for breaking an oath.

End quote from Fataawa ash-Shaykh, 2/145. 

And Allah knows best.