An oath is not binding unless it is sworn by one of the names of Allaah or by one of His attributes.
Swearing on the Qur’aan is swearing by the word of Allaah which is one of His attributes. As for swearing by the Mushaf, if what is meant is the words of Allaah contained therein, then it is an Islamically acceptable oath, but if what is meant is the paper and ink, then it is an oath by something other than Allah, which is shirk, as the Prophet (peace and blessings of Allaah be upon him) said: “Whoever swears by something other than Allaah has committed an act of kufr or shirk.” Narrated by al-Tirmidhi (1535) and Abu Dawood (3251); classed as saheeh by al-Albaani in Saheeh al-Tirmidhi.
Hence it is better not to swear by the Mushaf, because the Mushaf contains the words of Allaah, and it contains paper and ink.
As for putting one’s hand on the Mushaf or inside it, this is an innovated thing which some people do for emphasis to warn the one who is swearing the oath against lying.
Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) was asked about the ruling on swearing by the Mushaf.
He (may Allaah have mercy on him) replied: It is not permissible to make vows or swear oaths except by Allaah or by one of His attributes. If a person swears by Allaah, may He be glorified and exalted, then there is no need to bring the Mushaf for him to swear on it, because swearing on the Mushaf was not done at the time of the Prophet (peace and blessings of Allaah be upon him) or at the time of the Sahaabah; even after the Mushaf was compiled in book form they did not swear on it, rather a person would swear by Allaah without swearing on the Mushaf. End quote from Fataawa Noor ‘ala al-Darb.
It says in a statement of the Islamic Fiqh Council:
2 – It is not essential for the person who is swearing the oath to place his hand on the Mushaf, Torah or Gospel when swearing for his oath to be valid, but it is permissible if the judge deems it essential so as to warn the one who is swearing the oath against lying.
3 – It is not permissible for the Muslim to place his hand on the Torah or Gospel when swearing an oath, because these copies that are in circulation are distorted and that are not in the original form that was revealed to Moosa and ‘Eesa (peace be upon them), and because the sharee’ah with which Allaah sent His Prophet Muhammad (peace and blessings of Allaah be upon him) has abrogated the laws that came before it.
End quote from Fataawa Islamiyyah (3/463)
If a person swears an oath then he cannot fulfil that oath, he has to offer expiation. The expiation for breaking an oath (kafaarat yameen) is explained in the answer to question no. 45676.
And Allaah knows best.