Born as the son of a merchant on January 29, 1773 in Gernrode/Harz. He studied mathematics, physics and chemistry at the University of Halle/Saale. He completed his studies in these disciplines as well as in mechanics at the Mining Academy at Freiberg/Saxony. One of his teachers in Freiberg was the mineralogist and geognostician Abraham Gottlob Werner, who introduced Mohs to mineralogy and geognosy. In 1801, Mohs became a pit foreman at the Neudorf/Harz mine. He came to Vienna for the first time in 1802 in order to sort and annotate the important collection of the banker J.F. van der Nüll. (This collection was incorporated into the Imperial Mineralogical Collection in 1827). With his systematic classification of the realm of minerals, Mohs was in conflict with most of the other mineralogists of the time, due to his preference for physical attributes as the principles of classification (i.e., shape, cleavage, hardness and specific weight), as opposed to the chemical composition of the minerals. In 1812, Mohs was appointed Professor of Mineralogy at the Joanneum in Graz. During this time, he developed the Mohss Scale of Hardness. His stay in Graz was only of short duration and in 1817, he succeeded his teacher, A.G. Werner, at the Mining Academy in Freiberg. In 1826, Mohs was appointed Professor of Mineralogy at the University of Vienna. However, he held his inaugural lecture at the Imperial Mineralogical Cabinet where in 1834, he also held the post of custodian concurrently with his professorship.
In 1835, Mohs left the Mineralogical Cabinet and was commissioned with the establishment of a montanistic museum in Vienna, which he was in charge of during 1835-1839. Mohs died on September 29, 1839 during a trip to Italy. In 1888 he is granted a grave of honour in the Vienna Central Cemetery (Zentralfriedhof).
The mineral mohsite was named in his honour. Unfortunately, it was later recognised as a variety of the mineral crichtonite and therefore discredited as a species.