It may be cost. If a medal weighs several (troy) ounces, it would contain thousands of dollars of gold, compared to the tens/hundreds for silver, and even less for bronze. Also, as mentioned, gold is very soft. I would guess that the silver used for the interior of the gold medal is sterling silver, much stronger than pure silver, which is not much harder than gold.

An acquaintance had won a bronze medal at Lillehamar (short track relay), and that medal was actually mostly made of local granite, encased in a small metal frame plated with the approopriate material.

<link>