I found it weird that this site is one of the least visited UNESCO sites in the world, as remote as, say, Ogasawara. But after I came to Hiroshima station tourist information and asked them how to visit Iwami Ginzan, they immediately told me the way. I may confirm Frederic Dawson’s description that this site is easily accessible from Hiroshima for one day trip. The buses (as of April 2013) run from Hiroshima train station at 10:00 (12:40 at Iwami) and 14:55 (17:35), where buses back from Iwami are at 7:52 and 16:22. Although I found the bus price a bit too high (6000 JPY for roundtrip), Iwami Ginzan is definitely worth visiting.
I started my visit from World Heritage Center (the only place where foreigners are handicapped in ticket price), where I was provided with tablet explaining the exhibition in English. This is a good start to understand the history of the place and familiarize with miners’ day life. Then we moved by local bus to Omori town to see Ryugenji Mabu Mine Shaft, the main mine shaft in the area. I was surprised that the mine shaft could be easily visited even with baby-stroller (I travelled there with my wife and infant son). Then we moved back to Omori walking in picturesque landscape between traditional Japanese houses. In Omori I strongly recommend visiting Rakan-Ji Temple with hundreds small Buddha monuments (each one different).
Although we were the only non-Japanese tourists that day, there were lots of Japanese excursions there. Maybe this is the best recommendation that Iwami Ginzan is worth visiting.
Tip: if you want to visit the site in one-day trip from Hiroshima, you have only 3.5 hours in Omori. The best way to visit the site in this short time is renting a bike, available near the bus stop.