The history of Iwamura Castle dates back to the Kamakura Period when Kato Kagekado, a retainer of Minamoto no Yoritomo founded the castle. However, it is lkely that Kato stayed with the Minamoto in Kamakura and the first person to actually go to Iwamura and build the castle was his son Toyama Kagetomo who came in 1195. The Toyama continued to rule until the Sengoku Period.

Toyama Kageto, the last Toyama lord, ruled Iwamura as a retainer of of Oda Nobunaga. Iwamura Castle was besieged by Takeda Shingen in 1570, but with the help of Nobunaga it was successfully defended. Toyama Kageto suddenly died of illnes in 1571 leaving only his young adopted son (Nobunga's fifth son) and his wife (Nobunaga's aunt).

Akiyama Nobutomo, under the command of Takeda Shingen, took advantage of this weakness and attacked again in 1572. This time Kageto's widow negotiated a surrender of the castle. She agreed to wed Nobutomo for her own protection and sent her adopted son (Nobunaga's son) off to be cared for by Takeda Shingen.

After the crushing defeat to Oda and Tokugawa forces at Nagashino in 1575, the Takeda power was severely cripppled. Oda took this opportunity to take back Iwamura Castle. After a 5 month siege, Nobutomo negotiated a surrender that guaranteed their lives would be spared. Nobunaga however did not honor their truce and executed Nobutomo and his wife (Nobunaga's aunt) at the Nagara River by upside-down hanging. This cruel punishment, normally dealt to traitors, was a sign of Nobunaga's anger at his son being sent to Takeda.

Kawajiri Hidetaka, who first ruled Iwamura Castle under Nobunaga rebuilt much of the castle to it's modern structure. The castle was ruled by various lords until it was dismantled in 1873 under the castle abolishment law. Iwamura Castle is famous as one of the three best mountaintop castles and is also one of the top 100.