The original Kameyama Castle was founded by Seki Sanetada in 1264 to the West of the present Kameyama Castle. When Seki Kazumasa was relocated to Shirakawa in 1590, Okamoto Munenori was given control of the castle. Okamoto moved the castle ot the Southeast and constructed the all the main structures.

In 1632 when the Tokugawa Government issued a declaration that the donjon of Kameyama Castle in Tanba be rebuilt, the Horio Tadazane confused it with this Kameyama Castle (in Ise) and destroyed the donjon and started work on the stone walls before the mistake was discovered. Evenso, the Tokugawa government would not them permission to rebuild the donjon at (Ise) Kameyama Castle, so it was never rebuilt. Some also say, that this was all part an elaborate plan by the Tokugawa government.

In 1636, Honda Toshitsugu moved into the castle and built a tamon yagura on the site of the original donjon that was mistakenly destroyed. He also renamed a three story yagura in the honmaru and used it as a donjon.