The castle as it stands today was established in 1537 by Nobunaga's grandfather Oda Nobuyasu.

According to a Heian period book called "Enkishiki," at the site where Inuyama castle now stands, there was a shrine called Harigane Shrine (Harigane Jinja). This temple was later moved to Shirayamadaira to make way for the castle. The castle was not entirely constructed from scratch, but the main Tenshu (donjon; main keep) was actually moved here from a different castle known as Kanayama Castle.

After Hideyoshi took Odawara, he stationed Ishikawa Sadakiyo at Inuyama castle. Following the Battle of Sekigahara, Ishikawa was defeated and Matsudaira Tadayoshi (master of Kiyosu castle) stationed his retainer Ogasawara Yoshitsugu in Inuyama Castle. In 1616 Ogasawara was replaced by Naruse Masanari and his family have more or less owned the castle through the present day.

Inuyama-jo was famous for being the only privately owned castle in Japan. It had a few different owners early on, and was seized by the Meiji government who held it in their possession from 1872 to 1895. It suffered damage during an earthquake in 1891 and was returned to the Naruse family in 1895 under the condition that they repair the damages continue to keep up the castle into the future. It was recently sold to the city of Inuyama and will soon be taken over by Aichi Prefecture.