The castle currently known as Kasumiga-jo (Nihonmatsu-jo) actually encompasses two different castles. At the top of the hill are the remnants of the original castle built by the Hatakeyama in the Muromachi period (1333-1573) somewhere between 1394-1427. The exact date is unknown. At the base of the hill are the remnants of the castle built by the Niwa clan. Each of these castles experienced its own separate demise.

Hatakeyama Yoshitsugu captured Date Terumune, Date Masamune 's father, in 1585 and tried to take him back to Nihonmatsu. Upon hearing this, Masamune, who was out hunting, took off on his horse and caught up with Hatakeyama Yoshitsugu near the Abukuma River. He and his men shot the whole party dead and Terumune was killed in the struggle.

Hearing of the weakness of Nihonmatsu, a group of samurai from Hitachi (Ibaraki prefecture), including men from the Satake, Ashina, Iwaki and Ishikawa clans, set out to take Nihonmatsu for themselves. They clashed with the Date forces near present day Motomiya (between Koriyama and Nihonmatsu). This was known as the battle of Hitotoribashi. The battle ended in a standstill and Kasumiga-jo stood independent from both forces.

In April of the following year (1586) Date Masamune's forces once again laid seige to Kasumiga-jo. In July the son of Hatakeyama Yoshitsugu set fire to the castle and fled to Aizu. Thus the original castle at Nihonmatsu met its interesting demise.

The second castle was then built at the base of the hill and controlled by various families loyal to the Date, with the Niwa clan controlling it from 1643 until the Meiji Restoration.