Mori Takamasa built Saiki Castle in 1606. The Wakamiya Shrine (originally built in 1180) was moved from the top of the mountain in 1604 to accomodate the building of the castle.

For a fiefdom of 20,000 koku, Mori's castle was rather large. Centered around a 3 story main keep it was fortified by 5 two story yagura, a one story yagura, and 7 gates. The main keep burned down in 1617 not long after it was completed. Besides one gate, the only remnants of the castle are some stone walls on the mountain. 12 Generations of Mori ruled from Saiki Castle until the coming of the Meiji Period.

This picture is of the stairs leading to the top of the foundation the Honmaru sat on (the stairs face east, and lead up to the west)