Ogasawara Tadazane (former lord of Matsumoto), moved into the area in 1617. In 1619, under the orders of Tokugawa Hidetada he built Akashi Castle in just one year for the purposes of watching over the Western lords and building up their defenses in the region. He accomplished building this castle in so little time mainly because he used materials from castles in the area that were ordered to be dismantled under the one castle per country law of 1615.

The castle deftly makes use of the natural terrain in a 3 tiered castle compound. Ogasawara's father-in-law Honda Tadamasa, who also directed the construction of Himeji Castle, assisted with the construction of Akashi castle. Even though they build a large foundation for a large donjon, no donjon was ever built. In it's plac ethe honmaru had 4 large 3 story yagura, two of which are still standing today.

Eventually, Ogasawara Tadazane was moved to Kokura Castle and the lordship of Akashi Castle changed hands several times until it was taken over by Matsudaira Naoakira in 1682. The Matsudaira continued to ruled until the coming of the Meiji Restoration.