In 1637, the Dutch employed the hill as a gun-position to bombard and take Elmina Castle from the Portuguese. The following year, the Dutch, seeking to protect the castle from the landward side, built on St. Jago hill, 33 meters above sea level, a redoubt or fortified quadrilateral earthwork with a tower and gate and a single-storied building within a courtyard all surrounded by an embankment. In the 1660's, the Dutch used local sandstone rock to build a permanent fort to replace the earthen fortification, which was then destroyed.

The stone fort, named Coenraadsburg, is unique and impressive as “the oldest purely military architecture of the Gold Coast". It had no commercial warehouses of any kind and its military fortifications were based on the design of baroque military architecture. Its salient features comprised two giant, strong landward bastions on the northeast and northwest sides for defending the castle from land attacks and two smaller seaward bastions on the southwest and southeast sides. Curtain walling linked the bastions.