The first trading lodge in the area was built by the Portuguese in 1555 and was named “Cabo Corso”, which means short cape, later corrupted to Cape Coast. Sweden built the first permanent fort in 1653 and named it Carolusburg after King Charles X of Sweden. During the next eleven years, the Swedes, Danes, and the local Fetu chief each captured and controlled Carolusburg.

The English fleet finally captured Carolusburg and it remained in English possession until the late 19th century. It served as the headquarters of the British governor. It was the British who transformed the fort into a castle.

It is estimated that by 1700, the British were shipping 70,000 slaves per year from Cape Coast to the Americas as part of the trans-Atlantic slave trade. In 1766 the British undertook a major rehabilitation of the castle, giving it its present look.

After the slave trade was abolished the castle became an important post for legitimate trade. The British also used it as a training facility for Jamaican soldiers during their wars with the Asante in Ghana. It has also been used as a school for Ghanaian children and as the regional headquarters of the Museums and Monuments Board.