The building of the first lodge in the 17th century at Ursu or Osu is attributed to the Swedish African Company. In 1657, the Swedish headquarters at Carolusburg Castle, Cape Coast was captured by the Danish Guinea Company led by Heindrick Caerlof who himself was formerly the Swedish Africa Company's Commanding Director. This resulted in all Swedish establishments including Osu lodge passing into the hands of the Danish Africa Company. Caerlof defected to the Dutch in 1659. The Danish Commander of Carolusburg was tricked into believing that Denmark had been conquered. He therefore sold Carolusburg to the Dutch and with it the former Swedish establishments including Osu lodge. The Ga Paramount Chief Okaikoi, disgusted with their trickery, asked the Dutch to leave Osu. In 1661, Jost Cramer, Danish governor of Fredericksborg, near Cape Coast, acquired land from Chief Okaikoi for 3,200 gold florins. The Danes built a stone fort to replace the earthen lodge and named it Christiansborg (Christian's fortress) after the former King of Denmark, Christian IV who had died in 1648.

In 1679, Peder Bolt, a Greek who was deputy commandant at Christiansborg, instigated the murder of the Danish commandant Johan Ulrich and sold the castle to Julian de Campo Baretto, former governor of the Portuguese Island of Sao Thome.

The Portuguese renamed the castle "St. Francis Xavier", garrisoned it, constructed a Roman Catholic Chapel in it and made architectural improvement on the bastions.