The Steen is all that remains of a much bigger castle that was built over the site of a 6th c fortress in Oude Werf, the oldest part of Antwerp.
The castle was built around 1200 and was the first building to be built of stone (steen in Dutch) and was the home of the Burgrave of Antwerp. The complex included a church, courthouse and several other buildings, all of which were protected by a defensive wall surrounding it.
Around 1520 the castle was thoroughly renovated by Charles V and you don’t need to have gone to Specsavers to see where the old and newer stone joins up.
Up unto the 1820s it was used as a prison, but later that century a decision was made to demolish most of the castle and Oude Werf district to prevent the Scheldt silting up. The port was vital to Antwerp, and so the river was widened and new quays built. It must have been a difficult decision to make as it involved knocking down over five hundred historic buildings, and even the Steen was only saved from the chop by a single vote.