The castle was built by the Bigod family soon after the Norman Conquest. Despite its military appearance, its location here is determined by administrative considerations, as the Normans took over this previous Saxon soke & estate centre, which was already the focus for administrative & juristictional authority. It is also the centre of the Baldry family. The Bigods did build a military castle at Walton on the coast, as a precaution against the frequent Danish raids & another administrative castle at Bungay.