Caister was one of the finest of its kind but rather too much was pulled down in the eighteenth century. The castle stands in a wide moat still full of water. It is one of those with an inner quadrangle and a subsidiary base court for retainers. This is less obvious now because the arm of the moat between the two courtyards has been filled in.

There is also part of a third courtyard behind, arrested only by a circular corner tower incorporated in a later house. The base court, of inferior brick, is now fragmentary and the main quadrangle had suffered so much destruction that only its north and west walls still stand.