Strangely enough, the existing curtain and corner towers are not the first on the site. King John granted Goodrich to the mighty William Marshal, Earl of Pembroke, and a stone enclosure followed. Some of his masonry is embedded in the present east curtain and the foundations of a round tower underlie the present southwest tower.

A later Earl of Pembroke, William de Valence, tore this structure down and erected his own. His building here is contemporary and comparable with the Edwardian castles of Wales. Such a castle is a rarity in England. It is square in plan, the more vulnerable south and east sides being protected by a wide, rock-cut ditch. A thick curtain surrounds the bailey, with massive round towers at three corners and a gatehouse occupying the fourth. Each tower rises from a solid square base, which sinks back into the cylinder in pyramid fashion. Forming spurs. The spurs projecting from the southeast tower are particularly high.