In reply to blackmountainbiker:
Brychan was a 5th C Welsh king, Brecon would have been at the heart of his rather small kingdom. When the Normans arrived they obviously realised it was a good spot for a castle, and presumably the county and town names got confused.
Bannau is the plural of ban, meaning peak, mutated to Fan in Pen Y Fan.