In reply to northcave:
I've found the soft linings of many jackets get damaged around the hips, and surmised that nylon webbing belts may be the problem, since webbing can trap stray fibres at it flexes.
Something a bit more vigorous is going on in your case, if it's wearing through gore-tex patches, since they're pretty sturdy...
What sort of clothes are you wearing under the downie? tough nylon fabric? Something with velcro (although not usually a problem with smooth fabric like down shells).