UKC

I saw someone use a large stiff brush to vigorously "clean" a chalky hold.  It seemed to me that would be way more abrasive and have less cleaning effect than, instead, a gentle wipe with a damp cloth.  This would dry out in minutes and take off the excess dust.

I'm not a sport climber so what do I know?

Why don’t you try both methods and see what works best perhaps before posting? It wouldn’t take a lot of effort and it would answer you’re own question. 

Become a sport climber? 

Good climbing brushes are made with natural fibres - boar's hair allegedly the best, no idea what vegans use - that are supposed to be far less abrasive than nylon (old toothbrushes, what everybody used to use). 

A damp cloth would turn the chalk into a paste rather than removing it.

Brushing holds is not something exclusively to do with sport climbing. It's good to remove excess chalk build up on any and all rock types (and even, or especially plastic).

The word 'allegedly' is a word I would use too. I've often wondered how abrasive something like nylon could be against something as hard as rock. Maybe on soft sandstone it would make a difference but on anything else I'm sceptical. Would be interested to hear more on this.

Ever been down a cave? It's insane what 'soft' things can do with enough time and pressure. 

A stiff nylon brush can start to take the surface off the wrong kind of sandstone. (Previously) guilty as charged. I've seen it on well worn grit when the surface has gone. Hard to generalise, best be cautious - it doesn't grow back

Take a look at any of our hard bouldering caves. Plenty of wear and polish on handholds that don't make sense to use as footholds on any routes.