In reply to Big Sender:
If there was one black and white issue in climbing it is that we don't improve holds just because we feel like it.
Sandpaper is abrasive, yes? hence you are removing a layer of rock, even if you think it is only a very small one. Some rock types, especially sandstone, have a very thin patina of hard outer rock, and when this is gone it erodes extremely quickly. If a hold is polised then it is at least not being eroded, so you are accelerating the process of erosion. It is in no way, shape or form 'restoring the route to its natural state'
There may arguably be a fine line between 'acceptable' and 'over vigourous' cleaning but use of sandpaper to roughen holds is way over the line into hold improvement.
Also, it is cheating and bringing the route down to your level.
Manmade quarries don't make a difference, modern ethics don't allow chipping holds in quarries just because they are man-made either.