In reply to Croakinglizard:
Long Beach has a little bouldering in the caves, which is decent but probably not worth making the trip for.
Elephant Rocks and Hulk Hogan are both about 30 minutes drive inland from Oamaru. Elephant Rocks is like a smaller, less polished, quieter version of Castle Hill. Hulk Hogan is two minutes up the road from it and is about 150 problems of very very steep limestone. Big holds and big moves...pretty fun, but after a while it all gets a bit samey. The two venues together make for a good day or two bouldering.
Jardines (10 minutes drive from Queenstown) has some very good bouldering in an incredibly spectacular location (under the shadow of a mountain range, views of more mountains and a beautiful lake)...I'm not sure if it's in the South Island guide book though, so might be best to try and get a local to show you (NZ climbers are super friendly).
Wanaka has worldclass sport climbing and a few boulder problems, as does Paynes Ford (top of the South Island, near Takaka).
The North Island has some decent stuff near Wellington (Baring Head and Turakirae) and a few other bits and bobs, but I think South Island is probably your best bet for bouldering.
Oh and you could spend a month at Castle Hill and not get bored!