In reply to maxsmith:
>I am looking at the Garmin Forerunner 310xt which is available for a tenner-ish on ebay and seems to give uk grid references.
The Forerunner 310XT does indeed give OS grid references, but mine at least is terribly slow to get a location fix when first powered up. Slow like it sometimes still hasn't worked out where it is after ten minutes standing around outdoors waiting to start my run/bike ride/whatever.
I've now replaced it with a Forerunner 735XT which is "always on" (sitting in a drawer the battery lasts at least a week) and get a GPS fix in seconds when you want to start an activity. However, that's probably too expensive for what you're looking for, and doesn't really meet your criterion of being able to be switched fully off (it can be done, but there isn't a single straightforward button to do it like on the 310XT).
I do think that an eTrex or smartphone solution might actually be better for your needs. (As others have said, you can be uncontactable with a smartphone simply by putting it in to airplane mode. The GPS will still work because it's a passive system. And airplane mode is persistent over power off/power on cycles. On the down side, simply powering on a smartphone can eat quite a bit of battery on its own.)