In reply to David Rose:
Where the entrance is on access land then CRoW should let you walk to the entrance. The issue is then whether you can enter the cave, but this would be trespass not criminal. If CRoW applies you could insist on access where a landowner currently doesn't allow it, but on the other hand, there seem to be cases where caving groups themselves control access for (presumably) good reasons, and if CRoW does apply then this could be unlawful. Be careful what you wish for. How much of a problem is it in practice?
Is there any caving in Scotland? If so does the experience there shed any light on the issue? Scotland has much wider rights of access for recreational purposes and although caving is not specifically mentioned it will surely fall within that.