In reply to FesteringSore:
The problem is probably that the HDMI switch is either using a similar IR protocol to the TV, or is using a very basic protocol (e.g. a simple 'advance only', round-robin (1-2-3-1-2-3...) controller that takes any old IR pulse).
If it's a protocol problem, I'd try each of the TV remote buttons in turn (turn the TV off), and see which of them is causing the problem. At least you'll know not to touch them...
Does the HDMI remote cause any change to the TV? If not, it's probably a pretty dumb protocol being used by the switch.
There's not too much you can do about it, barring physical separation. You might be able to reduce the sensitivity of the HDMI switch by adding an extra piece of IR filter material over its IR control window.