In reply to IanC:
Many of the things described above are better classified as 'games' and this is the essence of the Hemingway quote, differentiating all these things, Olympic or not, as games and only bullfighting, motor racing and mountain climbing as sport.
The difference?
Not just a set of rules - those are for games.
Not just competition - again, games are often organised competitions.
Not just physicality - plenty of games require that.
The essence that differentiates sport is more intangible.
There is an element of chance.
There is an element of risk - there must be something important at stake - usually one's life.
There is an element of learned skill - to be able to take on that risk beyond everyday levels
It must be unnecessary - in true life or death utilitarian situations (war, work, survival) anything goes. The very uselessness of sport allows one to entertain and practice a degree of fairness and grace that fighting for one's survival or livelihood would not. Thus eye gouging is fine to defend yourself in war, but not in boxing. Bottled oxygen is fine for going into space, but not on mountains.
There is an element of fairness ("he's a good sport", "a sporting chance") that need not rely on rules or authorities, but on shared or established cultural or community values. Being able to appreciate the innate justice and quality of the unwritten rules and abide by them is part of being 'sporting'.
Sport enables one to show that you have evolved beyond craven survival and sustenance and can exhibit the development of your character in other ways, ways that are serious and testing - and beyond mere games.