I was starting new games this weekend, trying to get one on islands so I could play an island hopping game.
One of the games, I actually started underwater. The screen came up and it was dark and at first I thought it was still loading, then I saw the little air bubbles disappearing. So I aimed "Up" and pushed forward and sure enough I got out of the water to find myself in a swamp.
But does this indicate there is something wrong with my Minecraft program and I should reload? Or does it really choose a location underwater to start a person on occasion?
This is normal. The game doesn't care (or sometimes even know) what is there when it selects a spawn point for you. I've spawned in water, in mid-air over a ravine, and in one of my modded worlds I even spawned in dead because an AE2 meteor was placed right next to where I spawned (broke the bonus chest and everything).
The game has several measures in place to make sure that you do not spawn in an ocean although it can fail if the ocean is large enough; first, it checks for a plains, forest, jungle, or taiga within 256 blocks of the origin (0,0) and sets the world spawn point at a randomly chosen spot (sort of, it favors locations towards +x and +z) within one of these biomes (these biomes have not been changed since before 1.7) if it find a grass block at y=63 or higher (note - sea level is at y=62 in terms of the highest water block), otherwise, it will move the spawn point in a random walk until it finds one; however, this not however consider features that generate after the terrain does, such as water and lava lakes.
If you create a new world and happen to spawn right at or very close to 0,0 (note - the player's spawn point on each respawn is randomized within a 20x20 block area around the exact world spawn point, so it can spawn you over things like ravines, though I've always spawned at the bottom, not in mid-air) this is because it could not find one of the aforementioned biomes, but a biome covered in grass was at 0,0. If the game takes longer than usual to create a new world and you spawn much further away, sometimes 1000 blocks or more, then it is because there was an ocean or very large desert around the origin and the game makes up to 1000 attempts at finding a chunk with exposed grass before giving up; this was much more common prior to 1.7 due to the much larger oceans, as well as with larger biomes (Large Biomes or Customized).
Here is an example of what the game does in the worst-case scenario, using the seed "-1984348358409562691" (default) in 1.6.4:
The world spawn point is at the center of the big square on the left side, while the origin is near the right side, with a trail of chunks leading away from it. A few chunks did land on some exposed seafloor (1.6.4's version of "islands" in oceans) and a couple Mushroom Islands but neither were seen as valid spawn points.