I keep up to a double chest full and toss any excess beyond that. Dirt's pretty useful as others have mentioned for scaffolding, terraforming, repairing holes in the ground from creepers, building test forms for structure foundations before building the real thing in the material of choice, underground or undersea farms, etc.
I tend to use dirt as an outline on my bigger projects. At the moment I'm creating a 200x200 castle, so I'm building everything out of dirt to lay the outline and such, and I replace the dirt with the finished material (obisidan mostly, but also stone, diamonds, gold,etc) as I get hold of it.
me? well I keep all of it. REALY usefull on true SMP severs good for makeing your house hidden. In SSP good for, well about any thing that does't have to look good, or be creeper proof, of I can't see.
I use it mostly for terraforming. Either fleshing out hills after doing construction or building underground grassy areas.