Just stick with it, don't give up when few people are joining. If you provide people what they want and can use they will come.
My site (in my signature) was started last November (I think), and I just let it be and people joined ever so slowly last year, maybe 2 or 4 a day I guess, not too many anyway, then something happened in late January early February and now about 100 people a day are registering.
It just takes time, so keep at it even if people give you a hard time about it. You'll get a community going if you give the people what they want in a easy to use format and listen to your member base's suggestions.
Oh and most of all, enjoy what you are doing, it will show.