Could also be bacteria or fungus building up in the dirt. You could try heating it in an oven (that won't kill everything, but it may keep stuff down.)