I would remove it all. Mold could be growing on other parts, and you might not be able to see it.