You can't get rid of any fungi by pulling out the "mushroom cap." That is just the reproductive body of a spread out organism that is living of organic matter in enclosure. Don't think will hurt your frogs, and will probably go away by itself once the organic matter they feed off is gone.

One thing would check is the enclosure humidity level and that there is no waterlogged soil in enclosure. Lowering humidity might be all you need to do to make any fungi go away .