Hello and welcome to the forum!

To really scrub it out, you could use vinegar and rinse the tank until there is no odor.

Other than that, everything Eli has said is spot on. In a 5 gallon, a grey tree frog will eventually get ill from the lack of space. A 20g high would be a very good upgrade. The drainage layer will also keep the coco fiber from getting swamped and smelling. If you added a "cleaner crew" like isopods or springtails, you could go up to a year without changing it out. However, the BEST way to do it would be to use a drainage layer and a substrate that will support live plants, you can go a considerable amount of time without changing it then. I have never had any sort of odor doing things that way.

I would highly suggest reading this - Frog Forum - Gray Tree Frog Care and Breeding It will give you a lot of info in caring for a captive grey tree frog