If you can ID the insects first, you'll have a great deal of clarity as to the level of concern they pose. ID'ing them will tell you things like how they came to be in your frog's habitat and from that you can solve the problem. Cleaning is, of course, absolutely necessary even without an infestation but if your substrate or furnishings are the origin of the pests that may not suffice. ID'ing the invaders is the only way to know how to eradicate them.