You can't stop them from walking out the sides. Most of my fruit fly eating frogs are in critter keepers or faunariums (ones without vents on the sides). I put paper towels between the tops and the rest of the enclosure. If more than one paper towel is needed, then I put masking tape over the perforation between the towels. This will keep the flies inside the cage.
I would not put petroleum jelly in any frogs' cage.