I would stop handling him 100% right now. Whites may tolerate handling a little better than say, a red eyed tree frog, but it will still stress them out no matter how chill they may appear. If he's not doing great, losing weight, I would make sure to reduce the stress as much as possible. I'm not sure about the "panting." I've never seen a tree frog pant, maybe shedding? What are you feeding him? If poop is rare, feeders may be too big and he's having trouble digesting them. What kind of water are you using?

EDIT - How long have you had him? Where did you get him? Wild caught or captive bred? Ever had fecal tests done?