The frogs will track the coconut shell into the water, that is for certain, I am not sure if it will clog the filter though.
Live moss may be the best substrate for a semi-aquatic set-up.
I used coconut fiber for a couple weeks and it turned the water yellow and eventually clogged my water pumps. I went back to using rounded stones.