3-4 in. of water on a 20 gallon can be too much volume for a heat lamp; unless room stays steady and the heat lamp is capable to slowly bring up the whole enclosure to desired temperature. If heat lamp is not enough or if below situation is right, you need a submersible water heater rated for the tank water volume in addition to heat lamp. Do not recommend using an under tank heater to warm up the water.
Not sure understand clearly; but if your room temperature varies due to A/C being on/off, you will probably also need a thermostat to control the heat lamp (water heater has own thermostat). Good luck!