I've taken frequent 3-4 day trips and never had any issues. Feed them right before you leave like you said. ACF are capable of going a very very long time without food.
My biggest concern about leaving town is making sure the lid is secure so they do not escape while away.