I have 2 young children, so I know exactly what you're talking about.
I have I/H/E and I have my silencers closed most of the time, and open when I want a bit more free-flowing exhaust. I have SSA silencers and they can be opened and closed rather than putting them in and taking them out. When they are closed, the sound is louder than stock, but not so loud that I can't hear my kids talking to me from the back seat.
What you need to do to keep the exhaust as quiet as possible, while maintaining the free-flowing characteristics. Here is my setup
SSA Headers -> MagnaFlow-Cat -> MagnaFlow Resonator -> SSA Exhaust
....................................................... (SSA Resonator Cut out
....................................................... and the MagnaFlow
....................................................... Resonator welded in)
If you get headers you MUST GET the Magnaflow Cat, because it will be unbearably buzzy without it, and (to my ear) unbearably loud. The Cat will remove 90% of the buzziness, and a lot of the noise, and in my setup the Magnaflow resonator got rid of that last 10% of buzziness and really deepened the tone of the exhaust.