I think it helps that most parents don't know the rules, so they have no idea why I just disqualified their kid. (And yes, that is the only penalty in swimming. Your race just doesn't count. My daughter got DQ'd at State Championships this past year for a false start.)
Unfortunately it's still really hard to get parents to volunteer as officials. In our current team, with about 125 swimmers, we have 3 trained officials. It takes a minimum of 5 to run a meet. Most meets are several teams, so we get officials from all those teams at them most of the time.
I'm hopeful, though, because we just had two parents express interest in officiating at our team's annual Parents meeting last week.