I would take a day and see Mont-Saint-Michel. Then the next day, get a guide who will arrange transportation and hit the highlights of Normandy that you most want to see. We did a guide and it was worth every penny.  Then take one more day to visit other things on your own at your own speed.

Don't try and do it as a day trip though. You'll miss way too much.