I can't imagine going there during baseball season and not doing that.
I went two summers ago for the first time, and going to Fenway was the main reason, but we did some other stuff as well. I have a lot of family history in that area, my ancestors came to Massachusetts in the early 1600's, so we spent some time going and seeing some ancestral homes in Andover and the surrounding area.
We also went and ate some lobster, it is pretty inexpensive there in comparison to here, and of course some clam chowder. We just saw some basic city sights like the harbor, went and saw Cheers, and just checked out some historical stuff.
I'd like to go back and do some more. My wife and I have been considering going to Martha's Vineyard this summer and if we do I think we'll make a couple of days to see Boston since she hasn't been there yet.