Both places won't be touristy until just before July 4, so you'll have the entire place to yourself unless a cruise ship docks.

DiMillo's Floating Restaurant is on the water, people usually go for the lobster.
J's Oyster Bar which is hidden between 2 wharfs just happast DiMillo's, for the raw oysters.
Becky's Diner for breakfast. A bit of a greasy spoon but it's popular. The Lobster omelet is worth the 10 bucks IMHO.
Gritty's if you like English pub-style Fish & Chips.
Plenty of other good spots in the Old Port if you walk around a bit, like the Porthole, Ri Ra's, Gilbert's Chowder House, Sebago's for microbrewery type food, and so on.
Muddy Rudder just off the 295 in Yarmouth is good if you don't want to be in a urban place, but it can be pricey.
Freeport Inn for breakfast if greasy spoons are a no-no.
Binga's Wingas on Congress St. for damn good hot wings. The city would not let a Hooters come in but let Binga's in :)

I'm sure I am missing some good ones but I don't usually hit the pricey spots, usually Becky's and J's once in a while. There are a couple good sushi restaurants too, Benkay's and Sapporo's which I like for the Dynamite scallop rolls (10 bucks each though).

Bar Harbor, a lot of good restaurants up & down the street. I just go down there to hike on MDI. Try the Canada Cliffs trail (aka Beech Hill Ridge trail) for a decent look straight down a cliff at a lake. Too many trails, I will not ever do them all in my lifetime, but I really like up Acadia Mtn from the road, thru St. Sauveur Mtn and back down to the road, good views of Somes Sound. The mtns in MDI are not very high, but the scenic value is hard to beat. It'll be too cold this time of the year, but one favorite walk is where the high tide is past and starts to go out, then I walk onto the sandbar between Bar Harbor and Bar Island while the sandbar is still underwater. Eventually the entire sandbar is open and you could drive on the sandbar if you wanted. Anyway, the underwater part is fun because you can see crabs and lobsters going out with the tide, and they go right by you. Watch where the seagulls congregrate in the water and you'll know they have zeroed in on them. Haven't been pinched yet but I get out of their way fo sure.

The Info booth on Rt 3 just before Mount Desert Island is a must stop, you'll want to get a lot of info, map swag and pamphlets there. No WW to speak of, there's one steep creek near an old folks home that goes under Rt 3, its been done.