Right off Newburyport is Plum Island: 6 miles long, 1 mile wide, bird sanctuary, multiple points of access to ocean via dunes, don't disturb the wildlife, swim where and when you want. Ca. $5 entry fee. Relatives will know all about it. Bit further south, Crane Beach and Reservation, $$, more shells, swim where and when. Driving south through Essex County you'll encounter all sorts of wonderful ocean-based places: Cape Ann, Gloucester, Rockport, Marblehead, then might look at harbor islands in Boston Harbor (Legal Seafood), jump further south to Hingham, then Cape Cod and all its wonders.
But you seem to be describing Nantucket more than anything... ($$$$).
Driving north from Newburyport you'll quickly cross the little bit of NH with seafront (Old Orchard Beach) and hit ME. Kennebunkport, Cape Elizabeth, Portland (Casco Bay), Bristol, Five Islands, Cook's Restaurant for seafood on a neighboring peninsula, then on to Acadia and Bar Harbor, further Down East to Macchias (Bold Coast trail for hiking and puffin and whale watching), and so on to Eastport, then, if you've brought your passport, hop over to Rockefeller estate in Canada.
I know nothing about fine dining and wine lists. Might hit Legal Seafood in Boston to say you've done it.
For 2 nights away in one spot, I might pick Acadia. (It's rather more than 2 hours from Newburyport.) Maybe do a boat trip around harbor islands or helicopter ride or day hiking. Soames Sound?
Or maybe Camden, ME.
Basically, I would be open to what a new place has to offer, and there is a lot right around Newburyport. Not sure "neat beach communities" as such exist outside the Cape and the Islands.