Another told him that wasn't entirely true. It was once "a vastly wealthy community," as evidenced by the great mansions on High Street.
The first man disagreed. It may have been wealthy long ago, but the recent influx of new wealth meant many of the old working-class neighborhoods were disappearing.
Yet another man chimed in, a relative newcomer to town, offended at the notion that non-natives were ruining the town.
"Many of us are not native Newburyporters who wouldn't want to live anywhere else," he said.
It's a debate that could have occurred yesterday on the central waterfront boardwalk or at the lunch counter in Angie's Restaurant.
But it didn't. It took place almost 32 years ago, in front of a committee that was charged with probing the impact of Newburyport's unique brand of urban renewal.
Newburyport's downtown revival of the 1960s, '70s and '80s left an indelible mark. It made the city a national model for preservation. It restored a dying downtown.
And it caused enormous social changes that have stretched over 30 years and counting.
"A lot of people want progress without change. There's no way you can have progress without change," then-Mayor Byron Matthews said at that meeting in front of the Growth Policy Committee in 1975.
Newburyport wasn't the first to shun the "federal bulldozer," the pejorative name for the urban renewal practice of tearing down the core of old cities and rebuilding them.
But by the mid-1970s, its story was so compelling, its turnaround so quick and its physical attractiveness so becoming, it became the poster child for success - even though it hadn't entirely succeeded.
At that time the revival of the downtown was still far from finished. Many buildings were vacant, and the waterfront was an ugly dirt lot. But the first phase - the Inn Street Mall - was alive and well, several storefronts along State Street were restored and reopening with unique artisan shops, and the streets and sidewalks had been attractively redone in a Colonial style.
Against a national backdrop in which America was struggling to revive its inner cities, Newburyport's successes were widely applauded. The New York Times, Wall Street Journal, Christian Science Monitor and many other newspapers trumpeted the spirit of renewal and restored charm. Typical of the headlines was one from the Lowell Sun: "Newburyport: A City that Came Back."
Awards poured in - the Henry David Thoreau Landscape Award, the New England Architectural Award, a Boston Society of Architects award, and others.
The federal government heaped praise on the city, too. In 1976, the Inn Street Mall won the top prize from the Federal Highway Administration for "outstanding example of sympathetic treatment of historical, cultural or natural environment." And when the National Trust for Historic Preservation published a "how-to" handbook on "recycling old buildings," Newburyport was one of the featured cities.
Architects and planners made pilgrimages to Newburyport to study its success. Officials from other cities came to learn the nuts and bolts of restoring a downtown.
Tourists gawked at the changes in progress. The National Trust brought 200 planners and historic preservationists here in 1975.
"We love this city's downtown shopping area," said a visiting planner from Mobile, Ala. "We hate the bulldozer approach to everything."
Even the Cold War didn't prevent a peek at Newburyport from behind the Iron Curtain. The State Department brought a handful of Russians, Czechs and an East German to the city for a tour. Newburyporters opened their doors to them.
And Timothy Anderson - a noted architect hired by the Newburyport Redevelopment Authority as a consultant - brought Harvard School of Design students here, to a real-world classroom.
In late 1975, Newburyporters gathered to see themselves on the big screen. Filmmaker Larry Rosenblum had finished his three-year-long project, "A Measure of Change," a half-hour documentary that explored the city's battle to stop the federal bulldozer.
"The film may be a catalyst as well as a piece of Yankee advice, 'look before you leap,'" The Daily News stated in a film critique.
Within a few months, the film was getting international attention. It won several awards and was selected as the U.S. entry at urban planning conferences in Stockholm and Leningrad (now St. Petersburg).
Suddenly, the little old seaport was exporting to the world again. This time it wasn't goods, it was a concept: historic preservation and revitalization.
In an interview last week, Rosenblum noted that Newburyport's place in the preservation movement is significant.
"Urban renewal was conceived as a big-city initiative for tearing down large sections of what was considered blighted and unmitigatable areas," he said. "Newburyport was one of the first small cities to take advantage of urban renewal ... it was also one of the first small cities to go down that road, to stop and do a '180.'"
The film was widely viewed as an example of how preservation can work, but it also ran afoul of important issues that Rosenblum said are not unique to Newburyport.
By the time the film was shown, the city's redevelopment was mired in lawsuits that had halted progress. Some projects were restarted a few years later when the suits were settled, but some significant pieces - for example, the waterfront - are still awaiting redevelopment today.
"The story we told was not necessarily the story we intended to tell," he said. "The film became a 'process' film, as opposed to an 'outcome' film."
In retrospect, Rosenblum said, Newburyport's long and still continuing struggle to complete its downtown is a typical experience.
"It really reveals some of the underlying tension surrounding the preservation movement," he said, noting that factors like economics, public access and private land rights are common obstacles.
"I came to no longer be surprised that these issues often take decades to resolve themselves," Rosenblum said. "How long should it take? That's something that we just don't know."
Rosenblum is sill active in the preservation movement. He serves on the Planning Board in Plymouth, where he has been dealing with an enormous development undertaken by a name now familiar to Newburyporters - Stephen Karp.
Almost 32 years later, Jim Gaines still remembers those Growth Policy Committee meetings. He was the chairman, the "newcomer" who had argued that new residents were not carpetbaggers intent on changing the character of the city.
When he made those statements in 1976, he had lived in Newburyport for 12 years. It had already changed significantly since his first visit in 1960.
"When I first came here, half the downtown looked like Berlin, all bombed out," he said. "I thought, 'who would ever want to live in a dump like this?'"
But his wife was a Newburyporter, and so he moved to town.
"It was the best thing I ever did," he said.
Gaines served on the Planning Board for 20 years and was head of the Merrimack Valley Regional Planning Commission for a time. He said Newburyport stood as an example to other old cities in the region, such as Lawrence and Haverhill.
"To my way of thinking, it's the most successful community in the country," he said.
That kind of pride was exported by many Newburyporters.
Many who lived through the changes have kept scrapbooks documenting the changes. Among them was the late Hack Pramberg, who put together a slide show of the downtown's transformation. He showed it to many audiences.
He grew up in the city and became president of the Institution for Savings, where he made commercial loans for downtown projects that outside banks wouldn't touch due to the risk involved. But his faith in the city's future was deep, and he was proven right.
There's no doubt that the downtown's revival altered Newburyport, and there's plenty of evidence to show it was an earthshaking change.
Federal Census figures showed a sea of change occurred between 1970 and 1980, coinciding with the redevelopment of the downtown. The population had stayed at about 15,900, but the figures indicated younger, well-educated professionals had discovered the city in droves. Older residents were moving out.
In an analysis done for The Daily News in 1983, Charles McSweeney of the Center for Massachusetts Data noted, "everything gentrification represents seems exaggerated in Newburyport."
There had been an unusually large turnover in homeownership, median income had shot up far higher than the state and county averages, and the number of college graduates had increased almost 200 percent.
The value of homes had gone up 54 percent, more than three times the state average. Home values had reached $47,000 in 1980, but that was nothing compared to the real estate bubble to come.
Today, the average home is assessed at $488,000.
Those economic and social changes have spurred another debate - did Newburyport lose its "character?"
It depends on whom you ask.
During the writing of The Port in Progress series, many longtime residents, some of whom moved out years ago, argued that the spirit of the city had noticeably changed. Old neighborhoods had been broken up, particularly those that were home to immigrant populations. Bonds that had been forged over decades were gone.
John Lagoulis, who grew up in a neighborhood on Unicorn Street that was entirely erased by urban renewal, voiced that viewpoint.
"Newburyport had a stronger bond than today, maybe due to necessity," he said. "We had very difficult years for a long time. Now we have a more beautiful city, but I feel that people are cold to the bond."
Others argued that there's a rose-colored tint to those who look back on Newburyport's past. In the 1960s, as had happened at other times in its history, the city's economy was foundering, forcing an exodus of sons and daughters. But the renewal gave the city newfound hope and wealth.
Some Newburyporters left anyway, selling their homes for many times what they had been worth just a few years before. Some felt forced out. Others saw an opportunity to cash in.
"There were people who moved out because they could buy a home and put a couple hundred thousand dollars in the bank," Gaines said.
In the 1970s, the Growth Policy Committee struggled to answer the same questions. Then as now it was a matter of an individual's perception.
But there was one aspect of the change that the committee agreed upon.
In an analysis of the committee's findings in 1976, The Daily News wrote, "While the character of the city had changed, such was an inevitable result of the community's 'revival,' and no one wanted a return to the Depression hardship days of not so long ago."