City Survey Finds, Shockingly, the City is a Good Place to Live

I love spotting surveys like this -- a survey by city officials that discovers people who live in the city really like living in the city.  Below, I get into the methodology.  But first let's look at the city, in this case Fredericksburg, Virginia.  Here's a graph:

The survey was sent to 1,200 residents and 323 completed and returned it. A whopping 77 percent of them said called Fredericksburg an "excellent" or "good" place to live. The 20-question survey covered eight categories: community, quality, community design, public safety, environmental sustainability, recreation and wellness, community inclusiveness, civic engagement and public trust. 

Whopping?  Please, don't use whopping in a news story, even if something actually, um, whops.  Say instead: three-fourths of respondents called the town an "excellent" or "good" place to live.  Save the whopping for the city's PR folks.

The full report is here (annoying pdf format).  In it you'll find a few interesting tidbits, not because I care one way or the other about a city I've never visited, but it's telling for those interested in methodology and consultants who do this sort of thing.  There's nothing wrong with the survey.  It looks like a lot of similar projects.  But still, let's look deeper for the methodologically inclined.

• It's a mail survey.  That's unusual in these times.  Mail surveys tend to have lousy return rates (that is, how many surveys you send out versus how many returned).
• But this one has a great return rate, with 1200 households surveyed and 323 returns.  That's a 27 percent return rate.  Most mail surveys are lucky if they manage 10-15 percent.  They call it a "multi-contact" survey, meaning they annoyed people to up their return.  Which is good.
• The results were statistically weighted.  That means the consultants took census numbers and corrected their results by key demographic factors.  So if you came up with too many men in your survey as compared to what the census reports you have in the city, you statistically downgrade the results of men (or increase the opinions of women).
• There's a 5 percent margin of error, which is big but not huge.
• They took out the "don't know" responses on individual items.  This is a bit odd because, depending on their distribution, this may significantly alter the results.  Some of the "don't know" results on individual questions ranged as high as 70 percent.
• In the questionnaire, "excellent" is the first choice, followed by "good" and "fair" and "poor" and dunno.  By putting "excellent" first, you bias the results higher.  If you read the frequencies in Appendix A, you discover why they collapsed "excellent" and "good" together when reporting the results -- more people merely said "good."  A good reporter would have caught this and reported it as such.

Why am I picking on this town's survey?  Because lots of cities do this sort of thing, and lots of people do this kind of consulting, and I'm not against either.  But when reporting this journalistically, you have to dig into the methodological details to really tell what people think about an issue or place.