New Mexico Mercury

…There are still questions about the effectiveness of web panels, which can reach only the 81 percent of Americans who use the Internet. That’s worse than the 98 percent of households that can be reached by a live interview telephone survey, although it’s better than the 63.5 percent of Americans who have a landline telephone and can therefore be contacted by automated polling firms, which are prohibited by federal regulations from calling people on their cellphones.

Non-Internet users tend to be less educated, less affluent and more likely to be Hispanic or over age 65. These concerns aren’t strictly theoretical: YouGov most likely underestimated President Obama’s share of the Hispanic vote in 2012. Its final survey showed Mr. Obama with 59 percent of the Hispanic vote, far lower than the 71 percent in the exit polls.  – Nate Cohn, The New York Times (The Upshot), July 27, 2014

One September evening in 2012, I was coordinating the phone bank at an Obama campaign office. Completing the shift, I gathered up the call sheets. There weren’t a lot, as it was before the final weeks frenzy that really attracts the volunteers. And New Mexico was, by that time, pretty clearly not a battleground state. So it was just a few of us phoning.

The campaign had a pretty strict rule that the call sheet results should be uploaded to Chicago that same evening. Our data volunteer had not shown up. I hadn’t done that data upload task before, so I hustled in to headquarters in Santa Fe to get some on-the-job training. Headquarters was a sprawling big place and people were still there finishing up tasks. I quickly got some help and the results of the evening’s phone bank were entered into a system that put it all to Chicago campaign headquarters.

The Obama campaign data folks had this huge data stream updating every night from all over the country. With it, they could tell better than any poll just where the campaign was, state by state, down to precincts – and even down to individual voters. Certainly they could read into data by demographic and lifestyle slices just what the electorate was shaping up to be.

Not to say the data we uploaded was clean. In fact, it was subject to lots of error.

First, there was the difficulty of getting voters on the phone. A lot of people screen their calls and just don’t pick up very often.  When a voter did answer the call, volunteers varied in their abilities to ask the questions and to record the answers. Even with training, volunteers would sometimes find their own muses when it came to collecting data. So the data wasn’t perfect. It had process errors.

Nevertheless, the sheer scope of the nationwide effort gave the data wranglers at Obama headquarters a very good feel for the way the electorate was moving, in spite of all the unavoidable noise caused by process induced errors. The signal, so strong and broad and deep, would override the noise.

I was thinking about that experience collecting voter data as the U.S. political polling universe shifted a bit this past week when CBS and The New York Times decided to radically depart from their reliance on traditional live-interview telephone polling. In collaboration with the British marketing survey firm YouGov, CBS and The New York Times switched from random-digit dialing to a much more difficult to understand methodology for their political polling.

YouGov has been playing in the political polling sandbox for quite a while all over the globe, but the embrace of their internet interview methodology by the most respected news media corporations was a small shock. A shock that faded quickly enough. The political-journalism complex collectively shrugged and moved on.

But here in New Mexico, the YouGov methodology has some particular weaknesses I think we should keep in mind. Nate Cohn has pointed out potential weaknesses and some of these are also explained by Drew Silver of the Pew Research Center. Taniel at DailyKos has some further cautions about the results of all these CBS/NYT/YouGov polls.

The YouGov methodology has grown more robust and defensible as more of the population  connects to the internet. Internet connectedness overall in the U.S. is estimated at about 80%. New Mexicans, however, are not as connected. Governing magazine estimated last year that 39.6% of our population had no internet access. We are second to last in the states, partners in backwardness with Mississippi. 41% of its population had no internet access.

In this CBS/NYT/YouGov poll for governor, out of 931 interviews, just 107 were with Hispanics – 11%. In presidential elections over the last decade, the percentage of the vote from Hispanics in New Mexico ranged from 32% to 41%, according Pew Research.

The CBS/NYT/YouGov poll, conducted 7/5–24, found Gov. Martinez leading Gary King 52 to 44. The cross tabs showed Hispanics favor Gary King, 55 to 40.

It is usual polling practice to weight interviews to adjust for less than optimal samples, but here is a case where the weighting needs to pull 11% of the interviews up to represent 30% or more of the final results. Seems a heavy lift, and I’ll bet the YouGov statisticians wish they had some more Hispanic interviews from New Mexico. Or they would if they are paying close attention. They released a whole batch of poll data, and I can’t help but suspect a small state like New Mexico was not high on the priority of the middle-managers pushing out the publication of data.

I think the YouGov internet interview is a reasonable survey method in most cases. It certainly is cheaper than more traditional methods of political polling. But they need to have a lot more interviews in New Mexico, and a lot more Hispanic interviews, to sharpen the focus.

(Background data image by r2hox)