April 23, 2009 05:47 PM UTC

More on why PPP polls are lousy

•  
• by: twas brillig

(Yeah, baby, rumble – promoted by Colorado Pols)

Tom Jensen, some guy at Public Policy Polling, takes issue with my post yesterday. My central point was that any poll right now that attributes to US Sen. Michael Bennet 75% name rec — let alone 75% opinion-holding — among Colorado voters is laughable on its face. If the guy even breaks 30% name recognition in the state right now, he should take his team out for steak.

Here’s what Jensen has to say in reply — keep in mind, this guy regards himself and is treated in the press as a reputable pollster:

Some blogger in Colorado whacks our poll today, saying there’s no way 75% of voters in the state really have an opinion about Michael Bennet.

I think that’s probably true, but it doesn’t mean there’s anything wrong with our poll. [twas brillig emph.]

Um…Tom, yes it does. Big time. If you put out a poll memo making claims about Bennet’s standing, and then acknowledge your poll didn’t really measure Bennet’s standing, there’s something very wrong here indeed.

So before going on to accuse me of writing “nasty stuff,” Jensen agrees with my central premise: This poll is meaningless in determining Michael Bennet’s public standing.

However unintentionally, Jensen is conceding PPP’s core methodology is scientifically unreliable. First though, let’s give some leg room below the fold to his goofy rationalization:

We choose to include party labels whenever we do approval or favorability ratings on pretty much any politician. That’s because for a lot of voters party labels go a lot further toward their perceptions of elected officials than anything specific to the individuals themselves. So there were probably a lot of Republicans in this poll who don’t know much about Bennet but disapprove of him because he’s a Democrat and a lot of Democrats who don’t know much about him but said they approved of his work for the same reason. If you don’t include party label you probably get more than 50% with no opinion, but since voters making their decisions based solely on party labels and little else is a real fact in politics we give them when we’re polling.

So basically, Jensen is saying his poll bears no relation to Bennet’s actual performance rating; it’s all tied up in party ID, but it’s a personal gut-level guess how much; and if you don’t include party label “No Opinion” is probably more than 50%, but who knows! In the midst of a number of hackneyed assumptions, Jensen is essentially conceding they think their polling is simply a basic read on current party perception and little more, but then they deliberately write their script in a way to massage candidate results.

This is why reputable polling firms differentiate generic head-to-heads (testing party approval without candidate names) from testing individuals; to get a more scientifically reliable result. PPP’s poll script makes this entire exercise GIGO (garbage in, garbage out).

Jensen’s admissions today is in stark contrast to the confident and definitive claims made in yesterday’s polling memo, which was unfortunately treated as authoritative by press outlets:

41% say they disapprove of [Bennet’s] job performance so far, with 34% approving.  25% don’t have an opinion one way or the other.  He is meeting with approval from 59% of Democrats but only 11% of Republicans, and his overall reviews from independents are negative as well, with 32% approving but 43% disapproving.

We can see now that PPP is in no position to make any reliable claims about what anyone thinks of Michael Bennet, according to their own pollster. They do not have any reliable data upon which to base their claims.

This doesn’t even take into account the classic unreliability of robo-polling generally, the sample problem of this particular poll, and the unscientific quality of the script. Do they even randomize the script to switch up the candidate questions for increased reliability? Probably not, because that would take more work. This is a methodological nightmare!

That PPP seems oblivious to this is the most disconcerting thing. I’m sorry, but two people with a robo-dialer and a logo do not a polling firm make.