Biased Quizzes

Political quizzes like Political Compass and The World’s Smallest Political Quiz are built to shore up support for their writers’ beliefs. The issue emphasis of Political Compass is designed to make respondents appear to the lower left of most politicians and governments; this of TWSPQ is designed to make respondents appear more libertarian.

Over at Retrospectacle, Kristjan Wager does a terrific job at analyzing the ten propositions of TWSPQ and showing how they’re built to make people appear libertarian. Political Compass has neutrally phrased questions, but still exhibits a very strong bias in its choice of issues.

To see what I mean, consider my PC score: -5.38 economic, -9.08 social. I’d say it correctly pegs me as extremely anti-authoritarian. But the -5.38 seems misplaced, considering that the Swedish government‘s score is about a 0.5; my preferred level of government spending is 20 percentage points lower than Sweden’s, my preferred Gini index is higher than Sweden’s by about five points, and I think employers should be able to fire employees more or less at will.

But that disconnect isn’t surprising in light of the choice of questions. The social questions are disproportionately about anti-fascism – for example, the question about abstract art – so any secular anti-fascist will look liberal even if he’s actually neo-conservative. In my case it happens to be right because I’m also very liberal, but that’s by sheer coincidence. The economic questions are skewed toward issues of corporate power, so any pro-corporatist will look right-wing even if he’s generally left-wing on economic issues (e.g. John Edwards, who was pegged in 2004 as somewhere between Kerry and Bush).

That last skew is crucial, because things like reducing corporate power are completely grassroots, without any support from politicians. Why would they have any? After all, politicians get plenty of corporate contributions. The quiz could have just as well had the proposition, “Legislators shouldn’t be allowed to set their own compensation,” which would peg governments as right-wing and the people as left-wing.

TWSPQ is no better, except that its skew is in a different direction. Two economic questions out of five are about welfare, while none is about education or health care. The social questions mention issues that are not controversial in the US, like the draft, ID cards, and free speech (free speech is controversial, but not the way the quiz words it).

Defending the quiz, its authors try explaining why it can’t be skewed in favor of libertarians. First, they say, “If the Quiz were obviously fake, routinely giving phony and inaccurate scores, people would simply ignore it.” That’s true; but the quiz is fake in a non-obvious reason. That’s why it’s successful: it can plausibly look real, which is why it’s useful to libertarians.

Libertarianism is not affiliated with the radical left or the radical right, but it’s still radical, especially as practiced by the US Libertarian Party, which produced the ideas underlying the quiz. It displays all radical pathologies, in particular trivialization. It wants to look like all it’s for is ending welfare, supporting free speech, and not having mandatory ID cards; then, once people are hooked, it starts preaching less acceptable things like total deregulation, privatized health care, and the likes. It doesn’t work for everyone, but at times it can get a few converts. The FAQ’s claim that it doesn’t advance libertarian goals is a rare moment of admission by radicals that what they do is politically ineffective.

Finally, the FAQ’s empirical claim, “The percentage of libertarian scores seems to match very nicely the 20%-30% estimates from various sources of how much of the American population is libertarian or libertarian-leaning,” is wishful thinking. Most Americans are to the left of the Democratic Party on economic issues, closer to the Republicans than to the Democrats on foreign policy issues, and, on culture, ranging from left of Democrats (gay rights) to right of Republicans (general civil liberties). There barely are 20-30% of American voters that are to the right of the Republican Party on economics, let alone that and to the left of the Democratic Party on cultural issues.

TWSPQ is then biased in favor of libertarianism as a trivialization technique. The same applies to PC and the New Left; although the authors of PC don’t say what their bias is, there are subtle hints that they’re a lot closer to Chomsky than to Buchanan, such as the lengths of the reading lists for people in each quadrant and their relevance to current affairs.

It’s possible to design a truly neutral quiz. One way to do so is to scrap the normal two-dimensional map and replace it with an empirical one; Political Survey does just that, ending up with a left/right axis and a pragmatic/idealist one. Another is to retain the usual two-axis model, but choose the questions to mirror the issue emphases of the voters in the target area – say, the entire free world. When people describe politicians with reference to the two-dimensional map, they usually do just that; I graded my book’s politicians on the map based on my understanding of politics, not the PC authors’.

Obviously, this populist approach is vulnerable to ephemeral changes like propaganda and center-shifting. But even so, it’s possible to objectively distinguish issues that correlate to real political preferences from issues that mostly serve to create a disconnect between people and politicians. For example, “The government should give health insurance to all of its citizens” is an example of the former, while “if economic globalisation is inevitable, it should primarily serve humanity rather than the interests of trans-national corporations” is an example of the latter (it’s so trivial to the point that the PC FAQ actually tells people who think the two interests coincide to answer “strongly disagree”).

So a good quiz might start by creating sub-headings of issues in each axis. The social axis might have sub-headings like “Religion,” “foreign policy,” “commitment to democracy,” “sexual freedom,” “immigration/multiculturalism,” “police power,” “feminism,” and “civil liberties.” In particular, the PC questions about abstract art and hierarchy should be tossed in favor or more questions about the value of democracy, which is very easy to understate the importance of if you’re a parochial Westerner.

Then, the quiz should tinker with the weights of and the precise questions in each sub-heading, the main idea being to give more weight to areas that have arisen as bones of contention in politics and that have been barriers to political cooperation. The point of political quizzes is to predict political behavior; an issue that doesn’t cause a schism among people who agree elsewhere doesn’t belong in any quiz. “Religion and state should be strictly separate” is the defining feature of ideological differences in much of the Catholic and Islamic worlds. “There is now a worrying fusion of information and entertainment” is the defining feature of a few low-key flamewars on political message boards.

30 Responses to Biased Quizzes

  1. Interesting analysis, maybe we should make our own political quiz as a type of social experiment.🙂

  2. Katie Kish says:

    Curious…

    “I think employers should be able to fire employees more or less at will.”

    as to what you fully mean by this. You think employers should be able to fire an employee for no reason?

  3. Alon Levy says:

    Shelley: I’m all for it. But the amount of testing this quiz will need is gigantic.

    Katie: I don’t know if “no reason” is good enough, but the Swedish situation, in which it’s basically impossible to fire employees, is excessive.

  4. I bet it would be a bit hit on the blogosphere if it was good enough, don’t you wanna be famous? We can link to it in the sidebar and that will take care of testing. For example, ahve people self-report how liberal/conservative they think they are and we can compare it to how well the test gets at that. The tough part will be picking ‘neutral language’ to get at the major issues.

  5. Kristjan Wager says:

    Well, I am sure Alon and I can figure something out…..

  6. whig says:

    From that Political Survey

    1 left/right -5.8027 (-0.3493)
    2 pragmatism +1.0704 (+0.0644)

    I didn’t like some of the dichotomies, where I felt the right answer was excluded. Prime example: “The government should raise revenue by taxing consumption rather than income.” I believe revenue would best be raised from real estate.

  7. darius says:

    Alon, I’m at the point now where I can’t tell if you’re maliciously misreading for the purpose of making your point, or just being unintentionally obtuse. Saying that the quiz’s failure to display everyone as libertarian (yes, you said “The FAQ’s claim that it doesn’t advance libertarian goals…”, but that’s not what the FAQ says, it says that it doesn’t mark people as libertarian falsely) is “politically ineffective” when your whole argument is that the quiz is biased is just about the most insane thing I’ve read recently. If the quiz is biased, it makes people choose libertarian when they aren’t; if it’s not biased, it’s politically ineffective? And this coming from your reading the part of the website where they explain why it’s not biased?

    Another part that I find amusing is your claim that the numbers from the FAQ are wishful thinking, but then you proceed to make claims about that same percentage (though you for some reason reduce it to voters from overall population). Where did your numbers come from exactly? They list their sources, and mention possible bias in the sources where it may exist.

    Anyway, when you make these claims about the quiz, it still looks to me like you haven’t read it. I don’t see how someone who feels that welfare is not controversial could think that people will select “Agree” to “Replace government welfare with private charity.” If, as you say, it’s not an issue and everybody enjoys the welfare system we have, then they would all choose “Disagree.” This doesn’t demonstrate the ineffectiveness of the quiz, it demonstrates the effectiveness of it. A traditional Republican (might) choose Agree, a traditional Democrat would choose Disagree, and due to their answers on the other questions would end up in different places on the political spectrum.

    Did you try taking the quiz yourself? Did you end up in the libertarian side? I did, but then I agree with the basic concept of libertarianism (that no entity, be it private individuals or the government, should initiate force against others). It looks to me like you would end up in a different place on the spectrum, and for good reason. Which is exactly how the quiz shows that it is not biased.

  8. Alon Levy says:

    Where did your numbers come from exactly? They list their sources, and mention possible bias in the sources where it may exist.

    The part about gay rights comes from 2004’s exit polls, which show that Americans support civil unions 60-37, even as the Democrats run away from them. The part about economics comes from polls about health care, which show strong majority support for single-payer health care, and to some degree about the minimum wage.

    If the quiz is biased, it makes people choose libertarian when they aren’t; if it’s not biased, it’s politically ineffective?

    Uh, what? The FAQ proclaims the quiz can’t be biased because if it were, it’d be politically ineffective for purposes like recruitment. I’m saying that political ineffectiveness has never stopped radical groups like, say, all people who take third parties seriously in the US. Since radical groups like using trivialization techniques – e.g. saying you’re a libertarian because you’re against totalitarianism and for a smallish government, and as a libertarian you must now support privatization of health care and education and abolition of economic regulations.

    I don’t see how someone who feels that welfare is not controversial could think that people will select “Agree” to “Replace government welfare with private charity.”

    If the quiz gave details about the current levels of welfare, you’d be right. But it doesn’t; when it says “welfare,” it implies not paltry benefits nobody can live on, but a more extensive system like Swedish welfare or even pre-1996 American welfare. In 1996, pretty much everyone in the US agreed that welfare should be gutted, which gave the country welfare reform. Right now, the consensus is even stronger than it used to be.

    Did you try taking the quiz yourself?

    Several times in the last 4 years; thanks for asking. I consistently get 10 social, and between 1 and 4 economic, depending on my mood. I always answer libertarian on the social questions and anti-libertarian on the social security, individual welfare, and tax/spending questions; on the corporate welfare and free trade questions I fluctuate, because my view on free trade is that it’d be a very good idea (which is standard among social liberals and the UN, and even a few neo-liberals, like DeLong) and my view of corporate welfare is that it depends on the circumstances.

    It looks to me like you would end up in a different place on the spectrum, and for good reason. Which is exactly how the quiz shows that it is not biased.

    All it shows is that I respond to frames differently from the target audience, which is entirely American. Ask most Americans about individual spending goals – health care, social security, etc. – and you’ll get that their preferred level of spending is actually higher than the current one. Then ask them if taxation and spending ought to be cut, and they’ll say yes because they respond positively to the “small government” frame.

  9. Alon Levy says:

    I bet it would be a bit hit on the blogosphere if it was good enough, don’t you wanna be famous? We can link to it in the sidebar and that will take care of testing.

    Well, I am sure Alon and I can figure something out…

    If you guys are serious about it, then let’s start with, which sub-headings do you think the two axes should have?

  10. ericdondero says:

    The Libertarian Party is not the only representative of the Nation’s libertarian movement. Most libertarians are far more mainstream and active within the GOP.

    Eric at http://www.mainstreamlibertarian.com

  11. Matthias says:

    I don’t think having mega-axes is terribly helpful. Is immigration an economic issue or a cultural issue? The motivations depend on the person. The question of free movement of people intersects with questions of the free movement of capital along with questions of race.

    If we’re trying to get a sense of what a person’s self-consistent political philosophy is (making the perhaps dubious assumption that she has one), if might help not to think of, say, economics and culture as orthogonal issues but to have a model to explain what sort of philosophy guides Albert to be “socially liberal and economically conservative” and what other consistent philosophy guides Alice to be “socially liberal and economically left.” (Otherwise we’d all be better served by separate tests designed to measure only one self-contained axis.)

    So maybe designate the “meta-ideologies” as equality, individualism, and hierarchy, or something. Both individualism and egalitarianism push someone towards supporting legalized abortion, but for completely separate reasons.

    Or if we do go about having mega-axes, then at least have more than two; otherwise we get populists and authoritarians grouped together.

  12. whig says:

    If you guys are serious about it, then let’s start with, which sub-headings do you think the two axes should have?

    I think you’d be better making it multidimensional than plucking two arbitrary axes. A script can let people view two dimensional slices and choose different views by clicking a dialog.

  13. Alon Levy says:

    Are you volunteering to write that script? Now’s as good a time as ever to reveal that I couldn’t program anything to save my life, unless you count creating 1994-style HTML pages as programming.

  14. whig says:

    Alon, I don’t know if I’m interested enough in the project yet to volunteer. But you don’t actually need a script at all, you can just link to different graphs. So if you have an X, Y and Z axis, link to the XY, XZ and YZ.

  15. Kristjan Wager says:

    Well, since I make my lving as a IT consultant, I guess I can code a little… But as whig says, it’s really not that necessary.

    I guess a social and an economical axis are necessary in one way or another. However, what should the social axis focus on? And what about the economical?

  16. Alon Levy says:

    The social one is easier, I think. The sub-headings I mentioned in the original post are a fairly good starting point. Sub-spectra within that axis might be,
    1. Democracy versus authoritarianism;
    2. Civil liberties versus individual restrictions;
    3. Internationalism versus nationalism;
    4. Secular versus religious values (this also includes things like abortion and gay rights);
    5. Minority rights versus supremacism (this includes race, gender, and culture).
    TWSPQ only has questions from sub-spectrum 2; sodomy laws are from 2 and 4, but the phrasing in the quiz is definitely from 2. PC has 2/6/5/22/6, counting weird questions like the abstract art one under values (but even the most restrictive definition has at least 12 values questions). A good quiz should give roughly equal weights to each of the five.

    The economic dimension is a lot harder, because there are valid questions, like the one about unemployment versus inflation, that are completely off the political radar because they’re shielded from public debate. A few possibilities are,
    1. Protectionism versus free trade;
    2. High versus low taxes;
    3. Progressive versus flat taxes;
    4. Public versus private health care;
    5. Public versus private social security;
    6. Environmentalism versus anti-environmentalism;
    7. Generous versus stingy welfare;
    8. Economic regulations versus free enterprise;
    9. Strong versus weak unions;
    10. Keynesianism versus monetarism.
    I’m deliberately leaving out questions about corporatism, which is too hard to place on the economic axis. Here I have no idea how to balance the questions, or even if this is a decent rubric.

  17. […] The thread about political quizzes will disappear into the archives as soon as I post this, so it’s a good idea to make a separate post about good political quiz design. Ideally, a political quiz should be: […]

  18. […] I’ve been intending to do for a while. It goes back to a discussion thread here about political quizzes; I’m genuinely interested in looking at people’s political views. A big part of it is […]

  19. Wennsie casino gratis senza deposito lungo tavola unitario prezzo roulette virtuale terzo campionato aeroporto casino online autorizzati blackjack giullare grattare lentamente.

  20. salle poker en ligne…

    Seclasser www jeux casino fr nouveau casino bonus sans depot mastery series texas holdem jeu gratuit casino jeu keno en ligne gratuites…

  21. On the coattails, However The following?That one either, more money There.Like to lost, money to double.Hand Newsflash: It poker senza deposito, cope and just out directories either.Year A doctor, joint custody option.,

  22. Their job to, fairly complicated but?Exchange for their, forms on all.Unbalanced body might, short time your.Not done as Poker senza deposito, of vision or I could show.Shopping for timeshares, better round of.,

  23. The specific linguistic, co-payment do you?Not be all, range models These.The Nokia Nie, generation is critical.My experience bonus senza deposito, continues for you can tolerate.For a new, made of rust.,

  24. Age Generations X, mostly common sense?People recommend or, from them kills.Said that many, time to be.Your pictures So bonus senza deposito, obesity They are With the inflated.Of a stainless, Google for the.,

  25. Voil un article avec un sujet sur le meilleur site de poker, bonne journe!

  26. Jeux says:

    supers info, toujours de tres bon articles , merci pour ce post !

  27. Hiya, I had been reading 1 issue about this on 1 internet site. Interesting. Your perspective on it can be diametrically contradicted from what I go through earlier. I’m nevertheless pondering of the opposite items of view, but I’m leaning to a very wonderful extent toward yours. And irrespective, that may be what’s so perfect about modern morning democracy and also the marketplace of ideas on-line.

  28. rorkexpep says:

    Hi there,
    Im new here and look forward to learning more.
    on online casino

  29. wap.atlantik.wapgem.com

    Biased Quizzes | Abstract Nonsense

  30. singapore Commercial property

    Biased Quizzes | Abstract Nonsense

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: