Statistical literacy in society

The hottest news in the campus now appears to surround the latest findings from UM’s Centre for Democracy and Elections (UMCEDEL), which put some quarters ill at ease. You can read about the summary of the findings here. Many news portals quote the UMCEDEL study as reporting that 46% of young voters favour DSAI as compared to 39% for DSNR. Hillariously, this study is immediately attacked for being “flawed”, “insufficiently sampled”, etc, nevermind that the centre has previously published very favourable results for the grieving side.

These responses are reactionary and unwise. Fortunately, one can more or less decide whether to believe the findings by doing some research, with the help of some statistics. The study employed 20 enumerators and they sampled 1407 respondents across major ethnic groups randomly (and presumably, reflecting the ethic composition. However, the slides given in UMCEDEL do not explicitly mention this). The method used was face to face interview, using a structured questionaire. This method seems reasonable, and for now let’s suppose that the experimental design is OK.

Let us assume that the question gave three options (two options for each of the politicans, one for “others”). The standard error (SE) computed under the most conservative assumption (equal support for both politicians), is the square root of 1/4n, where n=1407. The margin of error at 95% confidence, is given by about two times of the SE. A simple calculation shows this to be about 2.6% (let’s take 3%).

However, a look at the UMCEDEL slides suggests that they phrased the question in a more complex way. umcedel_slide
This slide shows tabulated results from the UMCEDEL study, from which we can infer that a respondent was presented with the questions:

1) “Is DSAI your favourite leader?” (Yes,No,Unsure)
2) “Is DSNR your favorite leader?” (Yes,No,Unsure)

whereupon, 46% of them answered “Yes” in Question 1, and 39% of them answered “Yes” in Question 2.

The interpretation of this result is not straightforward. There are actually 9 possible combinations here, and it is not clear how to make sense of some of them (e.g. “Yes”, “Yes”). If I were to set the question, I would just ask them who was their favourite leader, and provide three options, the third being “someone else”. One way to analysis the result is to perform the chi-squared test to test the null hypothesis of equal support, e.g. the ratio of support is 1:1 for DSAI:DSNR.

Anyway, if we collapse the “No” and “Unsure” responses in both questions, then the 95% confidence interval of the proportion of young voters favouring DSAI as their leader is 46%, give or take 53% , which gives a low of 4143% and a high of 5149%. For the DSNR case, we have a low of 3436% and a high of 4442%. Note that the confidence intervals overlap, which may indicate that support for the two candidates is not statistically significant. However it is not clear to me how this can be tested – a two sample Z-test for binomial proportions seems tenuous, as both answers are obtained from the same person and we don’t have independence of samples.
P/S: After correcting for the calculation mistake, it seems that the 95% confidence intervals don’t overlap!

After going through this thinking, does the study really tell us much? It is unclear how the “random” sampling was done (hopefully, it was not haphazard sampling, which is frequently taken to be equal to random sampling!). Were the respondents sampled primarily in towns? hamlets? At what time of the day were the studies conducted? Did the respondents trust the professionalism of the interviewers (i.e. trying to guess what the interviewer’s political inclination and go along with it)? What was the nonresponse rate?

Is there a need to get upset with the findings of the study?

About Tsung Fei

A teacher, researcher in the bioinformatics division at the University of Malaya
This entry was posted in Coffee talk, Education. Bookmark the permalink.

4 Responses to Statistical literacy in society

  1. Esmondelwc says:

    1407… considering a 7% difference, seems to be significant to me. But that was just within campus.

  2. Tsung Fei says:

    It actually is. I made a calculation mistake (doubling the margin of error). However, it is still not straightforward to make sense of it, due to the way the questions were phrased.

  3. mashliner says:

    Considering it is only one survey and the pool of respondents wasn’t very big, I don’t think there is a need to get ‘upset’ or defensive about the findings. And as pointed out, several factors can influence the survey (I think the reliability of surveys and questionnaires are always questionable….).

    But hey, it is down to the wire so I guess emotions are running a little high…

    • Tsung Fei says:

      Actually the sample size is OK as the margin of error is about 3%. I guess all parties concerned need to develop better statistical literacy when confronted with published opinion polls like this, just so to avoid embarassing themselves.

Leave a Reply to Esmondelwc Cancel reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s