CyberTech Rambler

January 15, 2008

Incorrect conclusion, besides alienating the very people we need to convert

Filed under: Uncategorized — ctrambler @ 12:46 am

Some of the analysis around September ballot on OOXML, although done with good intentions, i.e., against a a bad standard being adopted by ISO, are quite simply misguided. The first problematic one I see was from EFFI about corrupted countries are more likely to support OOXML, which uses rather crude analysis. This made it not as bad as the latest one from Digistan which try to measure the relationship between GDP and the voting pattern. They erred spectacularly as they uses statistics that might not support their conclusion.

The problem? Alienating the very people one have to win over. Don’t get me wrong. I do not condone corruption. It is bad. FULL STOP. We need to stop it. FULL STOP. NO IF…, NO BUT… There is only one sure way to kill corruption: Make sure corruption does not pay. Hit both parties: the people who offers and the people who accept the bride. To penalize the people who offers bribe, fight fair and make sure the person who offers the bribe does not get what they want. On the way there, keep one’s eyes and ears open and expose particular corrupted practices from manipulation of rules (and room size) to outright “bribing” and whereever possible, punish those who accept the bribe. The battle here is to unearth and publicize incidents of wrongdoing by showing evidence of it happening. General statement like “corrupted countries are more likely to support OOXML” group is unproductive. It makes it sounds like you are shouting slogan, or is simply a sore loser. Worst of all, it catches countries that geniuinely believe OOXML is OK in the net as well. When this happens, all we do is to supply ammo to others that will jump on the opportunity to call us zealots. Worse, it alienate the “uncorrupted” countries whose opinions we need to fight to change.

Although seriously misguided, at least the EFFI is looking at something universally regarded as bad: corruption. The Digistan article target? Wealth. I know that its intention is to imply a big OOXML supporter is trying to buy its way, but the message can equally be perceived as if you are poor, you WILL be bought. Taking into account that the global wealth picture is shaped by history, the sensitivity of a lot of people who thinks (rightly or wrongly) that they are put into that situation by the force of history against them. Since Digistan is a Belgian website, there is an extra perception that it is yetanother exercise by rich country to preach to poor countries about corruption and the proof that rich countries are so snobbish that they believe they are immune from being bought (Not true, rich countries just cost more to bribe). Does this helps our cause?

Worse of all, the statistics does not appear to support their conclusion that the alternative hypothesis, i.e., “The average of GDP per capita of the countries who voted for the OOXML proposal is significantly lower than the GDP per capita of those who voted against it. They used the Wilcoxon rank sum test, which is equivalent to most people’s favorite statistics test, the Student T-test. In fact, being a non-parametric test, Wilcoxon rank sum test is probably better than the standard Student T-test because it does not make a fundamental assumption with t-test, i.e., that the underlying distribution is a gaussian distribution. However, every statisticians, every statistics teachers and every statistics book will tell you that the test will only allows you to reject or accept the null hypothesis and says NOTHING whatsoever about the alternative hypothesis.

Hence, if you take their statistics test, i.e., that “there is no significant difference between the average GDP per capita of countries who voted for OOXML and countries who voted against it”, their test appears to support the fact that this hypothesis is not true, leading to the conclusion that “there is significant difference between the average GDP per capita of countries who voted for OOXML and countries who voted against it”. It says nothing about whether the average GDP per capita of countries who voted for OOXML is higher or lower than those who voted against it. If they wanted to answer the question, the test will have to be reformulated to be “the average GDP per capita of countries who whoted for OOXML is significantly lower than those who voted against it”, which they did not. Furthermore, although they publish their raw data, they did not publish how they measured their test statistics. Making it difficult to check whether their test statistics support support their conclusion. My belief (without proof) is that they simply use the GDP values as the test statistics. If so, it is unlikely that their statistics test is different from what my proposed reformulation of the null hypothesis will requires.

(Aside):

Statistics is maths, and maths will always give you an answer. The role of anyone performing the statistical analysis is to make sure you collect the correct data, perform the correct mathematical manipulation of your data and to perform the correct statistical test.

That is not easy, and that is why all respectable journals will require you to expose your test method and walk the reader through what you did. Even with good journals, sometimes their referees can sometimes miss it. Recently, I think we caught thre instance of this happening. The question to answer is “Is group A behave differently from group B under drug X?” A research group published at least 4 papers on this. The first three appears to do two within-group analysis and use them to draw a conclusion. In other words, they ask “Is group A behave differently when drug X is used (with respect to placebo)” and the same for group B. What they found is that group A is behaving differently when drug X is applied, but group B does not. They link the results together and say that therefore group A is behaving differently from group B. I concur with my colleague that this might not be the case. Statistically speaking they haven’t compared group A with group B. They should had compared groupA’s drug A performance (w.r.t placebo) with that of groupB. The difference is subtle and difficult to grasp and explain. And I believe the referess miss this or probably overlook it. The statistical test I believe they should had done has been the standard test for at least 5 years before the first paper was publish. Therefore, they probably assumed that they did the expected test, or more likely, decided that their conclusion is correct despite the imperfection in their analysis method. I think the referrees saw the later as in their fourth paper, a new co-author appears to have spotted and corrected this problem. Good for them (and the research field.) What really struck me is that, if you ask me when I completed my PhD and before I take on this current job of mine, I will tell you this is OK.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: