Simmons JP, Nelson LD, Simonsohn U. (2011) False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science. 2011 Nov;22(11):1359-66. Epub 2011 Oct 17.
This article addresses the elephant in the room of statistical analysis in the social sciences - how perfectly legitimate analysis decisions made by researchers can lead to ever increasing chances of making Type I errors (finding significance where none exists). Using basic modelling techniques, it was shown that simple analysis decisions can very quickly increase the chances of making a Type I error from 5% to over 60%. In a follow up paper (http://www.ssrn.com/abstract=2160588), the authors suggest a simple solution - adding, and acting on, 21 words to every methods section "We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study."
This article addresses the elephant in the room of statistical analysis in the social sciences - how perfectly legitimate analysis decisions made by researchers can lead to ever increasing chances of making Type I errors (finding significance where none exists). Using basic modelling techniques, it was shown that simple analysis decisions can very quickly increase the chances of making a Type I error from 5% to over 60%. In a follow up paper (http://www.ssrn.com/abstract=2160588), the authors suggest a simple solution - adding, and acting on, 21 words to every methods section "We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study."
No comments:
Post a Comment