This post is a follow-up to my previous post, “Statistical alchemy and the ‘test for excess significance’”. In the comments on that post, Greg Francis objected to my points about the Test for Excess Significance. I laid out a challenge in which I would use simulation to demonstrate these points. Greg Francis agreed to the details; this post is about the results of the simulations (with links to the code, etc.)
In my previous post, I said this:
Greg Francis replied:
To which I replied,
This challenge will very clearly show that my situations are not “impossible”. I can sample them in a very simple simulation. Greg Francis agreed to the simulation:
I further clarified:
…to which Greg Francis agreed.
I have performed this simulation. Before reading on, you should read the web page containing the results:
- Web page (with code) outlining the results: http://learnbayes.org/talks/TES/TESsimulation.html
The table below shows the results of the simulation of 1000000 “sets” of studies. All simulated “studies” are published in this simulation, no questionable research practices are involved. The first column shows (n), and the second column shows the average number of non-significant studies for sets of (n), which is a Monte Carlo estimate of I&T’s (E). As you can see, it is not 2.5.
|Total studies (n)||Mean nonsig. studies||Expected by TES (E)||SD nonsig. studies||Count|
(I have truncated the table at (n=10); see the HTML file for the full table.)
I also showed that you can change the experimenter’s behaviour and make it 2.5. This indicates that the assumptions one makes about experimenter behavior matter to the expected number of non-significant studies in a particular set. Across all sets of studies, the expected proportion of significant studies is expected to be equal to the power. However, how this is distributed across studies of different lengths is a function of the decision rule.
The expression for the expected number of non-significant studies in a set of (n) is not correct (without further very strong, unwarranted assumptions).