I will try to correct some common misconceptions.

There is still a
confusing conflation in this thread between statistical significance and clinical
siignificance. In large studies it is possible to show statistical significance
(a.k.a. confidence) of a  treatment effect whose benefit is so small that it is
clinically of little significance (a.k.a. unimportant). It small studies, it is very
common, and unfortunately often published, that no statistical significance is found
(p>0.05)  in a study when careful reading of the confidence interval shows
that clinically significant (important) effects have not yet been excluded. In one
recent study I read, the authors cited confidence intervals of 0.4 to 20 as the
relative odds of the proposed versus the standard treatment causing harm. Since
these confidence interval included 1, they stated "no significant difference
between the two treatments." A 20 fold increase in complications is certainly
clinically important!  The proper interpretation of that experiment should not
be "showed no difference" but read "our experiment was woefully
inadequate to detect important differences, and therefore showed nothing."

The difference is type I and type II error in statistics.

The proper
method is to define some level of difference in therapy you would consider
important, then use Type II statistics to see if you can prove, with adequate
confidence, that you have shown any actual difference in treatments must be less
than that threshold. Beware, if you do that, you will be less likely to be able to
publish some nonsensical article which really showed nothing but gets on your resume

A useful way to debunk this error is to ask anyone asserting that
they found "no difference" what sort of difference they could have
detected." If they can't answer that reasonably, they didn't analyze it
correctly. At 100 yards without my glasses, my myopic eyes can't tell the difference
between Hillary Clinton, Arnold Schwarzenegger and Shaquille O'Neal. My inability to show or detect a visual difference
should not be interpreted as proving no difference exists.

Kevin Powell
[log in to unmask]
Saint Louis Children's Hospital

For more information, send mail to [log in to unmask] with the message: info PED-EM-L
The URL for the PED-EM-L Web Page is: