Wise Use of Null Hypothesis Tests

A Practitioner's Handbook

1st Edition - October 14, 2022
Imprint: Academic Press
Author: Frank S Corotto
Language: English
Paperback ISBN:
9 7 8 - 0 - 3 2 3 - 9 5 2 8 4 - 2
eBook ISBN:
9 7 8 - 0 - 3 2 3 - 9 5 2 8 5 - 9

Few students sitting in their introductory statistics class learn that they are being taught the product of a misguided effort to combine two methods into one. Few students le… Read more

Purchase options

BLOOM INTO SPRING SAVINGS

Save up to 25% on books and eBooks!

Shop now

Institutional subscription on ScienceDirect

Request a sales quote

Resources

Textbook support for instructors(opens in new tab/window)

Few students sitting in their introductory statistics class learn that they are being taught the product of a misguided effort to combine two methods into one. Few students learn that some think the method they are being taught should be banned. Wise Use of Null Hypothesis Tests: A Practitioner’s Handbook follows one of the two methods that were combined: the approach championed by Ronald Fisher. Fisher’s method is simple, intuitive, and immune to criticism.

Wise Use of Null Hypothesis Tests
is also a user-friendly handbook meant for practitioners. Rather than overwhelming the reader with endless mathematical operations that are rarely performed by hand, the author of Wise Use of Null Hypothesis Tests emphasizes concepts and reasoning. In Wise Use of Null Hypothesis Tests, the author explains what is accomplished by testing null hypotheses—and what is not. The author explains the misconceptions that concern null hypothesis testing. He explains why confidence intervals show the results of null hypothesis tests, performed backwards. Most importantly, the author explains the Big Secret. Many—some say all—null hypotheses must be false. But authorities tell us we should test false null hypotheses anyway to determine the direction of a difference that we know must be there (a topic unrelated to so-called one-tailed tests). In Wise Use of Null Hypothesis Tests, the author explains how to control how often we get the direction wrong (it is not half of alpha) and commit a Type III (or Type S) error.

Cover image
Title page
Table of Contents
Copyright
Epigraph
About the author
What makes this book different?
This is not a mathematics book
Conventional books get it wrong
Ronald Fisher got it right, and his method is simple
What conventional books leave out
How to use this book
Chapter one. The conventional method is a flawed fusion
Abstract
1.1 Three statisticians, two methods, and the mess that should be banned
1.2 Wise use and testing nulls that must be false
1.3 Null hypothesis testing in perspective
Notes
Chapter two. The point is to generalize beyond our results
Abstract
2.1 Samples and populations
2.2 Real and hypothetical populations
2.3 Randomization
2.4 Know your population, and do not generalize beyond it
Notes
Chapter three. Null hypothesis testing explained
Abstract
3.1 The effect of sampling error
3.2 The logic of testing a null hypothesis
3.3 We should know from the start that many null hypotheses cannot be correct
3.4 The traditional explanation of how to use p
3.5 What use of α accomplishes
3.6 The flawed hybrid in action
3.7 Criticisms of the flawed hybrid
3.8 We should test nulls in a way that answers the criticisms
3.9 How to use p and α
3.10 Mouse preference, done right this time
3.11 More p-values in action
3.12 What were the nulls and predictions?
3.13 What if p=0.05000?
3.14 A radical but wise way to use p
3.15 0.05 or .05? p or P?
Notes
Chapter four. How often do we get it wrong?
Abstract
4.1 Distributions around means
4.2 Distributions of test statistics
4.3 Null hypothesis testing explained with distributions
4.4 Type I errors explained
4.5 Probabilities before and after collecting data
4.6 The null’s precision explained
4.7 The awkward definition of p explained
4.8 Errors in direction
4.9 Power and errors in direction
4.10 Manipulating power to lower p-values
4.11 Increasing power with one-tailed tests
4.12 Power and why we should we set α to 0.10 or higher
4.13 Power, estimated effect size, and type M errors
4.14 How can we know a population’s distribution?
Notes
Chapter five. Important things to know about null hypothesis testing
Abstract
5.1 Examples of null hypotheses in proper statistics books and what they really mean
5.2 Categories of null hypotheses?
5.3 What if is important to accept the null?
5.4 Never do this
5.5 Null hypothesis testing as never explained before
5.6 Effect size: what is it and when is it important?
5.7 We should provide all results, even those not statistically “significant”
Notes
Chapter Six. Common misconceptions
Abstract
6.1 Null hypothesis testing is misunderstood by many
6.2 Statistical “significance” means a difference is large enough to be important—wrong!
6.3 p is the probability of a type I error—wrong!
6.4 If results are statistically “significant,” we should accept the alternative hypothesis that something other than the null is correct—wrong!
6.5 If results are not statistically “significant,” we should accept the null hypothesis—wrong!
6.6 Based on p we should either reject or fail to reject the null hypothesis—often wrong!
6.7 Null hypothesis testing is so flawed that we should use confidence intervals instead—wrong!
6.8 Power can be used to justify accepting the null hypothesis—wrong!
6.9 The null hypothesis is a statement of no difference—not always
6.10 The null hypothesis is that there will be no significant difference between the expected and observed values—very, very wrong!
6.11 A null hypothesis should not be a negative statement—wrong!
Notes
Chapter Seven. The debate over null hypothesis testing and wise use as the solution
Abstract
7.1 The debate over null hypothesis testing
7.2 Communicate to educate
7.3 Plan ahead
7.4 Test nulls when appropriate, not promiscuously
7.5 Strike the right balance between what is conventional and what is best
7.6 Think outside of the null hypothesis test
7.7 Encourage our audience to draw their own conclusions
7.8 Allow ourselves to draw our own conclusions
7.9 Strike the right balance when providing our results
7.10 Know the misconceptions and do not fall for them
7.11 Do not say that two groups “differ” or “do not differ”
7.12 Provide all results somehow
7.13 Other reformed methods of null hypothesis testing
Notes
Chapter Eight. Simple principles behind the mathematics and some essential concepts
Abstract
8.1 Why different types of data require different types of tests
8.2 Simple principles behind the analysis of groups of measurements and discrete numerical data
8.3 Drawing conclusions when we knew all along that the null must be false
8.4 Degrees of freedom explained
8.5 Other types of t tests
8.6 Analysis of variance and t tests have certain requirements
8.7 Do not test for equal variances unless …
8.8 Simple principles behind the analysis of counts of observations within categories
8.9 Interpreting p when the null hypothesis cannot be correct
8.10 2×2 Designs and other variations
8.11 The problem with chi-squared tests
8.12 The reasoning behind the mathematics
8.13 Rules for chi-squared tests
Notes
Chapter Nine. The two-sample t test and the importance of pooled variance
Abstract
Chapter Ten. Comparing more than two groups to each other
Abstract
10.1 If we have three or more samples, most say we cannot use two-sample t tests to compare them two samples at a time
10.2 Analysis of variance
10.3 The price we pay is power
10.4 Comparing every group to every other group
10.5 Comparing multiple groups to a single reference, like a control
10.6 Is all of this a load of rubbish?
Notes
Chapter Eleven. Assessing the combined effects of multiple independent variables
Abstract
11.1 Independent variables alone and in combination
11.2 No, we may not use multiple t tests
11.3 We have a statistical main effect: now what?
11.4 We have a statistical interaction: things to consider
11.5 We have a statistical interaction and we want to keep testing nulls
11.6 Which is more important, the main effect or the interaction?
11.7 Designs with more than two independent variables
11.8 Use of analysis of variance to reduce variation and increase power
Notes
Chapter Twelve. Comparing slopes: analysis of covariance
Abstract
12.1 Analysis of covariance
12.2 Use of analysis of covariance to reduce variation and increase power
12.3 More on the use of analysis of covariance to reduce variation and increase power
12.4 Use of analysis of covariance to limit the effects of a confound
Note
Chapter Thirteen. When data do not meet the requirements of t tests and analysis of variance
Abstract
13.1 When do we need to take action?
13.2 Floor effects and the square root transformation
13.3 Floor and ceiling effects and the arcsine transformation
13.4 Not as simple as a floor or ceiling effect—the rank transformation
13.5 Making analysis of variance sensitive to differences in proportion—the logarithmic transformation
13.6 Nonparametric tests
13.7 Transforming data changes the question being asked
Notes
Chapter Fourteen. Reducing variation and increasing power by comparing subjects to themselves
Abstract
14.1 The simple principle behind the mathematics
14.2 Repeated measures analysis of variances
14.3 Multiple comparisons tests on repeated measures
14.4 When subjects are not organisms
14.5 When repeated does not mean repeated over time
14.6 Pretest-posttest designs illustrate the danger of measures repeated over time
14.7 Repeated measures analysis of variance versus t tests
14.8 The problem with repeated measures
Notes
Chapter Fifteen. What do those error bars mean?
Abstract
15.1 Confidence intervals
15.2 Testing null hypotheses in our heads
15.3 Plotting confidence intervals
15.4 Error bars and repeated measures
15.5 Plot comparative confidence intervals to make the overlap myth a reality
Notes
Appendix A. Philosophical objections
Abstract
A.1 Decades of bitter debate
A.2 We want to know when we are wrong, not how often
A.3 Setting α to 0.05 does not mean that 5% of all null-based decisions are wrong
A.4 There are better ways to analyze and interpret data
A.5 The fallacy of affirming the consequent
A.6 Some say our method cannot be used to determine direction
Notes
Appendix B. How Fisher used null hypothesis tests
Abstract
B.1 Why follow my advice?
B.2 Fisher tested for direction
B.3 Others did too
B.4 Fisher believed α should vary according to the circumstances
B.5 Fisher came close to saying there should be no α at all
B.6 In practice, Fisher did not categorize outcomes
B.7 Fisher’s language answers many criticisms of null hypothesis testing
B.8 Except for Fisher’s use of “significant”
B.9 Fisher’s inconsistency explained
B.10 Fisher’s thinking expressed in one word
B.11 We have come a long way since Fisher, but the wrong way?
Notes
Appendix C. The method attributed to Neyman and Pearson
Abstract
C.1 Neyman and Pearson with Pearson
C.2 Neyman and Pearson without Pearson
C.3 An important limitation
C.4 Alternatives are always infinitely numerically precise
C.5 The method step-by-step
C.6 The method’s influence on the flawed hybrid
C.7 The method’s fate in the world of the flawed hybrid
C.8 Power spreads its wings
C.9 Neyman et al.’s method has no place in science
Notes
Index

Life Sciences

Physical Sciences & Engineering

Social Sciences & Humanities

Health