Matthew Fritz (University of Nebraska--Lincoln)

• A faculty member from another department recently reached out with questions about the graduate-level introductory statistics course I teach. This is common, as graduate programs across the university use my course to fulfill their statistics requirements. This faculty member wondered whether I could add “a few additional topics” specifically for their students. When I met with them, they explained that they led an accredited professional/clinical master’s program whose graduates needed to read medical research articles to support evidence-based decision making for clients, so their students needed to learn about the statistics used in these articles. When I asked specifically which statistics their students needed that were not already covered in my course, the faculty member gave me an extensive list of topics – enough to fill an intensive semester-long course in biostatistics. After getting over my initial surprise, I explained that my course is designed to introduce statistics to social and behavioral science students who truly have no prior experience with statistics. In my opinion, it seemed that what their students needed was a second course that covered their desired topics by expanding on the concepts covered in my introductory course. Without hesitation, they replied that was not possible as every credit in their degree program was already accounted for, thanked me for my time, and ended the meeting.

  While I have previously written about my frustration with the unrealistic expectation that everything psychology students need to know about statistics can be covered in a single introductory course (Fritz, 2020), something this faculty member said stuck with me. To paraphrase, they said:

  you know… you would have time to cover these other topics if you did not focus so much on computing statistics; our students do not need to compute statistics, our students just need to be able to understand the results sections when they read research articles – they are consumers, not producers.

  They were referring to the fact that I require students in my introductory course to compute the statistics discussed in class, both by hand and with statistical software, using small data sets. Many people have discussed the benefits of developing a student’s statistical and methodological literacy as opposed to focusing solely on calculations; see for example Grosofky, Wagge, and Branch’s E-xcellence post from August 2024. I redesigned my introductory statistics course to focus on statistical literacy years ago (Fritz, 2015) and continue to revise the course to provide a better understanding of how statistics fit within the broader scope of scientific research (Fritz, 2024). But I had never considered the idea that you could teach a student to be a consumer of statistics without also teaching them how to produce those same statistics.

              The first thought that came to mind after hearing this comment was whether you could teach someone to read (consume text) without also teaching them to write (produce text), as the two seem inseparably intertwined to me. A better analogy, however, is that to me, teaching a “consumer statistics” course that removes all calculation is like teaching a cooking class where students read recipes and look at pictures of food, but do not actually do any cooking. As I continued to consider the comment, my thoughts solidified into a question: what is the benefit, if any, of learning to produce statistics for a student who will only consume statistics? My answer is that I do not want students to just consume statistics (i.e., just read results sections), I want them to critically consume statistics so they can make decisions based on those statistics (i.e., evaluate the veracity of the conclusions in the discussion section), regardless of whether they are conducting research or supporting evidence-based practice (Spencer, Detrich, & Slocum, 2012). That is, students need the ability to be “amiably skeptical of the research you read, to question it even as you realize how much you depend on it” (Booth et al., 2016, p. 10). But how do I teach amiable skepticism about statistics without also teaching them to produce those statistics?

  At the A Meeting of Methodologies conference in March 2025 (Fouladi & Fritz, 2025), an event I co-organized, I attempted to get some clarity on this question by asking the senior participants a different question: how do you know what you know about methods? Most of us, whether we consider ourselves methodologists or just methods enthusiasts, had taken multiple courses and workshops on advanced methods, but the consensus was that the true understanding of these methods had come from experience. Being well-versed in methods, quantitative or qualitative, is not just about knowing a lot of different methods, but knowing for each method which rules you can break, which rules you can bend, and which rules must be followed to maintain the integrity of the conclusions drawn from that method. Years of applying these methods in suboptimal conditions – measures with low reliability, non-normal data, missing data, small sample sizes, etc. – had forced us to learn the nuances of these methods. In addition, when faced with multiple methodological problems that cannot all be addressed in a single model, our experience had taught us which problems to prioritize. Based on this understanding, when we read research articles, we can determine whether a specific method was applied correctly and whether the conclusions based on the results are valid. So how did we know what we knew? We had spent years in the research “kitchen” learning what happens when you change a recipe to accommodate a food allergy, or that even when you follow a recipe exactly, sometimes the soufflé still does not rise. (This is probably why my favorite recipes come from American’s Test Kitchen – chefs in an actual research kitchen baking a hundred versions of the same cookie to understand the effects of putting in less baking powder or baking at a higher temperature.)

              Gaining 16+ years of experience in 16 weeks is impossible. But per Kolb’s Experiential Learning Theory (1984), in addition to experiential learning being related to motivation and engagement (Kong, 2021), having students actually compute statistics does give them at least some Concrete Experience and Active Experimentation, while having them answer questions about this experience allows for Reflective Observation, all of which deepens their understanding of both how the statistics work and ways in which they can break (Abstract Conceptualism). This is the same reason why culinary programs require hands-on experience prepping and cooking food in a kitchen to graduate. Experiential learning is especially important for students learning statistics because being a critical consumer also means knowing how statistics can be purposefully manipulated á la the classic book How to Lie with Statistics (Huff, 1954), which was famously banned by the U.S. Department of Veterans Affairs (Anderson & Richardson, 2017) after it was mistaken for a how-to as opposed to a how-not-to.

  To that end, here are five lessons that I believe are essential for critical consumers of statistics that are best learned by producing statistics:

1. The operational definition of a construct greatly impacts the statistical results of a study, such that two studies using different definitions can have very different results that are both accurate from a statistical standpoint. I illustrate this by asking students:
   
   Suppose you were interested in studying the relation between fatigue in airline pilots and the number of fatal airplane crashes. Describe specifically how you would measure fatigue.
   
   I then have the students share their answers in class. When no students have exactly the same answer, I ask a follow-up question:
   
   If none of you are measuring fatigue the same way, what impact will that have when we try to compare the results from all your different studies?
   
   
2. A single extreme score can have a large impact, a small impact, or no impact on the results of a study depending on the statistic used and other factors such as sample size. For example, I have students calculate the median and mean of the sets [1, 3, 5, 7, 9] and [1, 3, 5, 7, 99], which shows that the mean changes from 5 to 23, but the median stays the same. Many students (and teachers) use this as evidence that the mean is biased because it is sensitive to extreme scores (by this logic, the median is also biased because it is insensitive to extreme scores), but really this just shows that a.) we should always report both and b.) this is why we should also report measures of variability.
   
   
3. A researcher never knows which sample from the population they actually drew, so they also never know how different their sample statistic is from the population parameter. This is why two studies using different samples can come to contradictory conclusions, even if both studies use the same operational definitions and statistical methods, especially when the sample sizes are small (i.e., sampling error).This can be illustrated by creating a set of data to serve as the population, then having each student randomly draw a sample from this population, compute the same statistic, and then compare their results; the repeated sampling can be easily accomplished using readily available simulation tools (e.g., Revelle, 2020).
   
   
4. Unless the effect is exactly zero, failing to reject the null hypothesis does not mean no effect was found—it just means that the researcher was not able to rule out the possibility that the effect they did find was not just due to sampling error based on their desired level of confidence. I illustrate this by having students compute a z statistic and Cohen’s d using a sample mean of 51, a population mean of 50, a population variance of 10, and sample sizes of 25 (z = 0.5, d = 0.1) and 400 (z = 2.0, d = 0.1). For alpha = 0.05, the effect is not statistically significant with the smaller sample size, but is statistically significant with the larger despite the effect size being identical in both samples.
   
   
5. The results reported in a study are always based on specific statistical models and if those models were changed in any way, the results can completely change. I have students estimate a single-predictor regression model with a significant predictor, then estimate a two-predictor regression model where adding the second predictor causes the first predictor to become nonsignificant. I also have students estimate a one-factor ANOVA model where the main effect is nonsignificant, then estimate a two-factor ANOVA model with a significant interaction such that the simple main effects of the first factor at levels of the second factor are equal in strength, but opposite. 

All of these are basic concepts that are covered in most introductory statistics courses, but in my opinion, these are all concepts that require  “playing with the numbers” to truly understand. A colleague who kindly reviewed this post asked whether just lecturing on these topics could achieve a similar outcome. I do lecture on all of these topics and show in class how the statistics are calculated with example data sets. But then I have the students go and calculate the statistics again using different data sets with different variables with the goal of seeing what happens when you change things. I believe this active learning reinforces what we covered in lecture and helps students understand the content more deeply. And I am not alone in this belief – for example, Morris, Murray, and Cook (2017) call experiential learning in statistics education “essential” and have developed an NSF-funded program called SCHOLAR to provide extra experiential learning opportunities for statistics students (Moss, 2019).

I would also point out that experiential learning is highly utilized in other areas of psychology. Research methods courses often require students to compete a research paper, not just talk about it in class. And clinical, counseling, and school psychology doctoral students must apply what they learned in their coursework by completing over 1,000 hours of experiential learning in the form of supervised practicums just to be competitive for their internships (Dittman Tracey, 2006) where they complete an additional 2,000 of hands-on experiential learning. Some may argue that comparing an introductory statistics class to doctoral-level clinical training is absurd, but these students all completed an introductory statistics course at some point. And remember the circumstance that prompted this post—a professional/clinical program whose graduates need to critically consume research in order to make decisions for clients for whom my course was going to be their only statistics course.

Many of us talk about “meeting students where they are”, but the point of that is to help them get to where they need to go, not for us to stay there with them (Hunter, 2026). In my opinion, a “consumer statistics” course cannot get these students to where they need to be. Regardless, I hope at least some of this resonates with you and that the next time someone complains about calculations in a statistics course, this will help you explain why producing statistics is necessary, even for students who will “only” be consumers, not producers. Good luck!

References:

Anderson, K., & Richardson, M. (2017, September 4). How to lie (to Congress) with

statistics. Statistics Teacher. https://www.statisticsteacher.org/2017/09/15/how-to-lie-to-congress-with-statistics/

Booth, W. C., Colomb, G. G., Williams, J. M., Bizup, J., & Fitzgerald, W. T. (2016). The

craft of research (4^th ed.). The University of Chicago Press.

Dittman Tracey, M. (2006, January). How many practicum hours do training directors

want? gradPSYCH Magazine. American Psychological Association. https://www.apa.org/gradpsych/2006/01/practicum

Fouladi, R. T., & Fritz, M. S. (Co-Chairs/Organizers). (2025, March 29-30). A Meeting

of Methodologies (AMOM) Conference: Advancing methods development, application, and teaching in psychology, the behavioural, and social sciences (2^nd meeting). Simon Fraser University, Burnaby, British Columbia, Canada. (145 registered participants from 6 continents) https://www.sfu.ca/psychology/research/mml/about/AMOM.html

Fritz, M. S. (2015). Benchmark portfolio for EDPS 859 Statistical Methods – Spring

2015. Peer Review of Teaching Project, University of Nebraska-Lincoln. Available to download at: https://digitalcommons.unl.edu/prtunl/80/   

Fritz, M. S. (2020). Singletons: Re-evaluating course objectives when an

introductory statistics course is a student’s only statistics course. In J. L. Rodgers (Ed.), Teaching statistics and quantitative methods into the 21^st century (pp. 87-101). Routledge. https://doi.org/10.4324/9780429442810

Fritz, M. S. (2024, December). Are we actually introducing students to statistics?

Teaching introductory statistics as a course in statistical literacy and critical thinking [Concurrent talk]. PsychTERMS: Teaching to Enhance Research Methods & Statistics in Psychology Conference (virtual).

Grosofky, A., Wagge, J. R., & Branch, J. G. (2024, August 2). PLUMS: Psychological

literacy for undergraduate methods and statistics. E-xcellence in Teaching Blog. https://teachpsych.org/E-xcellence-in-Teaching-Blog/13389637

Huff, D. (1954). How to lie with statistics. W. W. Norton & Company.

Hunter, W. (February 2, 2026). Stop meeting students where they are: What I learned

when I finally started assigning the hard reading again. The Atlantic. Available at: https://www.theatlantic.com/ideas/2026/02/youth-reading-books-professors/685825/

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and

development (Vol. 1). Prentice-Hall.

Kong, Y. (2021). The role of experiential learning on students’ motivation and classroom

engagement. Frontiers in Psychology, 12, 771272. https://doi.org/10.3389/fpsyg.2021.771272

Morris, T., Murray, C., & Cook, T. (2017, July). Transforming undergraduate statistics

education through experiential learning: It’s essential! Joint Statistical Meeting, Baltimore, MD. Available at: https://ww2.amstat.org/meetings/JSM/2017/OnlineProgram/AbstractDetails.cfm?abstractid=323782

Moss, E. (November 19, 2019). Transforming undergraduate statistics education

through experiential learning. Math Values Blog. Mathematical Association of America. Available at: https://maa.org/math-values/transforming-undergraduate-statistics-education-through-experiential-learning/

Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard:

Advantages of longhand over laptop note taking. Psychological Science, 25, 1159-1168. https://doi.org/10.1177/0956797614524581

Revelle, W. (2020). Teaching research methods using simulation. In J. L. Rodgers

(Ed.), Teaching statistics and quantitative methods into the 21^st century (pp. 217-237). Routledge. http://dx.doi.org/10.4324/9780429442810-16

Spencer, T. D., Detrich, R., & Slocum, T. A. (2012). Evidence-based practice: A

framework for making effective decisions. Education and Treatment of Children, 35, 127-151. http://www.jstor.org/stable/42900152