POWER CALCULATIONS through DATA MODELING
This application is designed to allow users to run power calculations of experimental designs beyond a simple t-test. Our tool allows users to generate random model data based on user inputs and Gaussian distributions. Multiple treatment groups, as well as categorical blocking factors can be incorporated into the experimental design. This data is run through a general linear model with pairwise comparison tests and the generation of multiple sets of random data allows for an estimation of statistical power. This tool is most useful for checking experiment statistical power, or generating model data that can be used in other statistics software (Ex. SAS, JMP) to run test analyses.
Program features are further explained below and the program is available to download HERE.
A more detailed description of the program can be found in the user manual HERE.
INPUT PARAMETERS
Sample Size: Sample size is inputted as total measurements across all groups. To estimate sample size we recommend using Mead’s Resource Equation (See suggested reading below)
Multiple treatments: Treatment is the primary fixed effect to be compared in your experiment. Rather than the two means usually compared in t-test based power calculations, the number of means to compare can be specified with group names and estimated means.
Blocking factors: Blocking factors are known sources of variance in your experiment that are not of primary interest. The program can accommodate multiple blocking factors and allows for the user to assign an estimated standard deviation to each effect. Then it assigns a deviation from the mean to each group within the blocking factor which is incorporated into the data generation. These can include mouse strain, sex, cage assignment, or other factors associated with your experiment.
DISTRIBUTE GROUPS and GENERATE DATA MODELS
Distributing Groups: You can distribute the assignments of groups to each of your measurements as you wish by hand. You can do so by filling in the lines directly in the app, loading in a correctly-formatted CSV, or utilizing the auto-gen feature. The auto-gen grid feature can help utilize perfect factorial design for smaller sample sizes (cartesian product assignment) or help evenly distribute assignments for larger experiments that cannot have perfect factorial design (quasi-random low discrepancy sequence assignment).
Generating Values: After filling out the input parameters and distributing assignments of groups, you can generate dependent variable values. You can do this multiple times to simulate multiple runs of the experiment. The more runs you generate, the better the power estimation will be (with diminishing returns).
GENERAL LINEAR MODEL AND POWER CALCULATION ESTIMATIONS
General Linear Model: The general linear model is a special case of the generalized linear model where the link function and response follow a normal distribution. General linear models can be used to evaluate the simultaneous effects of multiple explanatory variables on a continuous dependent variable - including both categorical and continuous fixed effects. Although, both continuous and categorical variables can be used as predictors in a general linear model, the model data generated in this app can only incorporate categorical variables at this time.
Estimate Power: Depending on how many sets of model data you have generated (runs), the program will parse the F-test and pairwise t-test results and estimate the power of your experiment based on the parameters you have supplied. F-test results can be used to determine if the inclusion of treatment (the primary effect) as a variable explains variance in the dependent variable (i.e. is there an overall effect of treatment). In addition to knowing whether there is an overall effect of treatment on the dependent variable, we are likely also interested to know if there are differences between treatments means. This can be tested using multiple pair-wise t-tests, and a Holm-Sidak correction for multiple comparisons.
Ready to download? The program is available to download HERE.
A more detailed description of the program can be found in the user manual HERE.
This tool was built by Peyton Chen (peytonc@stanford.edu). The project source code is available here on Github.
SUGGESTED READING
Mead R. 1988. The design of experiments: statistical principles for practical applications. New York (NY): Cambridge University Press.
Festing MF, Overend P, Gaines Das R, Cortina-Borja M, Berdoy M. 2002. The design of animal experiments: reducing the use of animals in research through better experimental design. London (United Kingdom): The Royal Society of Medicine Press Limited.
Gaskill, B. N, Garner, J. P. 2020. Power to the People: Power, Negative Results and Sample Size. JAALAS, 59, 1-8.