A GUIDE TO DETERMINING SAMPLE SIZE A PRIORI INTRODUCTION

A GUIDE TO DETERMINING SAMPLE SIZE A PRIORI
INTRODUCTION
A priori sample size determination is a complex task in particularly when the exact
population size is unknown. It requires a set of priori assumptions that most of the time is
not known to market researchers with absolute accuracy.
It is also specific to the type of statistical methodology1 that the research will aim to utilise
to answer unique market research questions2.
It also most often than not is dependent on the type of specifics of the statistical
methodology. For instance, for a logistic regression model, the research question maybe to
estimate the sample size necessary to achieve, in a two-sided test with a power of at least
0.95. Then the knowledge that the researcher need to (a priori) have are the distribution of
the population and the odds ratio3.
The following discussion is based on estimating the adequate sample size a priori for a
number of regression methods. The most commonly used regression types by Roy Morgan
are Multiple Linear Regression, Logistic Regression, Multinomial Logistics Regression and
Ordinal Regression.
MULTIPLE LINEAR REGRESSION
Linear regressions are the most commonly used regression techniques in the industry.
Therefore there is a large body of work in the literature when it comes to estimating a priori
sample sizes in relation to linear regressions. Brooks, G. and Barcikowski, R. (2012) point
two distinct approaches to estimating a priori sample size for multiple linear regressions.
1
T-test (means/proprtions), Odds ratio, Chi-Square and Contingency tables, ANOVA, ANCOVA, MANOVA,
MANCOVA, Correlation, Regression, Discriminant Function Analysis and Cox Regression
2
Dattalo, P. (2008)
3
Hsieh et al. (1998)
1
Conventional Rules
Conventions have evolved that are based on the premise that with a large enough ratio of
subjects to predictors, the sample regression coefficients will be reliable and will closely
estimate the true population values4. For example, Stevens (2002) suggested a ratio of 15
subjects for each predictor (i.e., with 7 predictors 105 subjects are required) and Pedhazur
and Schmelkin (1991) recommended N>30k, where k is the number of predictors5. These
rules lack any measure of effect size, therefore they can only be effective at specific—
usually unknown—effect sizes. For example, a 15:1 subject-to-predictor ratio is acceptable
only if the population squared multiple correlation is moderately large (i.e., over .40);
otherwise, as the true squared multiple correlation decreases, expected cross-validity
reduces (Brooks, 1998).
Statistical Power Methods6
From a statistical power perspective, Multiple Linear Regression provides several alternative
statistical significance tests that can be the basis for sample size selection7. These methods
incorporate the effect size8 as well as the statistical power parameters into the equation.
REGRESSION METHODS BASED ON ESTIMATING ODDS
Regression methods based on estimating Odds (i.e. Logistic, Multinomial Logistic and
Ordinal Regressions) require larger sample sizes than the simpler linear regression methods.
This is due to the fact that maximum likelihood estimates are less accurate than ordinary
least squares (i.e., simple linear and multiple linear regression models)9.
For instance, to achieve a 95% statistical power in a multiple regression method with less
than 10 independent variables requires about 100 cases whereas to reach the same
statistical power in a logistic regression requires close to 350 cases10.
4
Miller & Kunce (1973), Pedhazur & Schmelkin (1991), Tabachnick & Fidell (2001)
Others have provided rules that combine some minimum value with a subject-to-predictor ratio, including
N>30+10k (Knapp & Campbell-Heider, 1989),N>50+k (Harris, 1985), and N>50+8k (Green, 1991)
6
Statistical power is the probability of rejecting the null hypothesis when the null hypothesis is indeed false
7
Cohen (1988), Cohen & Cohen (1983), Green (1991), Kraemer & Thiemann (1987), Milton (1986)
8
Standardised mean difference between the two groups
9
Hsieh, F.Y. et al. (1998)
10
Author’s own calculations based on Demidenko (2007), Hsieh et al. (1998)
5
2
The example here is only for illustrative purposes and it should not be taken as a standard
guide. Each model requires a unique treatment of its own depending on the ultimate goal of
the model, the properties of the data and the desired specifications of its statistical power.
REFERENCES
Brooks, G. and Barcikowski, R. (2012) Multiple Linear Regression Viewpoints, Vol. 38(2).
Brooks, G. P. (1998). Precision Efficacy Analysis for Regression. Paper presented at the meeting of
the Mid-Western Educational Research Association, Chicago, IL.
Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences, (2nd Edition), Lawrence
Earlbaum Associates, Hillsdale, NJ.
Cohen, J. & Cohen, P. (1983) Applied multiple regression/correlation analysis for the behavioural
sciences (2nd ed.), Hillsdale, NJ: Erlbaum.
Cohen, J., Cohen, P., West, S.G., and Aiken, L.S. (2003) Applied Multiple Regression/Correlation
Analysis for the Behavioral Sciences (3rd edition), Lawrence Earlbaum Associates, Mahwah, NJ.
Dattalo, P. (2008) Determining Sample Size: Balancing Power, Precision, and Practicality, New York:
Oxford University Press.
Demidenko E. (2007) Sample size determination for logistic regression revisited, Statistics in
Medicine 26:3385-3397.
Green, S. B. (1991) How many subjects does it take to do a regression analysis? Multivariate
Behavioral Research, 26, 499-510.
Harris, R. J. (1985) A primer of multivariate statistics (2nd ed.), Orlando, FL: Academic Press.
Hsieh, F.Y., Block, D.A., and Larsen, M.D. (1998) A Simple Method of Sample Size Calculation for
Linear and Logistic Regression, Statistics in Medicine, Volume 17, pages 1623-1634.
Knapp, T. R., & Campbell-Heider, N. (1989) Numbers of observations and variables in multivariate
analyses, Western Journal of Nursing Research, 11, 634-641.
Kraemer, H. C., & Thiemann, S. (1987) How many subjects? Statistical power analysis in research,
Newbury Park, CA: Sage.
Miller, D. E., & Kunce, J. T. (1973) Prediction and statistical overkill revisited, Measurement and
Evaluation in Guidance, 6(3), 157-163.
Milton, S. (1986) A sample size formula for multiple regression studies, Public Opinion Quarterly, 50,
112-118.
3
Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated
approach. Hillsdale, NJ: Lawrence Erlbaum Associates.
Stevens, J. (2002) Applied multivariate statistics for the social sciences (4th ed.). Mahwah, NJ:
Lawrence Erlbaum Associates.
Tabachnick, B. G., & Fidell, L. S. (2001) Using multivariate statistics (4th ed.), Boston: Allyn and
Bacon.
4