Power and Size Advantages of Exact Parametric Methods

XPro procedures are based on generalized tests and confidence intervals. This not only delivers superior power for most ANOVA under unequal variances, but also provides substantially better size performance compared to traditional procedures. Two examples are shown below; additional examples can be found in the references listed on this page.

Example 1: Power Implications of Generalized F-Test

Suppose you have a data set like the one shown below for comparing the mean effects of two treatments and a placebo. After a preliminary analysis, you believe that the differences in treatment means are statistically significant. Although these data were indeed generated from a normal population with unequal means and variances, application of the classical F-test will not support your intuition in this situation at all, because the p- value of the usual F-test is as large as .2323.

XPro computes the p-value for testing the equality of treatment means both under the (unreasonable) assumption of equal variances and without that assumption. The p-values of the two sets computed using XPro are as follows:

• P-value without the equal variances assumption (the generalized F-test of XPro): .0388
• P-value under the equal variances assumption (the classical F-test from any software package): .2323

The discrepancy of the p-values in this example clearly demonstrates the serious weakness of the classical F-test in the presence of heterogeneity. Because it ignores the problem of heterogeneity, the classical F-test failes to detect significant differences in treatments, even though there is sufficient data.

Are you willing to risk your imporatnt findings from an experiment when the lack of power of a test is low? In applications such as biomedical experiments, weakness of a procedure at this magnitude is unacceptable. In summary, this example demonstrates how exact methods can help conserve resources by concluding experiments in a timely manner at a reduced cost.

Example 2: Size performance, a comparison with widely used MLE based methods

This comparison and underlying exact methods have been presented at ASA courses and technical sessions. Although the study findings have implications for all higher-way balanced mixed models, we present here the results as applied to a simple a x n one-way layout (with n being the number of repetitions), so that you can easily reproduce the results and check our claims. In this simulation, the error variance is fixed at 1 and the variance component of the random factor is set to three values around 1. The following table shows Type I error of each test when the test is carried out at the .05 confidence level (using 10,000 simulated samples). This also represents the rate at which the corresponding 95% confidence intervals did not contain the true value of the variance component.

Type I (false positive) Error of Competing .05-Size Tests

It is evident from this simulation study that regardless of how large n is, the Type I error rates of the asymptotic test range between approximately .2 and .6, unless the number of levels of the random factor is unusually large. XPro computes exact and/or size guaranteed p-values and confidence intervals on variance components of random effects and mixed models. It also allows one to compare variance components from two balanced models.

Application Example 3: _______________________________________

The Table below shows a representative data set from a pharmaceutical study on drug for treating male erectile dysfunction. The objective of the study was to determine the efficacy and safety of active treatment at doses 0.5mg, 1.5mg, 5mg compared to placebo. The four groups are denoted as 1 (placebo),2,3,and 4 in the table. The data was collected over a period of 10 weeks from 20 subjects.

It can be seen that this data has highly unequal group variances indicating that the assumption of equal variances that one assumes for applying the classical F-test is not reasonable. However, if the assumption has no bearing on the conclusion of the test, that is not a problem. Let us see what happens. Shown below is the ANOVA table for above data after running XPro. The table includes the p-values by the F-test available from general software packages and the generalized F-test obtained using XPro.

As evident from the generalized p-value of 0.005 in the ANOVA table that the differences in treatments is highly significant. In fact the effect of second and third dosages are higher than that of placebo. Despite the substantial evidence available in this data set to reject the hypothesis of equal treatment effect, the F-test available from most software packages would completely fail to detect the significance of treatment effects. With a p-value of .2416 the F-test does not even suggest us to continue with the experiment to establish statistical significance of results. Such a large p-value would have completely discouraged the researchers. In real world applications this would be a multi-million dollar mistake. With XPro you will not take the chance of making such mistakes because it computes p-values with and without assuming equal variances.

References

[1]. Krutchkoff, R.G. (1988). "One-Way Fixed Effects Analysis of Variance When the Error Variances May be Unequal", Journal of Statistics, Computational Simulation, 30, 177-183.

[2]. Thursby, J.G. (1992). "A Comparison of Several Exact and Approximate Tests for Structured Shift Under Hetereoscedasticity", Journal of Econometrics, 53, 363-386.

[3]. Weerahandi, S. (1995). Exact Statistical Methods for Data Analysis, Springer-erlag, New York.

[4]. Weerahandi, S. (1995). "ANOVA Under Unequal Error Variances", Biometrics, 51, 589-599.

[5]. Zhou, L. and Mathew, T. (1994), "Some Tests for Variance Components Using Generalized p-Values", Technometrics, 4, 394-402.

[6]. Weerahandi, S. (1991), "Testing Variance Components in Mixed Models with Generalized p Values," Journal of the American Statistical Association, 86, 151-153.

[7]. Chi, E. and Weerahandi, S (1998), "Comparing Treatments Under Growth Curve Models with Compound-Symmetric Covariance Structure: Exact Tests Using Generalized p-Values,", To appear in JSPI.

©2002 Dataxiom Software, Inc.. All rights reserved. e-Mail comments and questions to support@dataxiom.com.