Acad Ronin (anonymous) meinte am 29. Nov, 02:24:
Minor qualification
I agree with you (pl.) that ideally data should pass the "Tukey intra-ocular trauma test", i.e., it should hit you between the eyes; I am suspicious when I see GMM FIML 3-stage least squares with robust errors (I think I made that up but God knows it could exist); I always worry that the researcher followed an optimal stopping rule that says, keep improving your methods until the results come out right, and then stop.That said, as I read the Levitt response you cite, he is saying that introduced the interaction effects in response to criticism, and that even after he did so, the main results survived. That is, he got his results despite the interaction variables and not because of them.
HedgeFundGuy antwortete am 29. Nov, 03:30:
Acad: I read it again, and Levitt seems to be saying that after Foot and Goetz make adjustments, his results only survive if the interaction variables are included.Andrew: I looked at that first reference, and found "page" 79 to be a rather uncompelling. If any of the main coefficients went to of from insignificance from Model 1 to Model 4, would you really believe it?
Kaiser (anonymous) antwortete am 29. Nov, 07:25:
Is he serious?
Like Andrew, I preface my comments by disclosing that I haven't read the book, the articles or the Foot and Goetz critique. I'm only commenting on Levitt's comment, which is the only thing I read.He seems to be saying that even after including 2-way interactions, the main effects are still significant, which he implies is a good thing ("the results survive"). Is he serious? Whenever interaction effects are significant (which would be why one would keep them in the model), main effects cannot be interpreted in isolation so I don't know what he means by "the results survive".
A totally different line of thinking, which I think he is tangentially referring to, is the fact that his sample size is small so it may be "demanding" of the model to estimate interaction effects (takes away some degrees of freedom). I have no idea how big his sample size is so I can't really comment on whether this is reasonable or not.
Andrew Gelman (anonymous) antwortete am 29. Nov, 12:52:
Interactions are important even if hard to estimate from a given dataset
Hedgefundguy,I agree that the estimates on page 79 of that presentation are uncompelling. The N of that study is just too small to accurately estimate all the interactions that we care about there. The point of that slide to show an exapmle where interactions are important, even though we don't have enough data to estimate them well. The model on the previous slide, fit without interactions, has statistical significance but is not scientifically plausible (I don't really believe that giving a low-value postpaid gift will reduce response rates at all, let alone by 6.9 percentage points).
The actual analysis we did for this problem was far from perfect. But we needed to look at interactions for the purpose of our goal of estimating the effect per dollar of survey incentive. The full analysis (in the Journal of Business and Economic Statistics) is here:
http://www.stat.columbia.edu/~gelman/research/published/jbes01m045r3.pdf
