A quantitative defense of linguistic methodology
Jon Sprouse, Diogo Almeida
July 2010

It is often argued that the current divide between linguistic theory and other domains of language research can be traced to the unreliability of linguistic methodology, and the resulting unreliability of linguistic theory. Linguists rely upon informal acceptability judgment experiments as the primary source of linguistic data. These experiments tend to employ small samples, often composed of non-naïve participants, and generally forego the use of distracter items and inferential statistics in the analysis of the results. We present resampling analyses of a large acceptability judgment dataset that provides new quantitative evidence that traditional linguistic methodology is extremely reliable with very small samples, usually at the level of the individual participant. We also review the evidence that critics of informal linguistic experiments provide to justify their concerns, which shows that the evidence provided by critics turns out to mostly corroborate the reliability of informal judgments, and not undermine it. We conclude that there is no empirical, logical, or statistical reason to think that the informal experiments routinely performed by linguists are unreliable. In fact, we show evidence that these experiments might be, in some circumstances, much more powerful than formal experiments with naïve participants. Given the lack of evidence of problems with traditional linguistic methodology, we hypothesize that one potential reason for the recurrence of this debate is that the phenomena critics are particularly interested in often elicit effects that are very small and hard to detect in formal acceptability judgment tasks. This suggests that critics may be mistaking a property of specific phenomena for a property of the methodology. Taken as a whole, these results suggest that broad criticisms of linguistic theory based on the reliability of linguistic data are unfounded, and that methodological concerns should not influence the relationship between linguistic theory and other domains of language research.
Format: [ pdf ]
Reference: lingbuzz/001075
(please use that when you cite this article)
keywords: acceptability judgments, linguistic theory, linguistic methodology, quantitative standards, experimental syntax, syntax
Downloaded:2422 times


[ edit this article | back to article list ]