Assessing the reliability of journal data in syntax: Linguistic Inquiry 2001-2010
Jon Sprouse, Carson Schutze, Diogo Almeida
September 2011
 

There has been a consistent pattern of criticism of the reliability of acceptability judgment data in syntax for at least 50 years (e.g., Hill 1961), culminating in several high-profile criticisms within the past ten years (e.g., Edelman and Christiansen 2003, Ferreira 2005, Wasow and Arnold 2005, Gibson and Fedorenko 2010a, 2010b). The fundamental claim of these critics is that traditional acceptability judgment collection methods, which tend to be relatively informal compared to methods from experimental psychology, lead to an intolerably high number of false positive results. In this paper we empirically assess this claim by formally testing a random sample of 292 sentence types that form 146 two-condition phenomena taken from the most recent ten years of articles in a leading journal of theoretical linguistics (Linguistic Inquiry 2001-2010). We report the results of two experiments designed to assess the replication rate of these 146 phenomena under formal experimental methods (Experiment 1 used the magnitude estimation task and 168 participants, Experiment 2 used the forced-choice task and 96 participants). 139 of the 146 phenomena, or 95%, replicated in the formal experiments (with a margin of error of ±5%). This means that even under the (likely unwarranted) assumption that all of the discrepant results are false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the maximum proportion of such false positives in LI 2001-2010 is 5% (±5%). We discuss the implications of these results for questions about the reliability of syntactic data, as well as the practical consequences of these results for the methodological options available to syntacticians.
Format: [ pdf ]
Reference: lingbuzz/001352
(please use that when you cite this article, unless you want to cite the full url: http://ling.auf.net/lingbuzz/001352)
keywords: acceptability judgments, syntactic theory, linguistic methodology, quantitative standards, experimental syntax, syntax
Downloaded:539 times

 

[ edit this article | back to article list ]