Although the sheer number of key, review, and
individual studies populating the Flanagan et al. (2006) research
syntheses looks impressive, the syntheses suffer from four primary
limitations.
First, it would impossible to replicate the CHC
COG-ACH summaries of Flanagan et al. (2006) due to the lack of: (a)
specified and published database search terms and date ranges, (b)
significant CHC- ACH relation significance criteria, (c) study
inclusion and exclusion criteria, and (d) procedures to control for
possible publication bias (see Cooper, 1998; also see Pillemer
& Light, 1980). Second, the use of the broad
dependent
variable (DV) domains of reading, math, and
writing make it impossible to determine if the salient COG-ACH
relations generalize across sub-achievement domains (e.g., basic
reading skills vs. reading comprehension). Third, the select and
limited developmental CHC COG-ACH trends discussed by Flanagan et
al. (2006) are based only on a handful of key CHC studies.
The age characteristics of the bulk of the individual
studies included in the Flanagan et al. (2006) research
summaries are ignored (i.e., no attempt was made to present CHC
COG-ACH results by age- differentiated sub-samples).Finally,
specification error, sometimes
called
omitted-variable
bias (Keith, 2006), occurs when potentially
important variables in predictive or explanatory research are not
included in a study’s research design. Specification
error can lead to biased estimates of the effects (i.e., relative
importance) of individual predictor variables on the dependent
variable of interest (Pedhazur, 1997). Almost all the
reviewand individual studies listed by Flanagan et
al. (2006) likely suffer from an unknown degree of specification
error since the intent of their synthesis was to identify the
relative importance of the various CHC domains within the CHC
framework. One example is presented below.Floyd et al. (2003) used an analogy from sports to explain
specification error.
"As an analogy,
a college basketball coach could construct a regression model that
included season-long player statistics of starters in order to
predict post-season performance. If the coach omitted the game
statistics for the point guards and included only the statistics
for the remaining four starters, over- or under- estimates of the
importance of one or more of the other four starting positions
would probably result. These biased findings might lead the coach
to make erroneous decisions about the strengths of certain player
positions when developing game strategies. Because of the failure
to include measures of potentially important." (p.
156).
Flanagan et al. (2006) list the
individual
study of
Wagner et al. (1997) as
evidence for the importance of Gc, Ga, and Glr in
reading. Specification error is present in this study due to
the lack of indicators of the more complete array of important
predictor CHC abilities for reading. The CHC cognitive abilities of
Ga, Gc, and Glr may have displayed slightly different degrees of
relative importance (and might have demonstrated non-significance)
had Wagner et al. (1997) included measures of Gsm, Gf, Gs, and Gv
in their predictor models. In the Flanagan et al. (2006) summary tables,
Aaron (1995) is classified as a
review study. Inspection of the original article finds it
not to be a systematic review in the traditional sense of
aliterature
review (Cooper, 1998). Aaron's article
describes a diagnostic procedure for reading disabilities
and draws upon select literature sources. This is not a
criticism of the original purpose of the Aaron (1995) publication,
nor is this a criticism of the categorization of the Aaron study as
a review in the published summary tables. The point is
that the term review typically has the connotation in the
scientific literature of systematic narrative or empirical research
synthesis of extant research studies.
1. Introduction
