Understanding the impact of person and task characteristics on assessment: a cognitive framework for learning and assessment

CASCALLAR, E. C.; MUSSO, M. F.

Malta

Conferencia; AEA-Europe 10th Annual Conference; 2009

Association for Educational Assessment- Europe

Current advances in assessment, and in particular the emphasis in incorporating the concept of assessment for learning, has made it imperative to better understand the interaction between person and task characteristics in assessments, both traditional and incorporated in the learning process. This research examines the cognitive processes involved in the interaction between a test-taker and items/tasks characteristics. It has been shown that individuals differ fundamentally in the components of a four-factor model of the cognitive ability space (i.e., Kyllonen & Woltz, 1989; Anderson, 2007).  The four factors include: (1) processing speed, (2) working memory capacity, (3) declarative knowledge, and (4) procedural skills.  This model has been used to predict success in a variety of learning tasks. The conceptualization of assessment with this framework will facilitate the evaluation of these separate components of cognitive functioning to predict item and test performance and learning outcomes (Anderson, 2006, 2007). Data from two worldwide computer-delivered, multiple choice examinations were used. IRT analyses were used to score the examinations, and to determine the level of ability of the students belonging to each of the groups considered. Students were uniquely classified into groups representing different levels of performance attained in the clusters of items that required knowledge retrieval (declarative knowledge), and knowledge application (procedural skills).  Each group was further divided into two groups by a median split into overall low and high speed of processing and ability level according to their overall grade-point average.  The dependent variable used was differences in mean reaction time (in milliseconds) from a student’s baseline for the same set of items.  A total of 3,300 student results were collected for the first exam and 1,500 for the second. A complete set of descriptive and inferential results were obtained.  The split-plot factorial design was analyzed with ANOVA procedures.  In addition, a generalizability theory study was performed.  Results indicate significant effects for the interaction of overall ability with cognitive processing level required at the item level.  In addition, a complex pattern of results illustrates the interrelationships between level of processing, and overall ability in speed of processing patterns.  The discussion of results and the conclusions reached have relevance for the construction of item types that could tap to varying degrees different levels of processing and types of information.  This in turn could have profound impact on test takers with various strengths and weaknesses in their cognitive ability profile.  The presentation highlights results which are highly relevant for the field of assessment and for learning theory, and could facilitate the design of innovative assessments which take into account these important factors.