What’s a CAT? A Primer on Computer-Adaptive Testing
What are the key components of computer adaptive testing (CAT)? What differentiates this method from conventional testing? A conventional, or fixed-form, test administers the same items across students and can be taken using either a paper-and-pencil form or on a computer. An adaptive test adjusts the difficulty based on student performance, administering different test questions of varying difficulty to test takers.
This primer highlights the key components of CAT and differentiates CAT from traditional testing methods. The final section focuses on additional considerations for CAT as it relates to educational testing.
Overview of CAT
One of the foundations of adaptive testing is that the testing experience be dynamic and sensitive to the performance of the test taker. The idea of adaptive or adjustable testing is not new and was first implemented by Binet back in 1912 with his intelligence test. Binet’s method made each successive question reflect past performance and allowed his IQ test to be both efficient and able to match the skill of the respondent.
The availability of computers made it possible to improve on Binet’s method by use of Item Response Theory models (IRT) that improved the sensitivity of the test to student performance. The key advance was to automate the scoring process and remove the need for a human proctor as was the case with Binet’s IQ test. The IRT models in combination with the personal computer made CAT a method that could be used in the classroom or on secured mobile devices.
Advantages of CAT over Conventional Methods
As more schools have switched to computer-based testing it is important to distinguish a computer-based fixed-form test from a CAT. In a computer-based fixed-form test, students will complete a fixed number of test questions. Each student may or may not see the same questions and the item order can be randomized to discourage cheating. These computer-based tests share some benefits with CAT including immediate scoring and paper resource savings.
Adaptive testing offers additional benefits compared to the computer-based fixed-form test. The following four elements of a CAT provide context to the discussion of the advantages of CAT over other testing methods.
- Item selection that adapts to performance.
With a CAT, it is possible to select new items using feedback from previous responses. This means that the CAT will become easier if the student is struggling and more difficult if the student is performing well. The goal is to select items that are appropriate for test takers thus reducing examinee frustration or boredom. A conventional test must balance measuring some students well or every student with less precision.
- Improved efficiency.
Adaptive testing makes it possible to administer questions near a students’ ability level and provides more information about student performance. This improves the precision of measurement and reduces the number of questions a student must answer. Students need less time in front of computer to complete the test and teachers have more time for instruction. The end result is a test that is both shorter and more accurate than a standard conventional test.
- Test security and item bank maintenance.
The matching of items to students results in different item content being seen by different students and reduces the risk of cheating in the classroom. The reduced test length for CAT also reduces item exposure and the risk of students being coached on frequently seen items.
- Results delivered immediately.
Another advantage of computer-based testing is the immediate calculation of ability level and feedback for both students and teachers. Additionally, when CAT is used for formative assessment, a student’s past performance can be used as the starting point for subsequent exams to measure growth and target instruction to areas of weakness.
Characteristics of Computer Adaptive Testing
Adaptive testing provides numerous benefits relative to conventional tests. Characteristics of a CAT that should be considered before implementation include the following:
- Students cannot skip questions or perform a posttest review.
One difference between conventional and adaptive testing is that adaptive testing does not allow students to skip and return to questions later. Conventional tests allow a student to return to a question later after completing other items while CAT requires the test questions to be answered as they appear. Unlike conventional testing methods, student performance on an adaptive test does not rely on the number-correct score for determining student performance. As described above, each question given to the student depends on the answer to the previous question and therefore the entire response pattern is taken into consideration when determining the student’s overall performance.
- Item bank size limits application of CAT.
The number of test questions required to cover all the skills taught in school over a range of difficulty renders CAT a method limited to companies with substantial resources. A large effort is required to maintain a CAT item bank and keep the item content fresh and up-to-date as well as secure. However, the cost of developing and maintaining a CAT item bank is worth the effort as it improves student measurement and satisfaction with the testing process.
How can Scantron help?
Scantron has an extensive track record providing computer-adaptive (Performance Series) and fixed-form formative (Achievement Series) assessment solutions, as well as world-class item banks, to help thousands of customers succeed in developing assessments that measure and accelerate student growth.
“Scantron’s computer-adaptive diagnostic test, Performance Series, provides a solution for using the results from a single assessment for multiple purposes,” says Marissa Larsen, Scantron’s Vice President of Portfolio Planning & Strategy: Assessment. “Being able to use a single assessment saves our clients time and effort, reducing test overload while improving the information you get about each student’s attainment of crucial skills and standards.”
We were one of the first partners to align our existing computer-adaptive test to the Common Core State Standards (CCSS)—and we were one of the first to provide detailed reports to align results to the most granular standards level (not just to the strand). But we didn’t stop there: We were the first in the market to develop brand-new content expressly to support CCSS. We’ve been evolving this content ever since, increasing our understanding of the instructional shifts in the CCSS, especially considering the shift to College and Career Readiness, and balanced cognitive rigor, and reflecting that understanding in our item banks and tests.
We’ve helped districts with workshops and consulting for content development processes as well as working with customers to provide high-quality, standards-based content. In addition to Scantron’s bench strength in Common Core assessment, the product infrastructure is flexible enough to support clients using state-specific standards as well as integrated blends of state and Common Core Standards. “This is a key requirement for supporting organizations who work with educators in different states with different requirements,” says Nikki Eatchel, Scantron Senior Vice President of Assessment.
We’ve demonstrated positive impact to student learning with research studies ranging from determining growth targets to predictive validity studies.
Beyond assessment vehicles, Scantron offers Scantron Analytics, powered by the leading “big data” analytics engine: QlikView. Scantron Analytics presents up-to-date information through highly visual, easy-to-understand dashboards. By storing all information in memory, Scantron Analytics delivers powerful analytics without the need for a separate data warehouse. Using information you’re already collecting, sourced from a wide variety of educational systems, Scantron Analytics displays easy-to-read, graphical dashboards and data visualizations. Important trends and previously hidden connections jump out, so you can spend your time developing creative solutions instead of trying to make sense of rows and columns of numbers.
Whatever the assessment assistance you need, Scantron has the products, tools, services, and expertise to help you ensure that you have the right program for your students. Our award-winning web-based software, combined with our comprehensive suite of assessment services, help you get the most out of your assessments and results. See how Scantron can meet you where you are and help you get to where you want to be.
Back to Performance Series
© 2016 Scantron Corporation. All rights reserved.