Our expert consultants can work with you throughout the assessment development lifecycle to complete tasks such as:
- Design and Blueprint
- Exam Form and Bank Creation
- Item Development and Review
Scantron’s assessment development and psychometrics services are the backbone of the reliable assessment programs we develop and deliver.
Our decades of assessment and psychometrics expertise provide educators with sound examination programs. Scantron’s experts play an integral role in defining industry standards for test reliability, validity, and data integrity. Our experts are recognized for our leadership and continued contributions to education and the assessment industry. Our team is skilled in a wide array of psychometric models and statistics, but understand that each one of our customers has unique requirements. Our most successful clients operate with an openness and transparency with Scantron, which allows us to develop a more tailored assessment solution to meet all of their specific needs.
Assessment development and psychometric services Scantron provides include:
Beta and Live Test Creation – Using an approved blueprint and an item pool (either provided to Scantron or developed with Scantron), Scantron can build one or more forms that are given during the first test administration. If Scantron is provided an item pool that has existing statistics, exam forms can be built for a live administration. If the item pool is newly developed, exam forms can be created for a beta administration. The purpose of the beta is to collect exam results from a sample of test-takers representative of the target candidate population. Regardless of the type of administration (live or beta), Scantron assessment personnel creates the individual exam forms per category. Exam forms conform to the specifications outlined in the established blueprints.
Psychometric Analysis, Standard Setting, and Equating – Psychometric analysis of exam results occurs after the field test administration of the assessments. Scantron psychometricians analyze the field test data to determine the performance of individual items. Items performing outside acceptable statistical parameters are either removed from the pool or sent back through the review cycle to modify the item content as needed. Only items that have undergone both content and psychometric review are provided to the client for final approval in operational forms.
If so desired, Scantron can conduct a standard-setting process for purposes of determining a cut score/passing score for each exam. Once a cut score has been determined, equated operational exam forms are finalized. The equating process ensures that all exam forms in a given category are of equitable difficulty.
Data Integrity Analysis – Data integrity is critical to any testing program. Ensuring and evaluating data integrity is part of the overall assessment development, delivery, and reporting cycle and should be included in various checkpoints along the way. In general, there are three areas in which data forensics is focused:
- Item Content Exposure – Evaluating exposure of secure item content. Premature or unauthorized exposure of item content can increase error in student scores, as potential for memorization (versus true domain knowledge) increases when secure item content is compromised.
- Administration Anomalies – Reviewing data associated with assessment administration, to evaluate unexpected results that could be linked to a breach in administration protocol. Lack of standardized administration procedures can have significant impact on data integrity.
- Student Anomalies – Determining if outliers in student performance are indicative of potentially unexpected student activity that could impact accuracy of final scores. Various types of behavior are evaluated from both paper-based and computer-based administrations.
Bank Maintenance and Refresh – Item bank maintenance should be completed regularly for assessment programs. Scantron can provide assistance in routinely monitoring the health of the item banks and assessments from a number of dimensions (content coverage, statistical depth, etc.). Typical services requested by our clients include:
- Item Bank Inventory/Gap Analysis
- Statistical Health Check
- Assessment Development Forecasting
Item Bank Inventory/Gap Analysis
Scantron assessment professionals can review a client’s existing item bank and provide a corresponding inventory to the associated assessment blueprints. The inventory documents how many items (by content area, cognitive level, and item type) exist within the bank and identifies those areas that should be focused on for future assessment development.
Statistical Health Check
Scantron’s psychometricians can review exam- and item-level statistics to evaluate the statistical health of a program, providing specific recommendations for item review and ongoing assessment development.
Assessment Development Forecasting
Scantron’s consultants can work with client resources to review the following information:
- Program Goals
- Assessment Design
- Delivery Methodologies
- Delivery Channels
- Candidate Volume Trends
- Item Bank Inventories
- Statistical Health Check
- SME Availability