Measuring Course Quality:
Exploring Basic Issues
Student course evaluations involve assumptions about what makes good teaching and what performance standards in teaching can be expected.
Faculty members must be willing to be evaluated by students and must have a basic trust in the reliability of the procedure and its results. Without this cooperation, the goal of student course evaluation—optimizing instruction—remains out of reach. If instructors don’t have confidence in the accuracy of the results, they can hardly be expected to reflect on their teaching, or to consider making changes.
The choice of survey instruments—along with the organizational, trust-building measures to embed evaluations into a Quality Management process—plays a decisive role. Our clients using Class Climate have deployed a wide variety of questionnaires; there are effectively no limits to the surveys that can be developed and administered.
Legal Framework, Regulations, Data Security
Before beginning your first survey, decide who will carry what responsibility. You cannot underestimate the importance of trust in the procedure. Discuss the evaluation procedure’s goals in a broad-based forum:
- Which courses should be evaluated, and when?
- Who takes on the position of trust as evaluation coordinator/administrator?
- What access rights will deans and deans of studies have?
- What kinds of report should be generated?
- How should instructors and departments be involved in creating their own questions?
- What equipment is to be used?
- Who receives which reports?
- How will students receive feedback?
- What kind of security should students have?
- Should you use individual passwords for response tracking, or should you forego passwords and risk “stuffing the ballot box”?
- Should handwritten comments be transcribed from paper surveys to avoid having an instructor recognize a student’s handwriting?
- How high should the transcription threshold be set so that large—and inherently “anonymous” courses—are excluded from this expensive procedure?
Most universities develop and adopt specific policy regulations regarding these issues. You can then configure a system like Scantron’s Class Climate according to these specifications.
Developing and Managing Flexible Survey Instruments
Your evaluation system should free you to design single- or multi-page questionnaires containing numerous questions (items), which you can summarize individually or in the form of indicators. Scantron’s solution, Class Climate, supports the following types of questions:
- Scaled questions
- Open questions (free response format) with variable size response fields
- Single choice questions
- Multiple choice questions
- Matrix fields (for numerical data such as age, year or zip code)
- A variety of scaled response formats such as grade values from A to F a rating scale (e.g. 1–5 where 1 is Excellent and 5 is Needs Improvement)
A strong evaluation system should include a question library and questionnaire templates so you can store and re-use your work.You should be able to save questions and organize them to suit your needs, grouping them by department, subject matter, or whatever hierarchy helps you find questions faster and create questionnaires more easily.
The item library integrated in Class Climate enables you to organize related questions in up to 10 layers of categories.
In addition, you can create templates with pre-defined questions. Once you have a suite of templates (say, for a particular subject such as history, or type of class like a lab), you can easily customize the template each time you use it to include new or additional questions. Using templates to start the process dramatically speeds up your ability to create evaluations that are targeted and specific, giving you more accurate —and useful—responses.
It should be easy to create questionnaires in your evaluation system. Looks for a solution with a graphic layout, so you can see what your questionnaire looks like as you’re working on it and make changes faster. Class Climate provides two easy-to-use editing options:
- A simple, web-based editor you can use to design basic questionnaires.
- (As a system add-on) Visually create complex forms using VividForms Designer’s drag-and-drop design tool. Includes predefined templates for multiple columns, free-form layout for both portrait and landscape orientation, graphical items (such as lines and images), and other advanced design options.
Both editors support robust design functions and automatically adjust questionnaires for mobile, online, or paper distribution. Drag-and-drop questions from the question library or create questions on the fly using a simple question wizard for a clean, professional survey look. Once you’ve completed the wizard, you can always shift questions, exchange question groups, or correct typographical errors by double-clicking the question item. Simply and easily define font type and line spacing, among other design elements.
In addition to supporting a graphic layout process, you should be able to group questions by broader indicators. As well, your survey system should include a variety of additional information so you can analyze results and quickly identify appropriate changes to make to instruction. For example, Class Climate provides rules for creating norm reference values or quality standard objectives as well as details governing access rights to a questionnaire (domain, period, etc.).
See And the Survey Says…? Understanding Questionnaires in Class Climate for more details.
Defining the Organizational Structure
Each institution organizes departments, courses, and classes differently. An effective evaluation tool supports your organization structure, rather than forcing you to adapt and adjust a pre-define hierarchy. In addition, the most useful solutions enable you to develop and process questionnaires in batches to match your departmental needs.
The central evaluation approach in Class Climate provides a variety of functions for processing questionnaires in automated batches. For these functions to be effective and provide an overview of the different types of questionnaires to support the hundreds of courses you’re likely evaluating, you can define an organizational structure that matches your needs. You can:
- Set up subunits, such as departments in a college or colleges in a university
- Assign addresses, logos, and corporate design guidelines
- Specify the group of individuals who are responsible for implementing the evaluation
By providing a variety of user roles, Class Climate can accommodate the conditions of your particular organizational structure. Depending on the extent to which you follow a centralized evaluation model, you can authorize specific data access rights based on various roles.
Defining Evaluation Cycles and Interaction Times
Typically, your institution will deliver instruction using a semester- or quarter-based system. Some colleges and universities structure monthly classes to support returning education students. A strong solution supports your schedule so you can evaluate classes routinely without a lot of extra work.
Regardless of the term structure you use, you only need to set up the schedule once in Class Climate. If you have a blend of schedules (for example, your main campus follows a quarter system, while your Extension campus follows a monthly schedule or has classes of varying length), you can define individual periods for special programs or other unique circumstances.
While you prepare the organizational structure, also decide when you would like to evaluate during the cycle. Some institutions conduct course surveys towards the end of the term, so that instructors have the results in time to discuss them with their students. This creates a useful balance between including only students who have attended enough classes to be able to adequately assess a course and ensuring there is enough time remaining in the cycle for lecturers to reflect on the results and make changes if necessary.
You may also want to give instructors time at the beginning of the term to add their own questions to the survey (specifically relevant to their subjects or specific courses) so that these can be included when you produce the questionnaires.
Course Data and Instructor Addresses
Because of the high degree of automation provided, more and more universities are beginning to comprehensively evaluate all of their courses with a unified system like Class Climate. Metadata on the courses and their instructors (such as course names and numbers, faculty names and E-Mail addresses) provide electronic course catalogs.
Class Climate offers two procedures in order to enter the courses to be evaluated into the system: manual entry or import from CSV/Excel or XML data. Our Professional Services team can work with you to define an automated process. See the Product Training, Implementation, and Professional Services brochure for a complete list of the services we can provide.
How Can Scantron Help?
Scantron is a long-time leader in assessment solutions and services. We’ve served the education market from kindergarten to professional career development for more than 40 years. Our course evaluation solution, Class Climate, provides the best in course and instructor evaluations, and delivers broad-based survey functionality as well.
Class Climate provides world-class support for course evaluations. It’s built-in structure mimic department and course structure for most institutions—and can be customized to fit as necessary. Libraries make it easy to design and re-use questions year-over-year for longitudinal tracking. Core questionnaires that allow for instructor customization ensure that the needs of both administration and individual faculty or departments are met. Automated reminders and connectors to many common learning management systems make it easy to increase and track responses. Quality dashboards provide immediate, at-a-glance quality tracking and comparison.
Class Climate provides the greatest possible degree of automation for comprehensive course evaluation. The potential value of this solution to you can be seen in these critical areas:
- Digital document capture
“Scanning, not typing” means a drastic cost reduction in capturing data while minimizing errors. Within a few hours you can accurately analyze tens of thousands of survey questionnaires.
- Instant feedback for optimum self-reflection
Instant feedback means a new quality in evaluation for instructors. They can reflect on the feedback and make changes while they are still involved in the on-going course. Norm reference values and comparative profile lines motivate instructors to optimize their teaching.
- Batch processing
To systematically categorize hundreds or thousands of surveys, Class Climate imports existing data from course catalogs and faculty registers. The support provided by this metadata framework enables you to create reports, set up norm reference value data and make comparisons across subsets (programs of studies, departments, instructors, courses, etc.).
- Organizational overview and central process control
Class Climate provides a kind of “process control center” for the central evaluation coordinator. This perspective can give university executive officers, as well as department heads valuable insights. Optional QM views provide a high degree of openness. “Learning from the best” is supported, as is the identification of quality weaknesses.
Class Climate provides both decentral, instructor-oriented surveys using active survey accounts as well as central evaluations. Both procedures can be combined, yielding interesting possibilities such as giving university service organizations their own survey accounts (e.g. cafeteria, administration, library). And Class Climate can resolve the issue of ensuring relevance of centrally conducted course evaluations to a particular discipline. Faculty members are systematically involved and instructors with active accounts using the VividForms questionnaire designer can evaluate any aspect of the courses they are interested in—internet access is all it takes.
Class Climate provides specific advantages for everyone in the organization involved in evaluation:
- Students can expect that their judgments are analyzed and forwarded to their instructors quickly. Class Climate also takes security issues for students seriously. Class Climate promotes the dialog between teacher and student.
- Instructors spend a minimum amount of time and receive optimum feedback for self-reflection. Class Climate enables them to supplement the survey with their own individual questions. They receive reports that enable them to compare themselves with other reference groups.
- University executive officers establish an efficient service infrastructure and emphasize the service aspect of the central evaluation. University executives can also show that they take the needs, and anxieties, that often accompany an evaluation seriously.
- The Computer Center appreciates the web-based approach, which means that a client installation is not needed. As a result, the university-wide roll-out is smooth and trouble-free. Security and data protection aspects are high priorities for Class Climate. In addition, we offer hosting services for institutions who prefer to leave the IT work to us.
- Third parties: Student organizations, the administration, thesis students, research project members, PhD students, library, cafeteria—Class Climate is a web service that can be used anywhere in the organization where survey data needs to be gathered. Departments of social science especially appreciate this sophisticated mechanism for survey data capturing.
Class Climate users have been involved in its development from the very beginning. The requirements originated in practice and were decisive in developing the features that Class Climate now provides. Users of the system provide feedback on features, requests, and ideas for improvement are discussed intensively and have led to the development of standard software for course evaluation. This can also be seen in the detailed product documentation, training materials, and coherent support concept.
Scantron’s Professional Services team is here to help!
- Professional development enables you and your staff to navigate the software and interpret the results more effectively. Scantron Corporation has a fully staffed corporate development department through which we offer an array of product training and staff development classes and seminars for Class Climate. The team is dedicated to building capacity for change within organizations and institutions through a series of training and consulting activities.
- Get your first evaluation or survey project up and running faster with Scantron Implementation Services. We provide dedicated experts with extensive experience in course evaluation management to help you implement your system and begin processing your surveys quickly and efficiently for a more rapid return on investment. The Scantron implementation team works with you to create a manageable schedule customized to ensure your success in the evaluation process. Depending on your needs, consultations can be performed onsite or via the phone or web.
- Scantron offers a wide array of Professional Services to customize, enhance, and support your Class Climate capabilities. Save yourself time by letting Scantron do the work for you. We listen to your needs and then work with you to create the applications you require.
- Scantron’s world-class Print Services are available to print your custom forms and fulfill your large-volume needs with mailing and data processing services.
Whatever the evaluation or survey assistance you need, Scantron has the products, tools, services, and expertise to help you ensure that you have the right program for your students. Our web-based software, combined with our length and breadth of experience, help you get the most out of your evaluations and results. Contact us today and see how Scantron can meet you where you are and help you get to where you want to be.
Back to Article Series Home
© 2015 Scantron Corporation. All rights reserved.