Choosing the right assessment strategy for your platform and organization can be a confusing process. Over the years, we’ve helped many of our EdTech clients navigate these waters. This article aims to provide an overview of the options and a framework for developing your assessment strategy.
Assessment solutions are generally broken into four different components:
Features required to create and manage items and related metadata, as well as assemble tests. Sophisticated systems sometimes include software that manages a workflow as content is edited, approved, and deployed. There may also be a need to generate tests dynamically from a “bank” of items.
If your team is small and your test questions are relatively simple (multiple-choice, true/false, multiple-select), you may not need an authoring component. Many companies get by with simply importing content from Microsoft Word or Excel. However, if your team is large, if you have thousands of items, or if your item types are more complex, an authoring test management solution will be required.
| Spectrum of authoring and test management features | |
| Simple | Complex |
| ● Multiple Choice, Multiple Select, True/False, Constructed Response
● Static tests |
● Many question types, including technology-enhanced items (TEIs)
● Dynamically created tests ● Import and export capabilities ● Question banks ● Content Workflows
|
To give a test, you need to assign the test to learners. Here again, there is a wide spectrum of options for setting up a test. Simple solutions assign tests to a group of students. More complex solutions allow the configuration of when the test is available, deadlines, number of retakes, randomization of questions, and password protection.
It’s important to think through the assignment process for your learners and choose a solution that will work for you now and evolve with your needs relatively easily and cheaply in the future.
| Spectrum of scheduling/assignment features | |
| Simple | Complex |
| ● Static tests
● Assign tests to a group of students with a due date. |
● Available date for test
● Password for test ● Due date for test ● Number of retakes allowed ● Randomization of questions
|
The user experience for a student to take an assessment is arguably the most important part. The following are all features that you can consider as part of your solution:
| Spectrum of student administration features | |
| Simple | Complex |
| ● Web desktop only
● All questions presented on one page |
● Responsive for multiple devices
● Accessible (WGAC compliant) ● Present questions one at a time ● Ability to track time on task ● Incorporate video and audio ● Ability to take notes ● Ability to track skipped questions ● Ability to access a calculator.
|
Regardless of how assessments are administered, the point is to review the student input. Simple analytics present a grade book view of the grades for all students on a single test as well as a detailed view of how each student answered each question on a test.
More complex solutions roll up grades for multiple groups/classes on a single test or show item analysis – the percentage of students who answered certain answers for each question on a test. Beyond these, your system may require custom reporting based on specific assessment contexts.
| Spectrum of analytics features | |
| Simple | Complex |
| ● Single class performance on single test
● Detailed view of how a single student answered each question of a single test |
● Aggregate statistics on multiple tests
● Longitudinal reports for a given student or all students on a single test ● Comparison of teacher/class statistics for a common test ● Item analysis for single test across many students
|
Enterprise assessment solutions generally come in four flavors: Learning Management Systems (LMS), Exam Management Software, Assessment APIs, or bespoke. The table below outlines each type of solution.
| LMS | Exam Software | Assessment APIs | Bespoke | |
| Use Case | Built for course management | Built for exam management | Built for creating custom assessment solutions | A wide variety of use cases. |
| Integration Capabilities | Low
Assessments features cannot be decoupled from the course structure or other features of the LMS. |
Medium
Exam software generally must be launched with single-sign-on capabilities from an existing learning systems. This can cause a disjointed user experience. |
High
These solutions are designed as components that can be embedded and within existing learning applications. |
High
By nature, bespoke solutions offer the highest level of integration. |
| Customization Time/Costs | Low
If a course structure will work for your company, customization costs are generally low. There are many third-party plugins that extend the functionality of LMS. |
Medium
While these solutions must be launched from existing applications, this generally means that integration time and costs are not high.
|
High
The nature of these systems requires sophisticated software developers and significant planning to integrate the various components into your learning applications. |
High
Even building a simple assessment solution can require significant software development and hosting costs. However, ongoing software subscription fees can be avoided. |
| Typical Customers | Universities and schools | Test prep companies, universities and schools | Large variety | Large variety |
| Student User Experience | Find enrolled course – view all materials – interact with material | View exams assigned to you – take exam | Can be customized | Can be customized |
| Instructor User Experience | Find assigned course – create materials – view student reports for course | Create exams. Configure exams. View student reports for exam. | Can be customized | Can be customized |
| Example Companies | Canvas, Blackboard, Moodle, Schoology | Tao Testing, Examsoft | Learnosity, H5P | N/A |
< $100,000 per year
Assessment tends to be expensive. While there are lower budget options for assessment solutions, they often require expensive software developers and hosting. If your budget is below $100,000 and does not allow for ongoing maintenance, API solutions are likely to be out of your range. Simple bespoke solutions or Open Source LMSs such as Canvas, EdX or Moodle are probably the best route for you.
Keep in mind that choosing an open-sourced platform means that you will need to budget for software developers who are knowledgeable in the selected platforms to build and maintain your solution.
100,000 or above
All solutions can work, and the optimal solution can best be answered by the questions below.
As mentioned in the above table, the integration capabilities vary widely in assessment systems. There is often a tradeoff between providing a seamless user experience and cost. If it’s critical for your organization that assessments integrate seamlessly with an existing learning system, an API or bespoke solution may best for you. If you have some flexibility in the user experience, you can likely opt for a LMS, Exam Software, or a simple bespoke solution.
| Spectrum of integration | ||
| Low | Medium | High |
| LMS, Exam Software | LMS, Exam Software, Bespoke | API Solution or Bespoke |
If your decision hasn’t been determined by the questions above, the context of your learner may be a determining factor. Most solutions have been designed with a specific learner context in mind. Below is a guideline.
| Learner context | Potential assessment solution |
| Tests and quizzes that will be a part of courseware or eTextbooks | ● Learning Management System
● If you already have a LMS, integrating an API solution may be best. |
| Stand-alone tests, such as test-prep or high-stakes exams | ● Exam Software
|
| Other Contexts | ● API or bespoke |
Across all of the assessment solutions we’ve worked with, the vast majority of items authored and used fall into four types:
These question types are easier and cheaper to author and develop. In many cases, these types of test items can be authored in Microsoft Word or Excel. If you are a company that is new to assessment with a smaller customer base, developing this kind of bespoke solution is often a great place to start.
However, if you are in a market in which technology-enhanced items (TEIs) are a competitive necessity or you have a large customer base, bespoke solutions can quickly become unruly.
While choosing the right assessment solution for your organization is a complex decision, it can often be determined by three critical questions:
