Insights Blog Qualitest Develops Cutting-Edge AI Assessment Solution for EdTech Startup

Case Study

Qualitest Develops Cutting-Edge AI Assessment Solution for EdTech Startup

Ai Testbanks needed a partner that was well-versed in EdTech, assessment solutions, and AI to be able to apply artificial intelligence and machine learning to create a large volume of quality test items.

Qualitest Develops Cutting-Edge AI Assessment Solution for EdTech Startup
Overview
Challenges

To automatically generate test questions based on existing material using AI and ML.

Ai Testbanks needed a partner that was well-versed in EdTech, assessment solutions, and AI to accomplish this.

Solutions

Qualitest implemented an iterative process model, with three clear, well-defined phases.

The three phases were: Discovery, research & development, development of MVP, iterative improvement.

Results

Dramatically increased the value of the company and helped to bring their first customers onboard.

Among benefits gained: 25% increase in test item quality and 118% increase in profitability.

Client overview

Ai Testbanks is a software/service company founded by seasoned EdTech executives.  They have built disruptive technology to address the growing demand for assessments in the $11bn education market.

Ai Testbanks uses artificial intelligence and skilled subject matter experts to generate exponentially more and more varied test items directly from the content found in textbooks or online courses.

The question: who has EdTech and AI expertise?

With the wide adoption of online learning, the use of assessment has dramatically increased across all educational segments – K12, Higher Education, and adult learning. This trend has been accelerated further by the virtual schooling required by the worldwide pandemic.  At the same time, students have become increasingly savvy at sharing tests online. Once shared, the tests lose their value.

The vast majority of educational publishers have not changed their methods for creating assessments: disparate networks of subject matter experts and long and expensive editorial processes.  The current technology and processes simply cannot meet the growing demand.

The founders of Ai Testbanks believed that artificial intelligence and machine learning could be applied to create a large volume of quality test items. Specifically, their idea was to train AI to “read” textbooks, online courses, and video transcripts and automatically generate test questions based on the material. They needed a partner that was well-versed in EdTech, assessment solutions, and AI to accomplish this.

Providing the answers to the test of advanced technology

Qualitest’s WeBuildLearning division implemented an iterative process model, with three clear, well-defined phases:

  • Phase 1 – Discovery, research & development
  • Phase 2 – Development of MVP
  • Phase 3 – Iterative improvement

Phase 1 – Discovery, research & development

In this phase, Qualitest EdTech experts researched existing natural language, libraries and other methods available for parsing large amounts of hierarchical text. They also worked directly with several major educational publishers who provided examples of textbooks, online courses, video transcripts and quality test items across a wide range of disciplines.  Using this data as models, the team experimented on several proof of concepts (POCs) for:

  • Parsing long-form, hierarchical text.
  • Identifying content that can be repurposed as a test question.
  • Identifying potential incorrect answers (distractors) to multiple choice questions.
  • Providing useful metadata along with each test question.

Each proof of concept (POC) was presented to potential clients using their own content.  The feedback of these clients was then incorporated back into iterative releases.

One key element that was determined during this process was that given the nascence of the technology, a hybrid AI-human model would be best for Ai Testbanks. In this model, technology would generate a large number of potential questions. Humans would then use efficient web tools to efficiently review and approve the best items. The approved items would then be analyzed using machine learning to identify patterns.

This is the resulting process:

  1. We start with a library of fully edited textbooks, online courses, and video transcripts.
  2. Algorithms parse all content to create as many new test items as possible.
  3. Subject Matter Experts use custom web tools to efficiently review, refine and save the best items.
  4. Thousands of quality test items are created, significantly reducing time, cost and overhead.
  5. Machine learning compares inputs/outputs to continually refine quality and lower overhead.

The end result of this phase was a working proof-of-concept as well as a recommendation and estimate for a minimum viable product (MVP).  Qualitest EdTech experts worked with Ai Testbank’s founders and their potential clients to refine and finalize this MVP plan.

Phase 2 – Development of MVP

The MVP was to be a hybrid AI-human solution that consisted of:

  • A module for ingesting and parsing ePub (the major ebook format) content.
  • A module for identifying key concepts and answer options (distractors).
  • A web interface for subject matter experts (SMEs) to browse, search and review AI generated test questions. This web interface would enable the SMEs to lightly edit the best questions.
  • A web interface for clients to review and approve the finalized questions.
  • A machine learning process for comparing the original AI-generated test items with the finalized items to identify patterns.

This MVP was presented to potential clients – large educational publishers – resulting with two of these clients signing on for multi-product deals. The first delivery for Ai Testbanks was to cover 13 different textbooks across disciplines including Accounting, Biology, Business, Health, and Psychology.  The technology and test questions were to be delivered in 6 months.

The Qualitest technology team worked directly with the founders, subject matter experts, and Ai Testbanks clients to develop the MVP on time and budget.  Over 19,000 test items (an average of 100 per chapter) were delivered to the initial clients.  Some of these test items were tested in the market, where they outranked traditionally authored test items in the areas of variety, range of difficulty, and quantiy.

Phase 3 – Iterative improvement

Since the MVP release, Qualitest’s WeBuildLEarning EdTech experts have collaborated with Ai Testbanks, its SMEs, and customers to rapidly and exponentially improve the technology.

We developed the concept of “derivative questions” – test items that are derived from existing items, but test the student in fundamentally different ways. These items are fully automated, improving the speed, quantity, variety and profitability of items.

Key benefits

Given their expertise in assessment and artificial intelligence, the EdTech experts at Qualitest’s WeBuildLearning division were able to hit the ground running for Ai Testbanks.  We delivered cutting-edge technology on-time and budget that dramatically increased the value of the company and helped to bring their first customers onboard.  Since then, some of the world’s largest publishers signed on with Ai Testbanks.

The Client had their technology improved as follows:

  • 25% increase in test item quality.
  • 30% increase in the average number of questions generated.
  • 35% increase in subject matter expert productivity.
  • 100% increase in test item variety.
  • 118% increase in profitability.

 

quality engineering free assessment

 

download the pdf