Abstract

Picture Archiving and Communication Systems (PACS) have become increasingly complex while healthcare providers increasingly depend on their reliability and scalability. When PACS were first created, many PACS companies built features required by customers and ensured features worked and scaled as expected.  Over time, every PACS vendor has exceeded even their own expectations as to how much data these systems receive and how much customers rely on them. Small vendors have the challenge of proving to themselves and customers that they can scale and large vendors want to be sure that an acquired PACS can scale as expected.  While features are fairly easy to verify, scalability is challenging to test for the following reasons:

  • Customer sites have many multi-vendor multi-modality workstations acquiring images and sending them to the PACS.
  • PACS servers integrate with a standardized tag format (RIS), which may or may not be part of the PACS vendor’s ecosystem.
  • PACS servers must service incoming imaging, consolidation requirements with the RIS, Query Retrieve and other pressures while handling client requests, etc.
  • It is nearly impossible to acquire real sample image data (DICOM) from multiple modalities and vendors that is easy to search based on any DICOM tag or descriptive information.
  • It’s risky to keep this confidential data in an office on a media that can be carried away by anyone.
  • It can be difficult to edit large studies so that identifiers match across all images and related HL7. This requires gigabytes of data to be downloaded into a local system, updating each header, then sending it all to the PACS in a realistic fashion with matching HL7 going to the RIS to test load and consolidate studies.

The ideal situation would imply that everyone would have a fully functional Hospital in which to test their software and simulate the above behaviors, but that’s not realistic. Each vendor would have their own Radiology Environment to test the server in with all the modalities and a RIS. The Virtual Radiology Environment is a simulation system that provides all of the network patterns and data flow that occur natively in the Radiology domain.

Virtual Radiology Environment Data System

Qualitest’s VRE Test Accelerator possesses a DICOM image repository that allows for the safe anonymous storage of real world data, accessed from multiple vendors and multiple modalities. The datasets are fully indexed in a relational database, allowing for fast searching of files, series or studies based on any DICOM tag from the header.

All identifying patient, site and provider information is encrypted and cannot be used to determine their source, and all datasets are stored in a completely confidential manner. Datasets are easily searched through a web interface and can be received via email or sent directly into a PACS via C-STORE and a related HL7, if required.

Usage:

  • Testing (e.g. automatic tests, manual tests, workflow tests and simulation)
  • Development (e.g. prototyping, reverse engineering, bug fixing)
  • Support (e.g. assistance in problem analysis)
  • Training (e.g. application training, field service training on network traffic)
  • Demonstration (e.g. congresses, customers visits, seminars, meetings)
  • Connectathons

Rational:

  • Different kinds of media lying around, damaged or even lost
  • Difficulty storing and recovering old (non-DICOM) data
  • Disability to recover the need for storage of certain (old) data
  • Time consuming data searching, identification and retrieval
  • Difficult data sharing and (cross border) carriage problems
  • Invasive incorporation into business processes

Virtual Radiology Environment Simulation System

VRE simulates all data flows into and out of a PACS. It allows for the simple, single execution of point to point tests, as well as full IHE profiles, such as Scheduled Workflow. As well as performing each of these tests to verify results, each profile can be run under load for multiple hours or day with multiple interactions against a given server or set of servers. Instead of these tests just being run as a series of random load tests, organized logging occurs for each action taken and all errors are tracked. After test execution, detailed reports can be run to determine timing for each action and all errors as they occurred.

Tests can be run from a Nunit-driven set of scripts or a web interface. In either case, an email with information from the test is sent to the user, along with high-level details concerning the results. Information regarding jobs can be tracked using a Job Key Guide saved in the database and used across all data, tracking the test as it executes.

DICOM Operations:

C-ECHO

DMWL

C-STORE

Storage Commit

MPPS

C-FIND

C-MOVE

SCP (Permanent SCP running on VRE that accepts all DICOM Associations and logs activity)

 

HL7 Operations:

ADT:  Patient, merge, update and registration

ORM: Order and update

ORU:  Report(s)

Workflows:

Scheduled Workflow (with options)

ADT → ORM(s) → MWL → C-STORE → ORU(s)

Ability to edit all messages separately or as one.  Also the ability to

add or remove steps as options.

Case Studies

The VRE System has saved countless hours when it has been used to test a PACS product. Not only is it simpler to locate DICOM studies to use as sample data, it also greatly speeds up the process of editing each DICOM file and HL7 stream that is sent into the PACS(s).

The coordination and running of software across each file on disk and updating the header takes days to correctly update one Terabyte of DICOM images for multiple studies and patients, along with correctly edited HL7 for each. The coordination includes management of the Study Instance Unique Identification Numbers (UID), Series Instance UIDs, Image Instance UIDs, Accession Numbers, Patient Identifier(s), Provider Identifier(s) and Site Identifier(s), as well as any custom codes in the HL7, etc. Multiple staff must agree on what UIDs should look like and what the UID root should be, etc. The VRE system accomplishes these tasks automatically by using a proprietary rules engine internally to update all of the header data without loading the full image data into memory.  This saves system resources and time on the VRE server while realistically simulating a live Radiology Department.

Case Study 1: Fixing a defect

To fix a customer-facing problem in managing memory for a client viewer, test data needed to be loaded into a Vendor’s PACS using actual DICOM images of a given modality, size and slice count.

The datasets had to come in the PACS using C-STORE and consolidated by the system with required orders.  After the data was converted into the proprietary format, the issue was fully reproducible.  The real-world execution of the VRE System allowed this vendor to reproduce the issue in-house without delays and provide a patch to the impacted customer faster than previously possible.

VRE saved the Vendor 10 days of work for five engineers and greatly simplified paperwork and coordination for executing such a task.

Case Study 2: Performance-related 510K submission

The FDA requires minimum performance for PACS servers. A vendor was able to run the VRE System at a rate consistent with the performance level required. Because the engineers at the vendor’s site had familiarity with VRE, they were able to accomplish a quick turn-around in proving the system’s performance capacity. It was easy to prove and document the rate at which the data could be consolidated and viewed by the target PACS system. The exporting of data back out of the PACS was accomplished using C-MOVE to a third party system.  The work took far less time than with using typical testing systems.

In this scenario, 120 man hours were saved.  The effort of six full time QA Engineers was reduced to two, procuring an ROI of 66%.

Case Study 3: Evaluating a partner vendor

A vendor was required to validate the performance of a PACS server and RIS before committing to a partnership or purchase. Similarly to Case Study 1, VRE greatly simplified the work and increased accuracy.  Historically, the typical approach was to “shotgun” the server with random amounts of duplicate DICOM images with poorly coordinated inconsistent updates to the headers.  Executing a realistic load test with properly updated headers across multiple terabytes of data would take weeks to coordinate and send into the server.

VRE saved 80 man hours of labor for a single large-scale load test. Furthermore, during the course of the year, multiple tests of the same scope are executed, and each time will save the same amount of hours. The effort of four full time Engineers was reduced to one per test execution.

Return on Investment

The Case Studies show that enormous amounts of time can be saved by using the VRE System, ensuring a substantial Return On Investment.  Over the course of the first year, VRE saves customers more than their initial investment.  By not requiring engineers in large numbers for each task, combined with the accuracy and reproducibility of the testing, the cost savings make the investment an obvious choice.

The ROI gained by VRE is represented in the previous case studies as follows:

  • The client in Case Study 1 achieved 30 days of work in 5 days, resulting in an ROI of 83%.
  • In Case Study 2, 120 man hours were saved. The effort of six full time QA Engineers was reduced to two, procuring an ROI of 66%.
  • In Case Study 3, the effort of four full time Engineers was reduced to one per test execution, procuring an approximate ROI of 75%.

For many customers, the savings are over 1.5 million dollars per year in labor savings, due to the simplistic, understandable and replicable testing methodologies. Because VRE is PACS- and RIS- agnostic, large vendors can use it for many years to guarantee product quality and reliability for their customers.  Smaller vendors can use VRE to automate tests that require staff they may not have, as well as to test their system for larger deployments as they expand over the years.

 

Legally, VRE provides a guarantee for patient confidentiality, eliminating any risk to the vendors who use it for their testing.  Because vendors can run tests in a safe closed environment, there is no risk of broken promises to customers. This prevents lawsuits and lost opportunities due to a bad performance at a live customer site.  The potential savings to the vendor’s reputation have far-reaching benefits that are impossible to monetarily measure, but can affect all areas of the corporation’s profits.

When one considers the savings across a given vendor site, combined with the savings magnified across multiple development and QA sites internationally, the gains are enormous.

Conclusion

The VRE System as a whole greatly simplifies all testing related to Radiology systems.  It helps protect vendors from confidentiality issues while greatly enhancing the realism of the testing.  It radically simplifies all testing tasks that are time and cost prohibitive.  These tasks include steps such as acquiring large amounts of multi-vendor/multi-modality images and updating each file across multiple studies with potentially hundreds of slices.  Along with this it guarantees that all HL7 data is transferred in the correct order to the RIS and the images and all DICOM transactions occur in the expected format and sequence.  Each interaction is tracked and logged while tests run.

For a QA department this system allows staff with a novice skill in DICOM and technology generally to run complex tests in a simple automated fashion.  After these tests complete it is very easy for Engineers in a server or client team for the PACS or RIS to re-execute those test against their own development systems and debug code while data is flowing.

For a small vendor this system offers the ability to simulate running their systems in a large enterprise clinical environment.  For large vendors they can do the same for their systems but also for any system they consider acquiring.  This reduces costs and increases confidence in the commitments made to customers and other vendors.

Finally, every vendor in the Radiology domain has come across issues that occur at a site that are nearly impossible to reproduce in house.  This can be frustrating and expensive to investigate.  The VRE system allows for searching for datasets that match the issue and running simulations through the VRE System that emulate all possible causes.

Glossary:

PACS                     Picture Archive and Communication System

DICOM                 Digital Image Communication for Medicine

HL7                         Health Level 7

VRE Simulator    Virtual Radiology Environment Simulator

VRE Data              Virtual Radiology Environment Data System