Application Software Testing
By Peter M. Paulli
Manager, Consulting Services
Beacon Partners, Inc.

A basic woodworking rule to avoid wasting time, money and material is to measure twice and cut once. The same analogy can be applied to implementing new application software; test and test again to insure the software being installed meets your needs. Often, not enough time and effort is taken to verify that the software being installed will function in the planned environment. The failure to thoroughly test software prior to implementation is one of the biggest contributors to system implementation failures.

Failed software testing can be attributed to many things. This article will address some of the more prominent factors. These factors include:

• Unclear scope for testing
• Lack of understanding software functionality
• Missing or inadequate test plan
• Unrealistic test cases
• Insufficient time for testing

Unclear Scope of Testing

I look at software testing in three distinct phases: functional testing, integrated testing, and acceptance testing. The scope of each of these phases should build on the prior phase.

Functional Testing

• Functional testing should determine whether the standard design of the system would work in the planned environment. To do this, functional testing should, at a minimal, encompass the basic processing of the system. This should include the system's access to and use of reference files, application dictionaries and program tables. Functional testing should not be confused with acceptance testing; functional testing should be used to verify that the software vendor's baseline product works prior to the development of additional processing logic specific to the client needs. This is the time to identify and correct any anomalies with the standard product.
• Functional testing should also provide the opportunity to verify that the screen flow and cursor flow within the screens will work in the client's environment. By testing and identifying these issues up front, the software vendor should have sufficient time to implement any modifications that may be required.

Integrated Testing

• I have heard this term used in many different ways; however, when I look at developing an integrated test plan, I am interested in testing the flow of data to and from the new system. This flow of data can be between systems within the health care organization or between different modules of the new application.
• The integration testing of data inbound to the new system is extremely important as the incoming transactions may update critical data, such as patient demographics, clinical results or ancillary data. Integrated test data can also be used by systems receiving data from your system to insure that they can process and update their database accurately.

Acceptance Testing

• Client acceptance testing should encompass all aspects of how the system will be used when live. This includes all customization that has been done specifically for the organization. Acceptance testing should also include all production processes that will be run. This includes all daily, weekly and monthly jobs.
• This is the last and most comprehensive module of the test plan. Client acceptance testing should be used as the last point prior to implementation, where the client can reject the software as it is and identify the deficiencies in the system.

Lack of Understanding Software Functionality

Before developing the test plan, project team members should use all opportunities to play with the new system. Without this learning phase the test scenarios that are developed may not be comprehensive enough to verify that the software meets the client's needs. Some software vendors have pre-defined lessons to use to learn baseline functionality of the application. The experience gained through these exercises will (1) facilitate the understanding of how the software works, (2) determine if custom modifications are required, and (3) provide a knowledge base for the development of the test cases.

It is important that the end-users be encouraged to participate in this learning phase, as this will help in the transition from the old system to the new system. End-user participation may uncover obscure items from the old system that had not been previously identified as being needed in the new system. The identification of these types of modifications early on in the project will give your software vendor more time to analyze and develop the custom processes required for a successful implementation.

Missing or Inadequate Test Plan

Planning

• A detailed plan should be developed and used to manage the overall testing process. The plan should identify the scenario that is to be executed, and other items that may need to be run in order for the scenario to take effect (e.g., run nightly processing to post receivable, produce claim, update report file).
• Without a detailed test plan, the execution of the test scenarios may be disorganized or incomplete. Like a project plan, the test plan can define timelines and dependencies and identify resources who will be responsible for executing the test cases.

Results Reporting

• The most effective process for reporting errors is to use the form that has the test scenario details. By using this form, the person charged with reviewing (and resolving) the issue will have the pertinent data (e.g., patient name, account number) available to them. Equally important is that the person reporting the error should explain in detail what they were doing at the time of the error, and include screen prints, reports, etc., to further define the problem. The test form can be used as the source document for re-testing the case once the error has been corrected.
• During any testing there will be problems or questions that arise that will need to be reviewed by either the software vendor and/or the project team. Having a simple list of incidents is an effective way to manage issues through resolution. These tracking logs can be as simple or elaborate as needed.

Meetings

• Meetings of the project test team are beneficial as far as sharing information. However, during the testing period meetings should be kept to a minimum, as time spent in meetings is time not testing.
• The test team should use an electronic medium to report the results from testing, as this is one of the fastest ways of sharing information. By not using real-time distribution methods, critical information may not be reviewed, and decisions may not be made in a timely manner.

Unrealistic Test Cases 

When developing scenarios for testing the software, it is extremely important that they reflect "real-life" processing that will occur when the system is live. Overly simplistic cases will do nothing more than prove that the baseline product will function. They will not, however, verify that the organization's processing needs are being met.

The best approach for developing realistic test cases is to start with an examination of the existing processes and develop scenarios that will test each software application module. Each scenario should be written to expand as the test evolves. For example, if the first scenario is to register the patient with self-pay insurance, the next scenario could be to modify the patient's registration data (e.g., add an employer) and add new third-party health insurance. Concurrent with this test scenario is to enter charges for treatment and produce an insurance form. Once we have generated the insurance form and created the account receivable, we can then produce reports (e.g., aged trial balance) and post a third-party payment and adjustment. The last phase of the scenario is to change the balance of the receivable to a self-pay status. Then the statement and dunning processes can be tested. As we can see, there is a logical progression to the development of the scenario.

Insufficient Time for Testing

Acceptance testing is often performed at the end of the project life cycle with the thought that testing is the culmination of the project team's development efforts. While this is a true statement, executing functional or integrated testing at the end of the project may not leave the client or the vendor enough time to change programs and/or processes that require modifications.

The overall project plan should include realistic time frames for testing. Within the testing schedule there should be contingency time for re-testing scenarios that had errors and that need to function properly prior the continuation to the next test phase.

Conclusion

When installing a software application, the testing phase is the most critical time to determine whether the product is right for your organization. The testing phase will provide the opportunity to identify errors with the software prior to its implementation and allow all parties a chance to resolve the errors in time for a successful go-live.

Here are some simple things to remember when implementing a new software application, and you have been put in charge of verifying that the system will meet the organization's needs.

• When will the software be available for testing?

• What are the areas/modules that must be tested?

• Why are the test scenarios important for this test plan?

• Who will be the dedicated resources available for testing?

• Where will the testing team execute their test cases?

• How will we follow up on open issues with the vendor?

Once the testing has been completed go back through the test plan and the test results, and answer these questions:

1. Did we accomplish our objectives and achieve our expected results?

2. Did we test all areas and scenarios that will be used when we go live?

3. Did we re-test all errors that needed to be re-tested?

4. Does the system as designed and installed meet our needs?

If you answered "Yes" to all of the above questions, then you are on your way to a successful implementation.

If you answered "No" to any of the questions, it's time to "re-measure" and test again.

If you would like more information regarding Beacon Partners' Software Application Testing services, please call 1-800-4BEACON or e-mail services@beaconpartners.com.

# # #

Return to the Articles Index >>