It’s all about the Data

IT projects need quality verfication with test data

Today, nearly every IT project requires quality verification. But organisations cannot effectively evaluate IT solutions because they do not have the test data available to do so. Therefore creating such data manually can be quite tedious and is a time-consuming process. 

What is Test Data? 

Testing means running tests or exercising the code to run such tests so test data will be a requirement. This will consist of records which contains information that can be used to evaluate the system. Testers can create test data manually, by using an UI or via an automation tool. To summarise test data is a collection of data records that are used to evaluate or exercise the database, web server and system logic. 

But What if there was a Third Option?

Production data can be a reliable source of data in testing systems, but it may cause some issues such as: 

• Legal compliance with GDPR guidelines (as production data cannot be abused). 
• Authorised individual access to sensitive financial or other data.
• Data integrity within a multi-system (multi-database) environment. 

This post describes how to use production data to run tests without causing any of the above-mentioned problems. It shows how to use production data to enable automatic test data preparation with anonymisation for automated testing in the CI/CD pipeline. It also describes manual testing within an ecosystem called TDM (Test Data Management).

What is the solution?

So, how can the process be delivered? This can be achieved by using Soflab’s proposition namely Soflab GALL (Global Anonymisation Linked Loader). GALL has been created by our in-house team of experts, who have conducted many projects relating to comprehensive testing of environments in addition to deployments in the Business Intelligence domain (BI). 

The solution enables programming and testing teams to use databases containing credible information, that is secured by anonymisation, while maintaining its consistency. It also allows for the pseudonymisation and exercising the right to be forgotten in non-production environments. 

The main question is whether to prepare test data manually or automatically. The next question being is it profitable to implement tools to prepare test data based on production data? Manual test data preparation can be performed in non-repeatable processes. Where manual test data creation is not too complex it is easier and quicker to execute.

When creation of low-complexity test data is a cyclic process, it can be more profitable to implement automated test data preparation. Test data becomes more complicated in areas such as banking and telecommunication systems, where testers encounter specific test cases. When creating such test data from scratch it can be difficult to achieve within the planned test execution period.

In such situations, production data can then be modified by scripts that will anonymise any sensitive data. These processes can be somewhat time-consuming without having the appropriate tools to hand. Soflab’s solution allows the preparation of appropriate anonymisation rules and copies of the anonymised production data.

A typical example

An insurance company must evaluate the process of renewing an individual policy for a group of customers with a specific additional product. Based on test requirements, hundreds or thousands of similar anonymised records can be generated for manual and automated testing. This is based on production data meeting such criteria even if there is only one record that meet this criterion.

Due to anonymisation rules the generated data look like real data as first names (first names are still first names), surnames, policy and identification numbers also meet the validation criteria. Additionally, testers can maintain data integrity in complex multi-system and multi-database environments. The process of preparing data can be scheduled to run on demand or called by an API (Application Programming Interface).  

Soflabs’ GALL solution ensures that only authorised individuals have access to sensitive data. One of its roles is that it prepares rules, implements anonymisation of algorithms and performs algorithm tests with data preview. Another role that it performs is the system can schedule and run previously created rules without access to production data. The anonymisation process is irreversible.

Conclusion

Soflab’s GALL can fulfill specific GDPR guidelines such as the right to be forgotten. Therefore it has proven to be very effective at creating test data from production data. As a result, GALL has been incorporated into the TDM (Test Data Management System) where it can comprehensively manage the creation of test data in test environments.

Test data in TDM is created using GALL and other tools. This is achieved in TDM full coverage of test data preparation for different tests. It happens by utilising GALL via API, automated tests during “test data exhaustion” and does not stop. GALL on demand supplies the test environment with a new set of test data to meet the specified criteria. 

In developing the GALL solution, Soflab have met and exceeded the challenges that relates to the use of production data. This is important for the generation of test data such as GDPR guidelines, access to sensitive information and the integrity of data. It can manage large-scale test data in complex environments using anonymised production data which has made the validation process much faster. 

Time needed for testing is an important factor as organisation’s consider time to market to be one of their most critical KPIs. So how can they further improve their time to market strategies? One of the first steps is to understand what is required to improve the situation. Second step is to develop the correct test data strategy. The third step is to use the appropriate tools to create, generate and distribute the resultant test data. 

Please find articles in German about SOFLAB on this website:

7 Tipps für die erfolgreiche Umstellung auf SAP S4 Hana

Wir haben die Softwareentwickler, Tester und QA Experten, die Sie suchen!

Softwaretesting ist der Schlüssel zum Erfolg!

The author of this article is Karol Mioduszewski

Sales Director SOFLAB Technlology

Expert with 16 years of experience in software development and quality assurance. Possesses broad and practical knowledge related to leading IT projects. For over 6 years associated with the management of sales and marketing department.