How to automate GPU testing on multi-boot machine

It is always very time consuming process to ensure each new product release is stable and works flawlessly on all supported platforms. Virtualization helps a lot in every QA department when it comes to automation tests. But what if you need to work and test with GPU cards. Is virtualization still the right answer to automate GPU testing?

Comprimato SDK for JPEG 2000 supports a large variety of different hardware on all mainstream operating systems. Processors as well as large number of graphic cards from major vendors such as Nvidia and AMD can harness the power of our codec to deliver high speed and high quality image compression. When it comes to testing these numbers of software and hardware combinations virtualization should then be an obvious solution. Sadly there is no virtualization method that can fully utilise each one of our large set of supported hardware and operating system combinations without some of the test results becoming spoiled. Although there are technologies such as Nvidia GRID they are not sufficient for our needs. Fortunately the QA team at Comprimato found a different way to achieve automated testing without virtualization and unnecessary expenses and created a whole testing framework based on it.

How to run multiple OS on the workstation

For companies like Comprimato it is more effective to have one station with multiple operating systems rather than having multiple sets of workstations with the same hardware and different operating systems. To make this happen we used so-called multi-boot. If you have heard about dual-boot, you will understand multi-boot straight away. We managed to fully stack one hard disk with multiple different operating systems without affecting the others’ functionality. The tool GNOME partition editor (GParted) was used to split the disk into the desired number of partitions. After that, several chosen operating systems were carefully installed on their own partitions with a special focus on the booting aspect so that each operating system can be booted from its own partition. Although there is one simple drawback in that we cannot run these operating systems at the same time, we managed to obtain a test environment that fully satisfies our needs for accurate and correct testing and measuring of Comprimato SDK for JPEG 2000. But multi-boot was just only one part of the puzzle. Leaving out the human factor from the testing process and making it automatic was the second main part.

Here you can see how our testing workstations can be booted into demanded operating system.

How to automate OS setup for testing

While multi-boot kept hardware and software prices at the lowest cost possible, this soon-to-be testing framework still needed something to work automatically. We prepared several independent computers and preinstalled multiple operating systems on them thanks to multi-boot. The challenging problem then was to compel each one of those computers to boot into desired operating system without direct user interaction. The simplest way to achieve this was by using the standardized client-server environment known as Preboot eXecution Environment (PXE). So we build our testing framework around one custom DHCP server (or we can say PXE server) which manages other directly connected testing computers and provides them with information of what operating system to boot. It also gave us the ability to programmatically control those computers and opened the way to fully automatize our testing. Consequently we used the Jenkins Continuous Integration server to ensure this. Jenkins is based on Java and therefore it easily solved a lot of cross system compatibility issues. It also provided us with a way to execute arbitrary test scripts or programs on each one of our testing computer and a way to remotely boot the testing computer into the operating system currently needed. By putting these things together we were able to automate GPU testing.

Testing workflow for one testing workstation: one test is runned by multiple configurations

What we were able to get from that

Our testing framework has now been working for almost a year. It is able (when fully utilised) to execute hundreds of tests per day. We can relatively easily add new testing computers to ensure the compatibility of Comprimato SDK for JPEG 2000 with the latest GPUs or CPUs. We also elevated our DHCP/PXE server into some sort of test manager. Based on database and PHP with web interface it now works autonomously as much as possible. We are also thinking about extending this framework to several of our other projects thanks to the fact that it is modular and extensible. How do you test GPU or specific hardware? Also, if you would like to know more, leave a comment below or feel free to contact us at testers@comprimato.com.