A Brief Overview of Opentest Framework in Testing Automation

Opentest is a test automation framework designed to address the need of testing embedded software more efficiently by sharing resources across multiple users. The tedious process of bringing up hardware together with all new images is made extremely simple by just executing an automation test with help of Opentest framework automation. Enabling the use of single testcase for over 20+ different platforms makes the framework more efficient. Opentest provides a complete test automation solution by using Testlink. Opentest along with Jenkins  provides a good DevOPs solution.

Opentest is an open-source test framework suitable for embedded software testing. Opentest is based on STAF(Software Testing Automation Framework) framework which is open-source, multi-platform and multi-language framework. STAF works on the concept of re-usability and extensibility and achieves it by using STAF services. These STAF services provide different functionalities using which we can use exiting STAF service or define new  STAF services depending on the requirement. Opentest adds new STAF services on top of existing services.

 For setting up Opentest framework, we require below modules

1. Test Management system(TMS)
2. Test Master Control(TMC)
3. Build execution engine(BEE)
4. Test execution engine(TEE)
5. Service provider(Optional)

Below diagram shows the test execution flow within the Opentest framework.

Testlink is used as TMS which is an open-source test management system(TMS). Testlink supports creation of multiple test projects, creating multiple testplans, creating thousands of testcases, maintaining the testcase version, generating test reports in different formats(word,excel,etc). Testlink allows us to specify the Hardware we want to test and the machine were the Hardware is connected for test execution. We can add automation as well as manual tests under any testplan. We can link different bug tracking tools to testlink for traceability.

The main function of TMC is to keep track of capabilities of BEE, TEE and service providers and to redirect the request from TMS to appropriate BEE,TEE and service provider according to test request. The capabilities of BEE,TEE and service providers are maintained by resource manager module of TMC and the requests are directed by dispatcher module of TMC.TMC writer passes the test results from TEE back to TMS. Those registered to TMC can only be a part of the framework so the BEE,TEE and service providers have to register themselves to TMC once they are up.

To run a test we require the software(build) and hardware(device) on which test is performed. TMC by co-ordinating with BEE and TEE make sure the test gets the software required and the hardware on which test needs to be performed. BEE provides the software required for test. TEE does the actual test execution on the  hardware connected to it. The functionality of TEE includes bring up the hardware with the software from BEE, configure the hardware, perform the test and sending back the test result to TMC.

Each component of Opentest framework communicates with STAF request/Response message. So each Opentest module should have STAF installed and running on it to make the communication.

Below are the steps that are executed when we run a test from Opentest framework and are shown in diagram above

1. To execute any test, first create a test project, testplan containing testcases in Testlink.
2. Along with test execution steps we can also mention the software we want to use for test,the TEE and the hardware on which test needs to perform in Testlink. As there can be multiple hardware device connected to single TEE so we can even differentiate that in testlink.
3. Once the test is executed from testlink the request in the form of STAF+ XML request is send to dispatcher module of TMC.
4. Dispatcher sends information to resource manager so that the right BEE and TEE are assigned according to the capabilities.
5. BEE is assigned and the software required for test is taken from BEE and provided to TEE. The build is copied in TEE under /staf directoy (staf/data/STAF/user/sw_assets) of TEE and used for test execution.
6. The test execution steps are passed to TEE from the xml request file we get from testlink and the test is performed on the hardware connected to TEE. There may be need of additional service provider to perform the test which is assigned by resource manager.
7. After the test is completed the testresult are stored on TEE under /staf directory (staf/data/STAF/tmp) and are passed to dispatcher and then to TMS writer to save the results to testlink. The request xml and the result xml are also stored in this location. So in case we want to manually run the test on TEE we can take this request xml and run the test.
8. We get the test result PASS/FAIL for the testcase. The logs are saved to log server and the reference is passed to testlink so that we can analyse the logs.

We can get the latest version of Opentest installation under below mentioned path

http://arago-project.org/files/releases/opentest/

This installer provides the installation of STAF, Testlink, TMC, VATF, TEE, BEE and command line tools. VATF is Video Audio automation framework  written in ruby. It was developed to automate Digital Video Software development kit(DVSDK) system test cycle but it is generic enough to be used to automate test suites on other products as Linux,Android, etc. VATF provide services like reading test parameters from test suite database, initialize all equipment required to run the test and call the appropriate test script.

We have successfully built an Opentest framework for testing multicore SDK for more than 20+ devices and we were able to meet the hard release deadlines due to complete automation solution provided by Opentest. Also, by integrating Jenkins, DevOps solution is provided for the project.

eInfochips has been working on test automation frameworks for various projects in IoT, semiconductors, medical devices and more. Check this white paper on one of our frameworks called EzTest.