Automation in Virtualized Environment
Publication Date: 2013-May-07
The IP.com Prior Art Database
Virtualized Environment is widely used nowadays. However, automatically testing software installed in a virtualized environment is still a challenge, because, to the local operating system, a virtualized environment is a black box that hides applications within it. The problem we try to solve in this disclosure is to enhance GUI based test automation in a vitalized environment. The disadvantage of existing solutions are: 1. Difficult to create a test script. All target GUI objects needed to be specified manually. 2. The processing time grows linearly with the number of GUI objects involved in an automation task. This disclosure introduces an image processing approach to enable a natively installed automation software to interact with an application in a virtualized environment. Our approach firstly finds out landmark objects in a screenshot of a virtualized application; and then identifies the GUI instance of the application by predefined signatures; and then proceeds to locate other objects of interest, such as text fields and buttons, through coordinate transformation. Finally, interaction with a virtualized application is translated into keyboard/mouse actions towards those coordinates. The advantages of this approach are: 1. Fast and light weight. It is much more efficient than full image matching as only one or more landmark objects need to be identified. 2. Resilient. It is independent of the virtualized environment and OCR free. 3. It can be easily implemented and integrated to existing automation software.
Page 01 of 9
Automation in Virtualized Environment
In software engineering, graphical user interface (GUI) testing is the process of testing a product's GUI to ensure it meets its written specifications. To automate this process, automation tools need to programmatically get handles of GUI objects via API exposed by the product, and then interact with those objects like pressing buttons and typing in edit boxes. Traditional GUI object detection software such as test automation tool, windows spy tools, and accessibility test tool can find and interact with applications installed natively. Virtualization, such as virtual desktop and virtual application, is encapsulated in an isolated environment by virtualization layer. GUI objects' handles can not be directly obtained by automation softwares. As virtualization is used more and more frequently, a solution is needed to enable software to directly identify and interact with application objects inside the virtualization.
We will propose an algorithm to find the location of target GUI objects in a virtualized application. As a pure image processing approach, this method has inherent advantages of language free and virtualization technique independence. Compared with existing image-based approaches , it's a light-weighted algorithm which is easy to be implemented and integrated with existing automation software. Our method has time complexity $O(k)$ where $k$ is the number of ``landmark" objects. Previous image processing-based approaches  have time complexity $O(n)$ where $n$ is the number of target objects involved in the automation tasks, which increases when tasks get more complex. Typically, $k$ is much smaller than $n$, and therefore our method significantly reduces the time complexity of automation in
a virtualized environment, reducing substantially the processing time as image pattern recognition is a time-consuming task.
In section 2, previous works to address the problem will be introduced. In section 3, the major steps of our method will be introduced. In section 4, the technical detail of the method to locate objects in a virtualized application will be introduced. In section 5, we will detail how to integrate our approach to existing automation software.
2. Previous Work
Recognition technique has been employed in the testing automation. One approach is to make use of text extracted from OCR. A. Geva et al. introduced a framework to control a remote client by text information extracted by OCR. However, many GUI objects may not have visible text associated with them. Therefore, another approach, which is dominant, is proposed to use template matching to identify the occurrence
of specific image templates created before hand. One example is RoutineBot. Another example is Ranorex which employs a more robust recognition algorithm, taking neighborhood information into account to achieve better recognition rate. For these two software, the disadvantage is:...