Surety is performing system maintenance this weekend. Electronic date stamps on new Prior Art Database disclosures may be delayed.
Browse Prior Art Database

A Method & System of Image Structure Based Automatic Testing

IP.com Disclosure Number: IPCOM000237775D
Publication Date: 2014-Jul-10
Document File: 6 page(s) / 157K

Publishing Venue

The IP.com Prior Art Database


This article introduces a new kind of image based object recognition method for automation testing. This method can solve the issues of cross platform, various resolutions/colors/styles revealed by traditional automation testing methods.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 55% of the total text.

Page 01 of 6

&&& System of Image Structure Based Automatic Testing System of Image Structure Based Automatic Testing

Problem statement and background
Choosing the right toolis still a big problem for test automation, especially for mobile GUI test!

Cross platform, multiple app types, rapid script creation, low cost maintenanceshould be the major factors for consideration.

The application objects are the main building blocks of test automation. They consist of edit boxes, lists, links, text objects, etc. In the mobile space there are mainly two ways to recognize objects, each method has several solutions in the market now:

Meth od Nam e

A Method

A Method

Rec og niz ed Obj ect s

How they handle object recognition.


OS nati ve obj ect s

Works well on devices with same OS, but different resolution


Virtu al Obje ct Reco gniti on

(Non -Ima ge base


differe nt solutio ns for native object

UI Automation APIs for different mobile platforms:

a. Using javascript interface -- Access its UI element hierarchy by "UIATarget.localTarget().logElementTree();"

b. Using uiautomator API -- Access its UI element hierarchy by uiautomatorviewer tool:


Page 02 of 6

<img height="44" id="apthumb3"

src="./image.jpg" width="44" border="0">

and DOM object



develo pment skill neede d



stabilit y of test execut ion



cost and time intensi ve of script creati on cross platfor m Differe


nt DOM tree in variou s

DO M obj ect s


Page 03 of 6


ms results in low reusab ility of scripts

Vis ual Obj ect Rec ogn itio n

(Im age bas ed)

Im age Obj ect

 Object identificati on is totally based on image recognitio
However, the images/ic ons are easily affected by different OS/resolu tion/langu age.

Look into the traditional Image based automation test, the problems exposed are quite obvious:


Page 04 of 6

If there are several similarimages on the same screen, manual assistance should be needed for identifying.

If the color or styleof image has changed, object recognition should fail.

Different screen resolutions would make image sizechange, object recognition may fail.

High effortsare needed for script maintenance.

Our solution is aimed to overcome the obstacles above and get more closed...