Browse Prior Art Database

Virtual drive: Testing using simulation of real world sensor inputs

IP.com Disclosure Number: IPCOM000225331D
Original Publication Date: 2013-Feb-11
Included in the Prior Art Database: 2013-Feb-11
Document File: 7 page(s) / 114K

Publishing Venue

Microsoft

Related People

Ashish Gadre: INVENTOR [+2]

Abstract

In current Windows Phone® Location Service[1], End-to-End (E2E) testing is fractured into running individual test cases which may not represent the actual real world behavior of location service relying on different providers like Cell/WiFi/GPS. The location test team has traditionally heavily relied upon doing drive tests on roads for this kind of testing which turns out to be wasteful both in terms of time and man power. The verification in these tests is often visual rather than empirical. If any issues are identified and fixed the verification process is again the same time consuming process. Importantly the outdoor environment for these tests is neither constant, nor controllable. This makes the tests unreliable and not easily repeatable. Location service for Windows Phone® uses several internal software filters and algorithms to process the sensory inputs and compute Location. Currently there is no mechanism for effectively prototyping and validating the design and understanding the impact of those changes, thus forcing developers to come up with theoretical solutions and magic numbers for their designs and filters. Only way to validate if a new change would work is to do a drive test with the real device and monitoring the results manually. The current suite of automation tests for Windows Phone® location service lacks a long-haul test which validates various system behaviors in one complete test run. Instead the current tests focus is on validating different bits and components of the location service functionality. This publication outlines an approach where a golden-route regression test can be developed, which will cover the above mentioned gap, and this test shall behave exactly the same all the time, allowing easy detection of issues as a result of changes to location service code base. For E2E long term testing the location team currently relies on performing physical drive tests with the device. This is a good strategy for finding the E2E experience from a user perspective, but falls short on the following key points: • Pass/Fail criteria – The visual recording of the location on the map is a good indication that the user is tracked correctly. However it is not a good indication that the underlying location engine is behaving correctly. Example: When the GPS signal is lost location service should filter results from Cell/WiFi for certain duration to avoid sudden jump in the location reported. We will never know when this switch is happening by using a visual verification mechanism. • Test environment – Driving around outdoors gives us no control over the test environment. Location service behavior change, with respect to changes in GPS signal conditions, crossing Cell/WiFi coverage boundaries etc. To reproduce these environment-induced scenarios one has to drive in those areas physically while still only making a guess about the signal strength or boundaries. • Reliability of Test – Issue found during driving tests is hard to reproduce as the test environment is not constant. A result reported on one day may differ on the next at the same location. This adds to the frustration when we want to validate a location issue reported through the drive tests. • Long haul testing – The current tests have their own way of configuring the system and sometimes it means refreshing or rebooting the phone to achieve that. This causes loss of the state of location service between tests and thus hides some long term product bugs. • Time and Environmental Impact – Driving around with devices takes a lot of precious time, manpower and is also not a green solution. • No regression suite – Since the drive tests are costly, they can be run only at specific time intervals during a product development milestone. This can lead to finding bugs late in the product cycle which will be too difficult to fix.

This text was extracted from a Microsoft Word document.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 24% of the total text.

Document Author (alias)

AshishGa, ShRaghav

Defensive Publication Title 

Virtual drive: Testing using simulation of real world sensor inputs

Name(s) of All Contributors

Ashish Gadre, Sharath Raghavendra

 

 

 

 

Summary of the Defensive Publication/Abstract

In current Windows Phone® Location Service[1], End-to-End (E2E) testing is fractured into running individual test cases which may not represent the actual real world behavior of location service relying on different providers like Cell/WiFi/GPS. The location test team has traditionally heavily relied upon doing drive tests on roads for this kind of testing which turns out to be wasteful both in terms of time and man power. The verification in these tests is often visual rather than empirical. If any issues are identified and fixed the verification process is again the same time consuming process. Importantly the outdoor environment for these tests is neither constant, nor controllable. This makes the tests unreliable and not easily repeatable.

Location service for Windows Phone® uses several internal software filters and algorithms to process the sensory inputs and compute Location. Currently there is no mechanism for effectively prototyping and validating the design and understanding the impact of those changes, thus forcing developers to come up with theoretical solutions and magic numbers for their designs and filters. Only way to validate if a new change would work is to do a drive test with the real device and monitoring the results manually.

The current suite of automation tests for Windows Phone® location service lacks a long-haul test which validates various system behaviors in one complete test run. Instead the current tests focus is on validating different bits and components of the location service functionality. This publication outlines an approach where a golden-route regression test can be developed, which will cover the above mentioned gap, and this test shall behave exactly the same all the time, allowing easy detection of issues as a result of changes to location service code base.

For E2E long term testing the location team currently relies on performing physical drive tests with the device. This is a good strategy for finding the E2E experience from a user perspective, but falls short on the following key points:

·         Pass/Fail criteria – The visual recording of the location on the map is a good indication that the user is tracked correctly. However it is not a good indication that the underlying location engine is behaving correctly.

Example: When the GPS signal is lost location service should filter results from Cell/WiFi for certain duration to avoid sudden jump in the location reported. We will never know when this switch is happening by using a visual verification mechanism.

·         Test environment – Driving around outdoors gives us no control over the test environment. Location service behavior change, with respect to changes in GPS signal conditions, crossing Cell/WiFi c...