System and Methods for Record & Replay of Web-based Mobile Tasks
Publication Date: 2010-Aug-09
The IP.com Prior Art Database
In this patent disclosure we describe SessionTag, a system and associated methods for recording (or tagging) and replaying of mobile web-based tasks with three distinctive features: 1. Posterior automatic session tagging: SessionTag allows user to tag a just completed session without first signaling start of recording. SessionTag can automatically find the start of the session and eliminate interaction noise without any user involvement. 2. "Smart" session replay: SessionTag adapts to contextual differences when user tagged the session and when user replays the session, essentially allowing parameterization of a tagged session. SessionTag also adapts to changes in web pages, both to withstand dynamics of the web and to accommodate to changes in user tasks. 3. Remote session execution: Instead of re-executing event s from the mobile device, SessionTag replays much of tagged session at a proxy server with fixed-line high speed connection. Such remote execution and other optimizations reduce time for replaying a session to the order of time for loading the last page within the session. Conceptually, the SessionTag system can be split into two parts: the front-end components and the back-end components. While the back-end components implement SessionTag's distinctive features mentioned earlier (and discussed in detail in later sections), the front-end components deals with interactions with the user. Specifically, the front-end components monitor the interaction state and interaction context of the user, where interaction state is defined by the following set of member variables: 1. The page that is currently open in the browser; 2. The highlighted text on the current page; 3. Content of the clipboard; 4. Form inputs made to the current page, and 5. The current value of the URL input bar (if any). Interaction Context is then defined as the user's most recent sequence of interaction states. The transition from one interaction state to another corresponds to an interaction event. SessionTag front-end components intercept interaction events from within the browser to closely monitor the state of user interaction. Specifically, there are following types of interaction events in SessionTag: 1. Page-Load event, which corresponds to page loading resulted from user actions such as clicking a hyperlink, hitting enter key in the URL bar, clicking a form submit button, or clicking certain browser toolbar buttons or selecting some browser menu items (e.g., the "Home" and "Back/Forward" buttons and the bookmark menu). 2. URL-Input event, which corresponds to user typing in the URL bar. 3. Text-Highlight event, which corresponds to user highlighting a text string on the current page. 4. Text-Copy event, which corresponds to user copying the highlighted text to the clipboard. 5. Form-Input event, which corresponds to user inputs to an input field of a form on the current page. A Form-Input event can be caused by a Text-Paste event or simply by user typing text in an input field. 6. Paste-Input event, which is a special form of Form-Input event, and corresponds to user copying clipboard content to an input field of a form on the current page. 7. URL-Copy event, which corresponds to copying the URL of the current page to the clipboard. Besides their own unique attributes, all types events also have common attributes indicating their start and end times. Analogous to bookmarking the page after viewing a page of interest, a user can ask SessionTag to record, or "tag", a task that he has just completed and that he considers not easy to conduct but will likely be repeated in the future. The user can also assign a name to this just recorded session (Figure 2(a)). We call this style of session recording "posterior session tagging" as the "recording" happens without signaling of start and happens after the session has been completed. In the background, the SessionTag back-end gets the user's current interaction context, from it determines what events and states represent the session that the user intends to record and tag, and stores such events and states along with the assigned name. Later when the user, through the SessionTag front-end user interface (Figure 2(b)), tries to replay the tagged session, the back-end components of SessionTag get the interaction states and events corresponding to the tagged session, then attempts replay the recorded events of the session. However, such replay is typically not re-execution of exact "carbon copies" of the original events. This is because the user's interaction context (and hence his intended task) at the moment could be different from when he created the session tag. The other reason is that the web pages that the user visited when the session is tagged could have since changed, to the extent that verbatim replaying of some events no longer makes sense (such as highlighting non-existent phrases). The SessionTag back-end components thus need "smartness" in event replay to adapt to user context and web content changes. Finally, SessionTag back-end does not target faithful emulation of manual user interaction in session replay. Instead, it exploits various approaches, including remote execution of most of the session, for more efficient (and thus more usable) replay of tagged session.
1. TITLE OF THE INVENTION
System and methods for record & replay of web-based mobile tasks
2. INVENTOR(S) INFORMATION
Dong Zhou, Ajay Chander, Yusuke Kushida
3. FIELD OF THE INVENTION
This patent disclosure is in the field of easy recording and replaying of web-based tasks for mobile handset users.
4. BACKGROUND OF THE INVENTION
The conflux of mobile broadband and advanced mobile platforms (such as the iPhone, the Android, and the WebOS platforms) has greatly accelerated the wide spread use of mobile devices for web-related everyday tasks. Consumers increasingly use their smartphones to shop online, to blog or comment whenever they feel like to, and to check the current status of certain dynamically changing online information (such as play-by-play information of the game your favorite team is playing, or the soonest expiring auction item of something that you collect).
Nevertheless, compared with PCs with fixed-line high-speed connection, conducting web- related tasks from mobile devices still has several usability related concerns, which have roots in two of the inherent characteristics of such mobile devices: 1) weaker and less stable connection; and 2) small form-factor.
Despite recent progress in technology, cellular network connections are still relatively
slower and less stable than their fixed-line counterparts, especially when the device is close to cell edges and/or when there are severe interferences. The result of this is longer and less predictable network latency and user wait time, and consequently worse user experience, especially for web sessions involving sequences of user actions and Internet requests/responses. The small form-factor of mobile devices also makes it much harder to interact with: the small screen makes it difficult to read what's displayed or to click/tap at the right places; the lack of full-sized keyboard and stable arm-support makes it difficult to input text, or to move data from one application to another.
With that in mind, imagine a user browsing related news finds an interesting article at gizmodo.com (Figure 1(a)), and he wants to quote and comment about the article on his blog. To do that, he highlights and copies an interesting paragraph of article, then goes to blogger.com and log in by typing in username and password (b). The user next clicks "New Post" to create a new blog entry (c), where he pastes the previously copied paragraph from the gizmodo.com article, possibly as well as the URL of the article (d). This seemly simple task is much more than trivial on mobile devices, not only because it is difficult for the user to type and copy/paste, but also because the connectivity of the device may have turned worse and user needs to wait for multiple pages to slowly load.
4.1 Prior Art Solutions
WebVCR provides a VCR style user interface through which the user can record and replay browsing steps . WebVCR is intended to be used as shortcuts for accessing web data that...