Browse Prior Art Database

Rapid Experimentation Infrastructure (REI) for Web applications

IP.com Disclosure Number: IPCOM000238202D
Publication Date: 2014-Aug-07
Document File: 4 page(s) / 276K

Publishing Venue

The IP.com Prior Art Database

Related People

Linu Koshy: AUTHOR [+6]

Related Documents

148376: IP.COM

Abstract

The REI framework enables product managers and developers working on online web applications to release experimental features into production, get their experiments exposed to the right target customers, make quick modifications and iterate, increase or decrease their target audience, and even shut down the experiment at any time independent of release cycles. Teams can perform all these activities with the guarantee that the stable production application will be unaffected.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 43% of the total text.

Page 01 of 4

Rapid Experimentation Infrastructure (REI) for Web applications

Abstract

The REI framework enables product managers and developers working on online web applications to release experimental features into production, get their experiments exposed to the right target customers, make quick modifications and iterate, increase or decrease their target audience, and even shut down the experiment at any time independent of release cycles. Teams can perform all these activities with the guarantee that the stable production application will be unaffected.

Problem Statement

Experiments in market with real users are normally difficult to conduct. Teams need to make sure that:

 Prototype or experimental functionality doesn't disrupt, degrade or otherwise regress the experience for more than a small number (e.g. less than 1%) of users.

 Experiments never compromise any user data or privacy.
 Experimental code doesn't pollute or lower the application's quality standard.

These constraints usually force product managers and engineers to go through the following process for every experiment:

1. Manually identify test users

2. Build and test feature enablement

3. Test for regressions across all features, even those unrelated to the experiment 4. Request code and security reviews and await approval

5. Wait for a scheduled production release of the application for the changes to be released into production

6. Request ad hoc data warehouse queries to collect and then manually aggregate results.

These activities are time consuming (multiple weeks) and are unrelated to the specific experiment from which the teams were trying to learn.


Page 02 of 4

Solution

The diagram below illustrates the architecture of a typical web application. The REI Traffic Split component allows a team to create a new version of the web application and release it in parallel to the current production version of the same application in a matter of minutes. It then permits the developer to specify what percentage of traffic should be directed to the experimental version. Increasing or reducing the traffic split percentage is a simple configuration change that takes effect instantly.

REI is the culmination of numerous earlier efforts to allow rapid experimentation in web applications. Earlier efforts included creating an A/B Test framework that allowed for different user experiences to be shown to users based on random bucket assignments or a Feature Enablement framework to allow user assignment to the experiment to be programmatically controlled (rather than random assignment). The problem with all these approaches was deploying an experiment into production still involved making code changes to the stable/production release of the application, doing the full regression testing, and then deploying it into production.


Page 03 of 4

While brainstorming on alternate approaches to solve these problems, the team had a realization that they needed to provide run-time isolation o...