Dismiss
InnovationQ will be updated on Sunday, Oct. 22, from 10am ET - noon. You may experience brief service interruptions during that time.
Browse Prior Art Database

Method and System for Selecting Key Short-Term Metrics to Predict Long-Term User Behavior in Controlled Experiments

IP.com Disclosure Number: IPCOM000241139D
Publication Date: 2015-Mar-31
Document File: 5 page(s) / 39K

Publishing Venue

The IP.com Prior Art Database

Related People

Miao Chen: INVENTOR [+3]

Abstract

A method and system is disclosed for selecting key short-term metrics to predict long-term user behavior in controlled experiments. The method and system defines best short-term metrics in online controlled experiments, to quickly evaluate the effect of new features on a product, with the goal of increasing both short and long term-term user engagement with the product.

This text was extracted from a Microsoft Word document.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 22% of the total text.

Method and System for Selecting Key Short-Term Metrics to Predict Long-Term User Behavior in Controlled Experiments

Abstract

A method and system is disclosed for selecting key short-term metrics to predict long-term user behavior in controlled experiments.  The method and system defines best short-term metrics in online controlled experiments, to quickly evaluate the effect of new features on a product, with the goal of increasing both short and long term-term user engagement with the product.

Description

The goal of most online controlled experiments is to quickly evaluate the effect of new features on a product, with the goal of increasing both short and long-term user engagement.  To achieve this goal, it is critical to select the best short-term metrics to measure in an experiment that predict long-term user behavior.  

Disclosed is a method and system for selecting key short-term metrics to predict long-term user behavior in controlled experiments.  The method and system determines the key short-term metrics using a statistical approach which depends on available long-term metrics and long-term goal.  Depending on the given long-term goal and available long-term metrics, different short-term metrics are evaluated using a multiple regression model.  The evaluated short-term metrics are thus quantified and ranked based on importance and with regard to the long-term goal of driving user engagement.  

In accordance with the method and system, various parameters such as user cohorts, time period, and metrics of interest are initially defined.  Subsequently, user level data is obtained and each metric is standardized to evaluate short-term metric using a multiple regression model.  Multiple regression model studies the correlation between short-term metrics and the metrics reflecting long-term user engagement.  The fitted model returns pairs of values for each of the short-term metrics such as, p-value, and coefficient.  The p-value determines whether the corresponding metric is a significant predictor for long-term user engagement or not and the coefficient determines the magnitude of correlation.  Based on these values, the experiment owners can select various sets of short-term metrics to focus on according to their long-term goal.

Key short-term metrics are selected by studying the correlation between short-term metrics and long-term metrics.  Based on long-term goals, the product team can define long-term user engagement metrics that best describe the goals. For example, if a product team aims to make users use the product as a daily habit, which allows to measure total days visited per user.  Alternatively if the product team aim to increase certain usage of the product, such as, sending email or search queries, total email sent per user or total search queries per user are measured.  Further, the product team takes ads click per user as the long-term metric while aiming to increase ads clicks.  Different long-term metric...