Browse Prior Art Database

Method for Creating a Ranking List using Crowd Sourcing Service

IP.com Disclosure Number: IPCOM000237449D
Publication Date: 2014-Jun-18
Document File: 4 page(s) / 40K

Publishing Venue

The IP.com Prior Art Database

Related People

Roger Jie Luo: INVENTOR [+4]

Abstract

A method is disclosed for creating a ranking list using crowd sourcing service. The method includes combining pairwise preference of items provided by crowdsourcing participants into a full ranking list. Additionally, a Thurstonian Pairwise Preference (TPP) model is implemented to infer distribution of gold standard ranking lists as parameterized by relevance scores. The TPP model learns domain expertise and truthfulness of crowd sourcing workers as well as query difficulty.

This text was extracted from a Microsoft Word document.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 51% of the total text.

Method for Creating a Ranking List using Crowd Sourcing Service

Abstract

A method is disclosed for creating a ranking list using crowd sourcing service.  The method includes combining pairwise preference of items provided by crowdsourcing participants into a full ranking list.  Additionally, a Thurstonian Pairwise Preference (TPP) model is implemented to infer distribution of gold standard ranking lists as parameterized by relevance scores.  The TPP model learns domain expertise and truthfulness of crowd sourcing workers as well as query difficulty.

Description

Disclosed is a method for creating a ranking list using crowd sourcing service.  The method includes combining pairwise preference of items provided by crowdsourcing participants into a full ranking list.  Additionally, a Thurstonian Pairwise Preference (TPP) model is implemented to infer distribution of gold standard ranking lists as parameterized by relevance scores.  The TPP model learns domain expertise and truthfulness of crowd sourcing workers as well as query difficulty.

In accordance with the method, the TPP model is built on top of a Thurstonian ranking model.  In a rankling task for the Thurstonian ranking model,  workers are asked to rank  documents for annotating ranking lists.  It is postulated that ranking is determined by latent random variables , which is an assessment of relevance for document  perceived by worker , as Gaussian distributed with unknown mean  and variance   The -th judge produces a ranking outcome by sorting values of  for different document .  The goal is to infer  for each document  after observing the rankings made by  judges. The higher value of , the more relevant document  is.

Thereafter, a set of  queries  and  documents  in which pairwise preferences are made by  crowd workers .  The queries are from  domains.  Each worker  reads the queries and documents, then  perceives a latent relevance assessment  for each document  corresponding to query   It is postulated that  is drawn from a Gaussian distribution with mean  and variance , where  is the gold standard relevance score for document  over , and  measures the difficulty of the query.  In order to generate a pairwise preference, worker  chooses two documents  and  over query .  The pairwise comprision is denoted as , .  The  domain  of query  is represented as one single parameter .  Thereafter, two noisy scores  and  are received for two documents respectively.  In the TPP model, the noisy score  (or ) of document  (or ) is distributed as a Guassian with mean  (or ) and variance .  A positive value of  indicates that the annotator is truthful while negative  is identified as malicious worker.  Additionally, absolute value of  also implies knowledge of worker  on domain .  With large , the variance is small and thus noisy scores  (or ) is deviated slightly from perceived scores  (or ).  After noisy scores are sampled, a comparison ...