Managing Crowdsourcing Workers
The emergence of online crowdsourcing services (like Amazon's Mechanical Turk) presents us huge opportunities to distribute micro-tasks at an unprecedented rate and scale. Unfortunately, the inaccurate verification mechanism and unstable employment relationship give rise to opportunistic behaviors of workers, which in turn exposes the requesters to quality risks. we present an algorithm to estimate the quality of the workers by redundancy, which can easily separate the true (unrecoverable) error rate from the (recoverable) biases. Also, we can seamlessly integrate the existence of "gold" data for learning the quality of workers. In addition, we are trying to bring up an active learning approach for testing worker quality using gold label or imperfect label.

An Experimental Study of Cooperation in Evolving Social Networks
We study how cooperation emerges in a dynamic network environment where individuals are able to choose their actions and change who they are interacting with.

Paid and Unpaid Reviews
We plan to examine whether "paid" user‐generated content has any different characteristics compared to content generated by "unpaid" volunteers. If so, are consumers aware of that? The experiment is currently running on Amazon Mechanical Turk.

Estimating the Completion Time of Crowdsourced Tasks
We model the completion time as a stochastic process and build a statistical method for predicting the expected time for task completion. We use a survival analysis model based on Cox proportional hazards regression. Our model show how time-independent variables of posted tasks (e.g., type of the task, price of the HIT, day posted, etc) affect completion time.

Continuing work on Sheng et al's paper "Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers". With low-cost labeling driven by online micro-outsourcing systems such as Amazon's Mechanical Turk, it is possible for us to improve both data quality and model quality by repeated labeling within budgeted cost. In previous work, we've seen great performance of several selective strategies on data accuracy. Recently, we studied a novel label uncertainty estimation strategy (NLU) by taking the underlying class distribution into account. Also, we update the old LMU strategy by integrating NLU with MU. Moreover, we attempted to improve the classification accuracy in two aspects:

• Soft Labeling. Although the old majority voting worked well, the uncertainty differed across instances. Soft-labeling assigned two copies (one positive, one negative) to each single instances, with weights equal to the certainty score of the particular selective repated-labeling strategy.
• Weighted Sampling. In previous study, each instance was chosen with abosolute priority. Here, the probability of being chosen is propotional to the uncertainty score of each instance under the particular selective strategy.

Exploring the negative correlation between labelers. For most cases, we assume that labelers are independent, but this is neither true nor helpful for ensuring higher accuracy. There are opportunities as well as chanllenges on infering the correlation between labelers.

• Benefit Side. Negative correlation between labelers (as long as the quality is good than random) can give us better results than pure independence.
• Cost Side. Higher results appears when the quality is high, the number of labelers is big, and the examples per labelers is relatively small, which are irrelevant with the true correlation.

The realtionship between the payment to turkers and the outcomes. I designed three experiments: judging the emotion of the person in the image[See an example], rating movie reviews[See an example], and solving the chess puzzles[See an example]. The results confirm the claim that higher payment will induce better performance. Also, we know from this that turkers complete online micro-outscouring work not only for fun and challenge, but also for money.