Paper: pdf
Slides: pdf
Halfaker, A., Kittur, A., Kraut, R. E., & Riedl, J. A Jury of Your Peers: Quality, Experience and Ownership in Wikipedia. WikiSym'09 ACM, New York, NY, USA, (pp. 15:1-10).
@inproceedings{halfaker09jury, author = {Halfaker, Aaron and Kittur, Aniket and Kraut, Robert and Riedl, John}, title = {A jury of your peers: {Q}uality, {E}xperience and {O}wnership in {W}ikipedia}, booktitle = {WikiSym '09}, year = {2009}, isbn = {978-1-60558-730-1}, pages = {1--10}, location = {Orlando, Florida}, doi = {}, publisher = {ACM}, address = {New York, NY, USA} }

A Jury of Your Peers: Quality, Experience and Ownership in Wikipedia

Wikipedia is a highly successful example of what mass collaboration in an informal peer review system can accomplish. In this paper, we examine the role that the quality of the contributions, the experience of the contributors and the ownership of the content play in the decisions over which contributions become part of Wikipedia and which ones are rejected by the community. We introduce and justify a versatile metric for automatically measuring the quality of a contribution. We found little evidence that experience helps contributors avoid rejection. In fact, as they gain experience, contributors are even more likely to have their work rejected. We also found strong evidence of ownership behaviors in practice despite the fact that ownership of content is discouraged within Wikipedia.



Research Question: How do quality, experience and ownership effect which contributions are rejected in Wikipedia?


In order to answer this question, we constructed a logistic regression model with the boolean outcome variable whether or not a revision was reverted and explanatory variables related to the experience of the editor making a change, the recent quality of their work and the ownership structure of the edited version of the article.

We randomly sampled 1.4 million revisions from the January 2008 dump of English Wikipedia. Within our model, a revision is an observation.

Figure 1. The ideal model of rejection.

Figure 2. The actual model of rejection.

Two models of the effects of quality, experience and ownership on the probability of being reverted. On the left is the hypothesized "ideal" model. On the right is the actual model modified for the effects we saw in Wikipedia. (+) represents a positive correlation. (-) means a negative correlation. (0) means no correlation.

First we propose an ideal model(left) of how quality experience and ownership should effect the probability of being reverted. We then apply our sample to the model to determine the actual effects. Then we describe how the ideal model deviates from the actual model(left) in Wikipedia

Major Contributions

The following sections represent the three major contributions of this work.

Word persistence as a measure of contribution qaulity.

Figure 3. Persistence of words. A toy example of six revisions of an article. In this example, the editor named "Steve" adds the word "apple" to the article in the first revision. That word then "persists" for at least five subsequent revisions.

For this work, we developed a metric based on previous work by Priedhorsky et al. (2007) and Adler et al. (2007) that we call word persistence. Word persistence is simply a measure of how long a word lasts through the revision history of an article. We use word persistence to estimate the quality of a contribution based on the assumption that editors who edit an article act as reviewers of the content of the article. We assume that the more revisions that take place which do not remove the word from the article, the higher quality in quality the original contribution that added it is.

Beyond simple tests for interference with other variables and independent significance in our statistical prediction model, we found a significant correlation between editors with a recent history of high word persistence edits and increases in the Wikipedia 1.0 Assessment rating of the articles they edit.

Editors don't exhibit a learning effect.

We hypothesized that editor would exhibit a learning effect in Wikipedia through which the probability of their changes being rejected would decrease as they gained experience. We did find that experience was useful for predicting when a change would be rejected; however, when we controlled for the amount of time an editor would eventually continue to edit Wikipedia, we found that the effect we saw was not, in fact, an effect of experience. Instead, there seems to be a drop-out effect whereby editors who are reverted often do not stay long in Wikipedia.

Figure 4. The (lack of a) learning effect. The probability that a revision is reverted is plotted by the tenure (experience) of that editor. The plot on the left represents edits by all editors while the plot on the right represents edits by editors who continued editing for at least a year.

As figure 4 suggests, the probability of a contribution rejected appears to be effected by tenure when we sample all editors(left), but when we control for the lifetime, the amount of time and editor will continue to edit, by subsampling only those editors that will stick around for at least a year(right), the effect of tenure disappears.

Ownership has a powerful effect over rejection of contributions.

Figure 5. The effect of ownership. The probability of a revision being reverted is plotted by the number of toes that were stepped on.

When we examined the effect of our metric for ownership, stepping-on-toes or "removing words added by other editors when they are likely to notice", we found that it had both a powerful and independent effect the probability of a change being rejected in Wikipedia.

As figure 5 suggests, the number of "toes" stepped on by an editor was a strong predictor of rejection. In other words if an editor removes the work of nine other editors when they are likely to notice your change, you have approximately a 50% chance of being reverted.

The number of "toes" stepped was a strong predictor consistently across all types of editors in our model. Essentially, what that means is this: No matter how much experience you have, how high in quality your edits generally are or how often your changes are accepted by the Wikipedia community, if you step on other editors toes, that probability that you are reverted increases all the same.