Research talk:Exploring systematic bias in ORES/Work log/2019-05-19

Sunday, May 19, 2019

"Counterfactual unfairness" is reasonable notion of fairness that we can operationalize without ground-truth about "correct" decisions. When applied to reversion on Wikipedia, this notion of fainess asks: "Would this edit have been reverted if the editor were not a newcomer?" This intuitive question can be formalized in terms of the probability an edit is reverted ($P(R)$), conditional on the class of editor ($A\in \{\mathrm {anon} ,\mathrm {newcomer} ,\mathrm {established} \}$) and other factors relevant to the edit, which are understood as appropriate and ethical reasons to revert an edit ($X$). The difference in probability conditional on the class of editor measures counterfactual unfairness.

$P(R|X=x,A=\mathrm {newcomer} )-P(R|X=x,A=\mathrm {established} )$

We are interested in how introducing ORES-powered RCFilters effects unfairness on Wikipedia, which is written: $[P(R|X=x,A=\mathrm {newcomer} ,D=\mathrm {has~ores} )-P(R|X=x,A=\mathrm {established} ,D=\mathrm {has~ores} )]-$ $[P(R|X=x,A=\mathrm {newcomer} ,D=\mathrm {no~ores} )-P(R|X=x,A=\mathrm {established} ,D=\mathrm {no~ores} )]$

We can estimate this difference using logistic regression models for each wiki predicting whether an edit was reverted as a function of $X,A\mathrm {~and~} D$.

$R\sim \mathbf {B} {X:wiki}+cA+dA\land D$ and the average causal effect on the probability a newcomer is reverted will be $logit^{-1}(d_{\mathrm {newcomer} }-d_{\mathrm {established} })$

We can put this into our DID frameworks by including treatment status, predictors of treatment, and IP weighting in the model.