Research talk:Revision scoring as a service/Work log/2016-02-23
Tuesday, February 23, 2016[edit]
OK. Today I'm trying to do what we were doing with Urdu Wikipedia but with Polish Wikipedia instead. Here's list of 500K randomly sampled edits: http://quarry.wmflabs.org/query/7543
Prelabel is running now Amir (talk) 18:24, 23 February 2016 (UTC)
OK. It's done:
(3.4)ladsgroup@ores-compute:~/editquality/datasets$ wc plwiki.prelabeled_revisions.500k_2015.tsv 499736 1933819 13821243 plwiki.prelabeled_revisions.500k_2015.tsv (3.4)ladsgroup@ores-compute:~/editquality/datasets$ cat plwiki.prelabeled_revisions.500k_2015.tsv | grep "True" | wc 82484 264812 1720937 (3.4)ladsgroup@ores-compute:~/editquality/datasets$ cat plwiki.prelabeled_revisions.500k_2015.tsv | grep "reverted" | wc 14861 59444 416108
So 16.5% of edits needs review. That's good :) and 3% are reverted.
I sampled 5K to load it up to Wikilabels:
(
echo "rev_id\tneeds_review\treason";
(
cat datasets/plwiki.prelabeled_revisions.500k_2015.tsv | \
grep "True" | \
shuf -n 2500; \
cat datasets/plwiki.prelabeled_revisions.500k_2015.tsv | \
grep "False" | \
shuf -n 2500 \
) | \
shuf \
) > datasets/plwiki.revisions_for_review.5k_2015.tsv
Using shuffle, I extracted 20K revs to build the reverted model:
cat datasets/plwiki.sampled_revisions.500k_2015.tsv | \
shuf -n 20000 > datasets/plwiki.sampled_revisions.20k_2015.tsv
Then we should add "rev_id" to the first line and check if "rev_id" is not accidentally added to revs. (check) Then running label reverted:
cat datasets/plwiki.sampled_revisions.20k_2015.tsv | \
./utility label_reverted \
--host https://pl.wikipedia.org \
--revert-radius 3 \
--verbose > datasets/plwiki.rev_reverted.20k_2015.tsv
It's labeling them.
Now I'm extracting features:
cat datasets/plwiki.rev_reverted.20k_2015.tsv | \
revscoring extract_features \
editquality.feature_lists.plwiki.reverted \
--host https://pl.wikipedia.org \
--include-revid \
--verbose > \
datasets/plwiki.features_reverted.20k_2015.tsv
OK. I ran tuning reports and turned out RF is the best. Strange. Everything I touch turns into RF :D
Running with best settings:
> revscoring train_test \
> revscoring.scorer_models.RF \
> editquality.feature_lists.plwiki.reverted \
> --version 0.1.0 \
> -p 'max_features="log2"' \
> -p 'criterion="entropy"' \
> -p 'min_samples_leaf=7' \
> -p 'n_estimators=640' \
> -s 'pr' -s 'roc' \
> -s 'recall_at_fpr(max_fpr=0.10)' \
> -s 'filter_rate_at_recall(min_recall=0.90)' \
> -s 'filter_rate_at_recall(min_recall=0.75)' \
> --balance-sample-weight \
> --center --scale \
> --label-type=bool > \
> models/plwiki.reverted.rf.model
2016-02-23 22:21:47,424 INFO:revscoring.utilities.train_test -- Training model...
2016-02-23 22:22:08,411 INFO:revscoring.utilities.train_test -- Testing model...
ScikitLearnClassifier
- type: RF
- params: oob_score=false, scale=true, center=true, warm_start=false, criterion="entropy", random_state=null, max_leaf_nodes=null, class_weight=null, n_jobs=1, n_estimators=640, min_samples_leaf=7, min_weight_fraction_leaf=0.0, verbose=0, balanced_sample_weight=true, min_samples_split=2, max_depth=null, max_features="log2", bootstrap=true
- version: 0.1.0
- trained: 2016-02-23T22:22:08.411624
~False ~True
----- -------- -------
False 3723 155
True 50 66
Accuracy: 0.9486730095142714
PR-AUC: 0.327
Filter rate @ 0.9 recall: threshold=0.08, filter_rate=0.736, recall=0.905
Recall @ 0.1 false-positive rate: threshold=0.918, recall=0.017, fpr=0.0
Filter rate @ 0.75 recall: threshold=0.235, filter_rate=0.903, recall=0.75
ROC-AUC: 0.912
Look "Recall @ 0.1" Wooot! Amir (talk) 22:37, 23 February 2016 (UTC)
Size of the model:
(3.4)ladsgroup@ores-compute:~/editquality/models$ ls -Ssh | grep plwiki.reverted.rf.model 17M plwiki.reverted.rf.model
OK. We are good to go! Amir (talk) 22:51, 23 February 2016 (UTC)