Data Collection

...

Revision Selection

Subsequent revisions by the same user are considered as an individual revision (independently of the time between them)

Metrics

URL-wise metrics

Metrics evaluated on each combination of page & source found in the dataset. Each metric is evaluated on events of addition and removal of a source. We are considering "removal" an edit that removes the source, with the source not being added again before one day. If the source is removed and added again within one day, the inbetween revisions are ignored

Age/Lifetime

lifetime

A measure of time (days) from a source addition to its removal.

lifetime_nrev

A measure of number of revisions from a source addition to its removal

age

days between the first time that url was added on that page, and collection time

current

A value in ${0,1}$ indicating wether the source is present (1) or not (0) in the page, at the time of the data collection

age_nrev

number of edits received by the page, between the time when the url was first added up to collection time

Permanence

permanence

The measure of how much time (days) a source has been on a page. It is the sum of lifetimes

norm_permanence

The permanence (days) of a url on a page, divided by the lifetime (days) of the page itself. It is a measure ranging in $[0,1]$ of how much time a source has been on a page, proportionally to the life of the page. Considering $t_{c}$ the collection time, $t_{0}^{p},t_{0}^{u}$ the times when the page was created and the url was added, $U_{\tau }^{p}=1,0$ whether the url was present or not on page $p$ at time $\tau$

$NP_{t_{c}}^{u,p}={\frac {\int _{t_{0}^{u}}^{t_{c}}U_{\tau }^{p}d\tau }{t_{c}-t_{0}^{p}}}={\frac {Permanence}{Age(page)}}$

selfnorm_permanence

The permanence (days) of a url on a page, divided by the age of that url on that page. It is a measure ranging in $[0,1]$ of how much time a source has been on a page, proportionally to the life of the url. Considering $t_{c}$ the collection time, $t_{0}^{u}$ the times when the url was added, $U_{\tau }^{p}=1,0$ whether the url was present or not on page $p$ at time $\tau$

$SP_{t_{c}}^{u,p}={\frac {\int _{t_{0}^{u}}^{t_{c}}U_{\tau }^{p}d\tau }{t_{c}-t_{0}^{u}}}={\frac {Permanence}{Age(url)}}$

permanence_nrev

The measure of how many revisions a source has been on a page. It is the sum of lifetimes_nrev

norm_permanence_nrev

The permanence_nrev (number of revisions) of a url on a page, divided by the lifetime_nrev (number of revisions) of the page itself. It is a measure ranging in $[0,1]$ of how much time a source has been on a page, proportionally to the life of the page

selfnorm_permanence_nrev

The permanence (number of revsions) of a url on a page, divided by the number of revisions since the url was first added. It is a measure ranging in $[0,1]$ of how many revisions a source was involved in, since the first addition on it

Involvement/Edits

inv_count

Involvement score. It is a measure relative to a page and url, of how many times a sentence using that url as a source has been edited. If a URL is present in our dataset, but is not involved in such revisions, its inv_count will be 0 (might change in the future)

inv_count_normalized

Involvement score "locally normalized". For each revision, the number of times a source has been edited is divided by the number of sources that have been involved in that revision

inv_score_globnorm

Involvement score of a url on a page, divided by its permanence (days)

inv_score_globnorm_nrev

Involvement score of a url on a page, divided by its permanence_nrev (number of edits)

${\text{involvement score (nrev)}}={\frac {{\text{N context edit}}_{(url,page)}}{{\text{N edit }}_{(url,page)}}}={\frac {\text{inv count}}{\text{age nrev}}}$

edit count

inv_count + number of lifes of a url on a page. This measure account for both the edits received by [sentences containing] a source, and the number of times it was added

edit_score_globnorm

edit count, divided by the number of days the source has been on the page

edit_score_globnorm_nrev

edit count, divided by the age_nrev (number of revisions) of the url on that page

${\text{Edit score (nrev)}}={\frac {{\text{N context edit}}_{(url,page)}+{\text{N lifes}}_{(url,page)}}{{\text{N edit }}_{(url,page)}}}={\frac {{\text{inv count}}+{\text{n lifes}}}{\text{age nrev}}}$

Users/Editors_add/rem/start/end

Editors_start/end

number editors who start/end a lifetime

Editors_add/rem

number editors who add/remove that url

Registerededitors_add/rem

number registered users who add/remove that url

Registerededitors_start/end

number registered users who start/end a lifetime

norm_Editors_start/end

measure the variety of editors who start/end the lifetime of a url:

${\frac {\text{number unique editors who start/end a lifetime}}{\text{number lifetimes starts/ends}}}$

norm_Editors_add/rem

measure the variety of editors who add/remove that url:

${\frac {\text{number unique editors who add/remove that url}}{\text{number of addition/removal}}}$

norm_Registerededitors_start/end

probability that the lifetime of that url is started/ended by a registered user:

${\frac {\text{n times registered users started/ended a lifetime}}{\text{number lifetimes starts/end}}}$

norm_Registerededitors_add/rem

probability that url is added/removed a registered user:

${\frac {\text{n times registered users added/removed that url}}{\text{number of addition/removal}}}$

Domain-wise metrics

{current}_{url-wise metric}_{mean/median/sum}_urlwise

For each url-wise metric there is a domain aggregated version of it, obtained by aggregatig each page&url metric over domain and evaluating the mean/median/sum. "Current" version measure the metrics on only the urls that are used at collection time

{current}_{url-wise metric}_{mean/median/sum}_pagewise

For each url-wise metric there is a domain-wise version of it, obtained in the same way but considering the pair of page&domain during computation (using the domain of a url instead of the url itself). The values are then aggregated for each domain as the avg/median/sum over the pages. "Current" version measure the metrics on only the pages that are used at collection time

n_life

Number of lifetimes of urls from a domain on all the pages

{current}_n_page

number of pages where the domain is used as a source. Current version count the number of page where it was used at collection time

{current}_n_pageurls

number of unique page&url combinations, for urls from that domain. Current version count the number of unique page&urls combination where it was used at collection time

normalized extensive features

${\text{norm n pages}}={\frac {\text{number of pages that domain has been used}}{\text{total pages in the dataset}}}$

${\text{norm permanence sum}}={\frac {\text{sum of permanence in all pages}}{\text{sum of pages' ages}}}$

${\text{norm editors (add/rem/start/end)}}={\frac {\text{number of editors adding, removing, ...}}{\text{total number of editors in the dataset}}}$

Modelling approaches

We are testing (18/10/23) different models, using different approaches with regards to data representation, data selection, preprocessing and learning algorithms

Target

We aim to predict reliability/controversiality of url/domains using the metrics described above. To do so, we use a discriminative approach using a ground truth provided by perennial status and/or mbfc status

Perennial status

Perennial sources are a classification of several domains in Gen. Reliable, Gen. Unreliable, Mixed, No consensus, Blacklisted and Deprecated. For a clean target we use the Gen. Reliable class as a positive class, and Gen. Unreliable class as a negative class

MBFC Status

We are not using (18/10/23) mbfc statuses as a target, but we can train on perennial status as target and then validate (qualitatively) on mbfc status

Data Representation

We train over the domains which are classified as Generally reliable (as a positive target), or Generally unreliable (as a negative target) in the Perennial Sources

Domain Dataset

In the domain, the rows of the dataset represent information we gathered about each domain

URL+Domain Dataset

In this dataset, each row represent information about the page&url combination AND information about its domain (same of Domain Dataset); we thus train over the information we gathered about the usage of a url in a given page, in addition to the information of its domain. This makes so that part of the values in the rows (the ones of Domain information) are repeated across different urls with the same domain.

Data selection

Subsampling

In the URL+Domain Dataset, we perform subsampling of urls from highly used domains. This subsampling is performed applying a maximum number of url from same domain cutoff, and with this strategy we obtain a more balanced url+domain dataset. It is required to avoid the dataset/fold (during crossvalidation) to be dominated by few popular domains.

Balancing

In order to have a balanced target for the training, we subsample the url/domain obtaining the same amount of positive (Generally Reliable) and negative (Generally Unreliable) entries.

Preprocessing

Normalization

If required (ex. for logistic regression model) before using the features in the model, each feature is normalized indipendently by ranking. Each feature is thus represented in a uniform interval from 0,1

VIF

If required, we check for VIF values across all the features and iteratively remove the highest VIF ones, up to the point where each feature remaining has a VIF value lower than a threshold (generally VIF<5)

Discriminative modelling Modelling

LogReg

Does require Normalization and (VIF based) feature selection

XGBoost

Does not require Normalization or feature selection - both performed by the algorithm itself

Performance Evaluation Approaches

Iterations

We tested several strategies, changing:

balanced/not balanced dataset
Domain data/URL+Domain data
data from project Climate change/ COVID-19
data from english pages/other languages alltogether/some specific language
using scores features/not using them
weighted/not weighted target

Evaluation Strategies

Early strategies (december 2023)

In-dataset validation

To assess the quality of model prediction over unobserved domains, we cross-validated the model within each dataset. To do so, we train on 4/5 of the domains and validate on the remaining 1/5, repeating the process for 100 times. Since there are only few domains, we used this strategy to avoid fold-dependent noise in performance metrics. Using the 100 folds results we measure F1, precision, recall averages and standard deviations, on either positive entries (Gen. Reliable) and negative entries (Gen. Unreliable)

Cross-dataset validation

To check whether the results are dataset dependent, we trained and validated the model across different datasets (different project and language)

Current strategy

Tools

F1 Macro: Since we are interested in recognizing either negative and positive domains (consensus-reliable or consensus-unreliable) we are measuring model performances using the unweighted averages of F1 score for both the classes
Leave-one-out: When training and testing on the same dataset, we are using leave-one-out validation to measure F1 scores. This also happends when training on a dataset that includes the test domains (es training on all domains from all languages of Climate Change pages, and testing on Climate Change Spanish pages)
Bootstrapping: To obtain a confidence interval on each dataset measure, we take the list of results and resample them 100 times with replacement; we then compute performance metrics once for each iteration. The result will be a distribution of 100 values representing the model performance on a given dataset
Mann-Whitney: To compare performances from two different models on the same dataset, we use Mann-Whitney P values corrected with Bonferroni Approach. This means that for each pair of models we have a p value telling us the probability of them being comparable (p>0.05) or statistically significant different (p<0.05)

Model Performances

Random Model For each dataset, meaning a combination of set of pages and language, we perform a measure for a random model predicting reliability of the domains appearing in that page. This is done by assigning random binary values to each domain and bootstrapping the results

Results

Dataset statistics

URL+Domain

deprecated

Domain

Iteration of datasets (by project and language): support sizes for positive entries (generally reliable) and negative ones (generally unreliable).

Support sizes for different datasets
project	lang	pos size	neg size
COVID-19	de	96	52
COVID-19	en	136	125
COVID-19	other	133	134
COVID-19	ru	93	34
Climate change	de	100	52
Climate change	en	130	152
Climate change	other	138	145
Climate change	ru	87	43

Model performances

Model selection

Observations

Overfitting is more present with Logistic regressor (deprecated) even though it can be prevented by VIF Selection. Testing on Climate Change and english shows that overfitting is still present ( 0.9 on training vs 0.8 on validation) although the performances are still good on validation set. Using Leave-one-out validation moving forward
Weighted model provide best performances with no further assumption, and without having to remove domains from the datasets
Scores provide no additional information to the model, while being difficult to compute on languages different than english. They are dropped out of the features moving forward
URLS info: if training is performed at the url-level, F1 scores and the model is strongly biased by the size of some domains (up to thousands urls). Moving forward we will use domain-wise information only
XGBoost: provide better F1 performances than Logistic regressor, without assumptions to make on features selection
Other languages: aggregation of all other languages together bias the resulting dataset in favor of more active wikis (es, de, fr,... ), so aggregation could be not the best strategy for different languages. Also, we expect the model to perform worse on other languages due to
- Less importance to Perennial
- Fewer revisions
- Noise from low resource languages

Early results (december 2023): URL+Domain dataset

Performance metrics are obtained following strategy described in "In-dataset validation" (above).

Over the URL+Domain dataset, we tested the XGBoost model changing:

Climate change/Covid
english/other languages
weighting/not weighting,
using/not using the scores based features

XGBoost: unbalanced URL+Domain dataset (11/10/23)
iteration_name	project	language	use_scores	weighted	train_f1_avg	train_f1_std	valid_f1_avg	valid_f1_std
covid-19_english	COVID-19	english	False	False	0.995	0.001	0.951	0.042
covid-19_english_weighted	COVID-19	english	False	True	0.995	0.002	0.951	0.038
covid-19_english_+scores	COVID-19	english	True	False	0.997	0.001	0.947	0.037
covid-19_english_+scores_weighted	COVID-19	english	True	True	0.995	0.002	0.945	0.036
covid-19_other_languages	COVID-19	other_languages	False	False	0.980	0.007	0.859	0.058
covid-19_other_languages_weighted	COVID-19	other_languages	False	True	0.980	0.004	0.844	0.065
climate_change_english	Climate change	english	False	False	0.993	0.002	0.936	0.051
climate_change_english_weighted	Climate change	english	False	True	0.988	0.002	0.939	0.049
climate_change_english_+scores	Climate change	english	True	False	0.992	0.002	0.941	0.039
climate_change_english_+scores_weighted	Climate change	english	True	True	0.986	0.003	0.936	0.035
climate_change_other_languages	Climate change	other_languages	False	False	0.968	0.007	0.876	0.056
climate_change_other_languages_weighted	Climate change	other_languages	False	True	0.966	0.007	0.842	0.062

Early results (december 2023): Domain dataset

Performance metrics are obtained following strategy described in "In-dataset validation" (above). Over the Domain dataset, we tested the XGBoost model changing:

Climate change/Covid
english/all other languages/de/ru
weighting/not weighting/balancing dataset,
using/not using the scores based features

XGBoost: Domain dataset (11/10/23)
iteration_name	project	lang	use scores	balanced dataset	weighted	pos f1	neg f1	pos precision	neg precision	pos recall	neg recall
covid-19_other_languages_de	COVID-19	de	False	False	False	0.811	0.583	0.774	0.684	0.862	0.538
covid-19_other_languages_de_weighted	COVID-19	de	False	False	True	0.786	0.630	0.812	0.614	0.771	0.672
covid-19_other_languages_balanced_de	COVID-19	de	False	True	False	0.668	0.668	0.672	0.692	0.690	0.673
covid-19_english	COVID-19	en	False	False	False	0.847	0.827	0.848	0.832	0.851	0.827
covid-19_english_weighted	COVID-19	en	False	False	True	0.847	0.828	0.852	0.829	0.847	0.833
covid-19_english_balanced	COVID-19	en	False	True	False	0.849	0.842	0.848	0.848	0.855	0.842
covid-19_english_+scores	COVID-19	en	True	False	False	0.858	0.836	0.854	0.845	0.868	0.833
covid-19_english_+scores_weighted	COVID-19	en	True	False	True	0.860	0.838	0.858	0.844	0.866	0.838
covid-19_english_+scores_balanced	COVID-19	en	True	True	False	0.838	0.837	0.843	0.839	0.840	0.840
covid-19_other_languages	COVID-19	other	False	False	False	0.774	0.779	0.792	0.769	0.764	0.796
covid-19_other_languages_weighted	COVID-19	other	False	False	True	0.778	0.781	0.794	0.772	0.768	0.795
covid-19_other_languages_balanced	COVID-19	other	False	True	False	0.776	0.772	0.793	0.763	0.766	0.789
covid-19_other_languages_ru	COVID-19	ru	False	False	False	0.826	0.364	0.777	0.527	0.890	0.311
covid-19_other_languages_ru_weighted	COVID-19	ru	False	False	True	0.799	0.522	0.839	0.498	0.774	0.596
covid-19_other_languages_balanced_ru	COVID-19	ru	False	True	False	0.691	0.671	0.689	0.726	0.734	0.663
climate_change_other_languages_de	Climate change	de	False	False	False	0.786	0.548	0.764	0.610	0.820	0.530
climate_change_other_languages_de_weighted	Climate change	de	False	False	True	0.752	0.580	0.790	0.555	0.728	0.640
climate_change_other_languages_balanced_de	Climate change	de	False	True	False	0.637	0.654	0.649	0.667	0.649	0.663
climate_change_english	Climate change	en	False	False	False	0.781	0.818	0.792	0.814	0.778	0.828
climate_change_english_weighted	Climate change	en	False	False	True	0.783	0.814	0.781	0.821	0.792	0.813
climate_change_english_balanced	Climate change	en	False	True	False	0.784	0.778	0.792	0.781	0.786	0.786
climate_change_english_+scores	Climate change	en	True	False	False	0.788	0.816	0.800	0.812	0.783	0.826
climate_change_english_+scores_weighted	Climate change	en	True	False	True	0.790	0.810	0.785	0.821	0.801	0.804
climate_change_english_+scores_balanced	Climate change	en	True	True	False	0.779	0.781	0.777	0.792	0.790	0.779
climate_change_other_languages	Climate change	other	False	False	False	0.699	0.729	0.722	0.717	0.685	0.749
climate_change_other_languages_weighted	Climate change	other	False	False	True	0.700	0.723	0.714	0.719	0.695	0.733
climate_change_other_languages_balanced	Climate change	other	False	True	False	0.704	0.714	0.731	0.702	0.691	0.740
climate_change_other_languages_ru	Climate change	ru	False	False	False	0.777	0.397	0.730	0.513	0.840	0.350
climate_change_other_languages_ru_weighted	Climate change	ru	False	False	True	0.760	0.519	0.777	0.516	0.754	0.550
climate_change_other_languages_balanced_ru	Climate change	ru	False	True	False	0.664	0.644	0.677	0.671	0.684	0.660