Research:Language-Agnostic Topic Classification/Outlink model performance
Appearance
This page provides details about the current (as of December 2020) model for Wikipedia topic classification based on outlinks (as represented by their corresponding Wikidata IDs):
- Code: https://github.com/geohci/wikipedia-language-agnostic-topic-classification
- Model architecture: multi-label fastText supervised model
- Training data used a 90% sample of every language in Wikipedia. In practice, this meant that English Wikipedia provides 11.4% of the data and then Cebuano (8.8%), Swedish (6.4%), German (4.6%), French (4.2%), and all other languages are below 4%.
- Epochs: 2
- Learning rate: 0.1
- Window size: 20
- Min count (under which QID is not retained in vocab): 20
- No pre-trained embeddings used
- Embeddings dimension: 50
- Total number of model params: 3200 (50 x 64)
- Vocab size: 4,145,064
- Total number of embeddings params: 207,253,200 (vocab size * embeddings dimension)
- Model size on disk: 863 MB
Test Results
[edit]Regarding the test results below, there is a strong limitation in that there is no labeled data for articles that do not have English Wikipedia equivalents. It is very possible that the performance degrades for these articles (assuming they also have outlinks to different types of articles).
Overall results:
- Precision: 0.877 (micro); 0.836 (macro)
- Recall: 0.793 (micro); 0.678 (macro)
- F1: 0.830 (micro); 0.744 (macro)
- Average precision: 0.891 (micro); 0.795 (macro)
For qualitative evaluations (of a slightly older model), see task T266201. Overall, precision was 94.7% in Arabic; 81.4% in Czech, 89.5% in English, 88.0% in French, 91.3% in Vietnamese.
Topic | n | TP | FP | TN | FN | Precision | Recall | F1 | Ave. Pre. |
---|---|---|---|---|---|---|---|---|---|
europe | 783307 | 681220 | 90139 | 1544254 | 102087 | 0.883 | 0.870 | 0.876 | 0.952 |
biography | 677093 | 621664 | 49035 | 1691572 | 55429 | 0.927 | 0.918 | 0.922 | 0.971 |
stem | 480236 | 431709 | 24123 | 1913341 | 48527 | 0.947 | 0.899 | 0.922 | 0.973 |
asia | 360865 | 304097 | 35810 | 2021025 | 56768 | 0.895 | 0.843 | 0.868 | 0.933 |
sports | 334774 | 315905 | 10497 | 2072429 | 18869 | 0.968 | 0.944 | 0.956 | 0.980 |
media | 327692 | 281754 | 37092 | 2052916 | 45938 | 0.884 | 0.860 | 0.872 | 0.938 |
western-europe | 290047 | 243700 | 28115 | 2099538 | 46347 | 0.897 | 0.840 | 0.867 | 0.944 |
north-america | 282981 | 216595 | 40953 | 2093766 | 66386 | 0.841 | 0.765 | 0.801 | 0.895 |
biology | 247481 | 232241 | 6260 | 2163959 | 15240 | 0.974 | 0.938 | 0.956 | 0.985 |
geographical | 213947 | 145537 | 33160 | 2170593 | 68410 | 0.814 | 0.680 | 0.741 | 0.825 |
eastern-europe | 176961 | 149351 | 14603 | 2226136 | 27610 | 0.911 | 0.844 | 0.876 | 0.938 |
southern-europe | 172638 | 138080 | 19706 | 2225356 | 34558 | 0.875 | 0.800 | 0.836 | 0.913 |
northern-europe | 161724 | 118894 | 21354 | 2234622 | 42830 | 0.848 | 0.735 | 0.787 | 0.869 |
history | 142121 | 83506 | 20703 | 2254876 | 58615 | 0.801 | 0.588 | 0.678 | 0.757 |
music | 127211 | 110574 | 9764 | 2280725 | 16637 | 0.919 | 0.869 | 0.893 | 0.939 |
women | 113835 | 47597 | 20674 | 2283191 | 66238 | 0.697 | 0.418 | 0.523 | 0.612 |
films | 112002 | 90799 | 15972 | 2289726 | 21203 | 0.850 | 0.811 | 0.830 | 0.904 |
east-asia | 97601 | 80108 | 10181 | 2309918 | 17493 | 0.887 | 0.821 | 0.853 | 0.909 |
military-and-warfare | 103937 | 64034 | 15022 | 2298741 | 39903 | 0.810 | 0.616 | 0.700 | 0.774 |
politics-and-government | 95746 | 53169 | 15573 | 2306381 | 42577 | 0.773 | 0.555 | 0.646 | 0.724 |
west-asia | 91404 | 73965 | 8718 | 2317578 | 17439 | 0.895 | 0.809 | 0.850 | 0.914 |
philosophy-and-religion | 89395 | 51774 | 14468 | 2313837 | 37621 | 0.782 | 0.579 | 0.665 | 0.717 |
visual-arts | 86498 | 51830 | 14574 | 2316628 | 34668 | 0.781 | 0.599 | 0.678 | 0.744 |
transportation | 84279 | 71175 | 6045 | 2327376 | 13104 | 0.922 | 0.845 | 0.881 | 0.921 |
literature | 75739 | 41881 | 11840 | 2330121 | 33858 | 0.780 | 0.553 | 0.647 | 0.718 |
south-asia | 72071 | 60734 | 5569 | 2340060 | 11337 | 0.916 | 0.843 | 0.878 | 0.919 |
africa | 71004 | 49691 | 8820 | 2337876 | 21313 | 0.849 | 0.700 | 0.767 | 0.835 |
south-america | 61705 | 46888 | 7646 | 2348349 | 14817 | 0.860 | 0.760 | 0.807 | 0.877 |
north-asia | 63870 | 47955 | 7789 | 2346041 | 15915 | 0.860 | 0.751 | 0.802 | 0.871 |
oceania | 60421 | 46368 | 5013 | 2352266 | 14053 | 0.902 | 0.767 | 0.829 | 0.876 |
business-and-economics | 53360 | 25192 | 9361 | 2354979 | 28168 | 0.729 | 0.472 | 0.573 | 0.614 |
technology | 44378 | 25983 | 8303 | 2365019 | 18395 | 0.758 | 0.585 | 0.661 | 0.721 |
engineering | 45147 | 30979 | 4465 | 2368088 | 14168 | 0.874 | 0.686 | 0.769 | 0.820 |
architecture | 42920 | 22882 | 7432 | 2367348 | 20038 | 0.755 | 0.533 | 0.625 | 0.684 |
medicine-and-health | 41775 | 28278 | 4943 | 2370982 | 13497 | 0.851 | 0.677 | 0.754 | 0.815 |
earth-and-environment | 40405 | 26281 | 4867 | 2372428 | 14124 | 0.844 | 0.650 | 0.735 | 0.779 |
television | 38695 | 26503 | 5099 | 2373906 | 12192 | 0.839 | 0.685 | 0.754 | 0.806 |
society | 38816 | 10600 | 6410 | 2372474 | 28216 | 0.623 | 0.273 | 0.380 | 0.407 |
southeast-asia | 38135 | 27982 | 3772 | 2375793 | 10153 | 0.881 | 0.734 | 0.801 | 0.856 |
space | 36009 | 32368 | 1331 | 2380360 | 3641 | 0.961 | 0.899 | 0.929 | 0.960 |
linguistics | 32034 | 20520 | 2452 | 2383214 | 11514 | 0.893 | 0.641 | 0.746 | 0.772 |
computing | 25619 | 18487 | 3769 | 2388312 | 7132 | 0.831 | 0.722 | 0.772 | 0.837 |
central-america | 24075 | 14708 | 2659 | 2390966 | 9367 | 0.847 | 0.611 | 0.710 | 0.762 |
entertainment | 22751 | 9633 | 4024 | 2390925 | 13118 | 0.705 | 0.423 | 0.529 | 0.587 |
internet-culture | 23239 | 17083 | 2064 | 2392397 | 6156 | 0.892 | 0.735 | 0.806 | 0.872 |
education | 23323 | 5774 | 3312 | 2391065 | 17549 | 0.635 | 0.248 | 0.356 | 0.381 |
chemistry | 22115 | 16800 | 2614 | 2392971 | 5315 | 0.865 | 0.760 | 0.809 | 0.884 |
northern-africa | 20544 | 11847 | 3437 | 2393719 | 8697 | 0.775 | 0.577 | 0.661 | 0.706 |
food-and-drink | 19547 | 11808 | 2595 | 2395558 | 7739 | 0.820 | 0.604 | 0.696 | 0.731 |
performing-arts | 17512 | 8030 | 2796 | 2397392 | 9482 | 0.742 | 0.459 | 0.567 | 0.587 |
physics | 16666 | 9738 | 2722 | 2398312 | 6928 | 0.782 | 0.584 | 0.669 | 0.730 |
books | 17010 | 9479 | 2561 | 2398129 | 7531 | 0.787 | 0.557 | 0.653 | 0.686 |
video-games | 16301 | 14382 | 700 | 2400699 | 1919 | 0.954 | 0.882 | 0.917 | 0.947 |
mathematics | 15628 | 10965 | 2110 | 2399962 | 4663 | 0.839 | 0.702 | 0.764 | 0.820 |
eastern-africa | 16386 | 10657 | 1831 | 2399483 | 5729 | 0.853 | 0.650 | 0.738 | 0.793 |
comics-and-anime | 14774 | 11416 | 1248 | 2401678 | 3358 | 0.901 | 0.773 | 0.832 | 0.862 |
software | 14288 | 8377 | 3391 | 2400021 | 5911 | 0.712 | 0.586 | 0.643 | 0.692 |
western-africa | 14481 | 9804 | 1640 | 2401579 | 4677 | 0.857 | 0.677 | 0.756 | 0.816 |
southern-africa | 10012 | 6318 | 1018 | 2406670 | 3694 | 0.861 | 0.631 | 0.728 | 0.758 |
central-asia | 9549 | 5304 | 1602 | 2406549 | 4245 | 0.768 | 0.555 | 0.645 | 0.687 |
central-africa | 6698 | 3932 | 985 | 2410017 | 2766 | 0.800 | 0.587 | 0.677 | 0.722 |
fashion | 5789 | 2679 | 892 | 2411019 | 3110 | 0.750 | 0.463 | 0.572 | 0.579 |
radio | 4373 | 2349 | 522 | 2412805 | 2024 | 0.818 | 0.537 | 0.649 | 0.636 |
libraries-and-information | 3765 | 1449 | 537 | 2413398 | 2316 | 0.730 | 0.385 | 0.504 | 0.470 |