Jump to content

Research:Wikipedia Knowledge Integrity Risk Observatory/List of Academic Papers

From Meta, a Wikimedia project coordination wiki

Given well-known issues affecting knowledge integrity in Wikipedia, including both forms of low-quality content and ways in which editor accounts violate community norms, this table compiles scientific articles on these issues. Articles were collected from the Research Newsletter, the feed of Wikimedia Research Twitter account and Google Scholar (e.g., query for vandalism in Wikipedia is https://scholar.google.com/scholar?q=allintitle:+wikipedia+(vandalism+OR+vandal+OR+vandals)). The number of citations on Google Scholar (GS) corresponds to September 2022.

Year Category Subcategory Citations (GS) Paper
2011 Advertising Spam 25 West, A. G., Chang, J., Venkatasubramanian, K., Sokolsky, O., & Lee, I. (2011, September). Link spamming Wikipedia for profit. In Proceedings of the 8th Annual Collaboration, Electronic messaging, Anti-Abuse and Spam Conference (pp. 152-161).
2011 Advertising Spam 12 West, A. G., Agrawal, A., Baker, P., Exline, B., & Lee, I. (2011, October). Autonomous link spam detection in purely collaborative environments. In Proceedings of the 7th international symposium on wikis and open collaboration (pp. 91-100).
2013 Advertising Promotional tone 16 Bhosale, S., Vinicombe, H., & Mooney, R. (2013, October). Detecting promotional content in wikipedia. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (pp. 1851-1857).
2017 Advertising Spam 19 Green, T., & Spezzano, F. (2017, May). Spam users identification in wikipedia via editing behavior. In Eleventh International AAAI Conference on Web and Social Media.
2018 Advertising Conflict of interest 2 Orizu, U., & He, Y. (2018). Content-Based Conflict-of-Interest Detection on Wikipedia.
2020 Advertising Conflict of interest 5 Joshi, N., Spezzano, F., Green, M., & Hill, E. (2020, April). Detecting Undisclosed Paid Editing in Wikipedia. In Proceedings of The Web Conference 2020 (pp. 2899-2905).
2022 Advertising Promotional tone 0 De Kock, C., & Vlachos, A. (2022, May). Leveraging Wikipedia article evolution for promotional tone detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5601-5613).
2022 Advertising Promotional tone 0 Li, M., Zhou, H., Hou, J., Wang, P., & Gao, E. (2022). Is cross-linguistic advert flaw detection in Wikipedia feasible? A multilingual-BERT-based transfer learning approach. Knowledge-Based Systems, 252, 109330.
2011 Biases Sentence 8 Herzig, L., Nunes, A., & Snir, B. (2011, June). An annotation scheme for automated bias detection in wikipedia. In Proceedings of the 5th Linguistic Annotation Workshop (pp. 47-55).
2014 Biases Political 21 Rogers, R., & Sendijarevic, E. (2012). Neutral or national point of view? A comparison of Srebrenica articles across Wikipedia’s language versions. Proc. Wikipedia Academy.
2014 Biases Political 13 Bilic, P., & Bulian, L. (2014). Lost in translation: Contexts, computing, disputing on Wikipedia. iconference 2014 proceedings.
2018 Biases Sentence 40 Hube, C., & Fetahu, B. (2018, April). Detecting biased statements in wikipedia. In Companion proceedings of the the web conference 2018 (pp. 1779-1786).
2019 Biases Political 5 Umarova, K., & Mustafaraj, E. (2019, May). How partisanship and perceived political bias affect wikipedia entries of news sources. In Companion Proceedings of The 2019 World Wide Web Conference (pp. 1248-1253).
2019 Biases Sentence 10 Aleksandrova, D., Lareau, F., & Ménard, P. A. (2019, September). Multilingual sentence-level bias detection in Wikipedia. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019) (pp. 42-51).
2021 Biases Sentence 2 Bertsch, A., & Bethard, S. (2021, November). Detection of Puffery on the English Wikipedia. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021) (pp. 329-333).
2021 Biases Sentence 0 Madanagopal, K., & Caverlee, J. (2021, April). Towards Ongoing Detection of Linguistic Bias on Wikipedia. In Companion Proceedings of the Web Conference 2021 (pp. 629-631).
2022 Biases Political 0 Leshnick, A. (2022). Deletion discussions on Hebrew Wikipedia: Negotiating global and local ideologies. New Media & Society, 14614448211067836.
2022 Biases Sentence 0 Madanagopal, K., & Caverlee, J. (2022, April). Improving Linguistic Bias Detection in Wikipedia using Cross-Domain Adaptive Pre-Training. In Companion Proceedings of the Web Conference 2022 (pp. 1301-1309).
2022 Biases Sentence 0 Stankiewicz, W., Baraniak, K., & Sydow, M. (2021, December). Bias detection in Wikipedia articles. A study on Polish and English Datasets. In IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (pp. 589-594).
2015 False information Contradictory information 17 Weissman, S., Ayhan, S., Bradley, J., & Lin, J. (2015, June). Identifying duplicate and contradictory information in wikipedia. In Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries (pp. 57-60).
2016 False information Hoaxes 305 Kumar, S., West, R., & Leskovec, J. (2016, April). Disinformation on the web: Impact, characteristics, and detection of wikipedia hoaxes. In Proceedings of the 25th international conference on World Wide Web (pp. 591-602).
2021 False information Contradictory information 3 Hsu, C., Li, C. T., Saez-Trumper, D., & Hsu, Y. Z. (2021, December). WikiContradiction: Detecting Self-Contradiction Articles on Wikipedia. In 2021 IEEE International Conference on Big Data (Big Data) (pp. 427-436). IEEE.
2022 False information Hoaxes 0 Elebiary, A., & Ciampaglia, G. L. The role of online attention in the supply of disinformation in Wikipedia.
2014 Non-verifiability 14 Nicholson, J. M., Uppala, A., Sieber, M., Grabitz, P., Mordaunt, M., & Rife, S. C. (2021). Measuring the quality of scientific references in Wikipedia: an analysis of more than 115M citations to over 800 000 scientific articles. The FEBS Journal, 288(14), 4242-4248.
2014 Non-verifiability 9 Viseur, R. (2014, August). Reliability of user-generated data: The case of biographical data in Wikipedia. In Proceedings of The International Symposium on Open Collaboration (pp. 1-3).
2017 Non-verifiability 7 Harder, R. H., Velasco, A., Evans, M., An, C., & Rockmore, D. (2017, April). Wikipedia verification check: A chrome browser extension. In Proceedings of the 26th International Conference on World Wide Web Companion (pp. 1619-1625).
2018 Non-verifiability 12 Ferretti, E., Cagnina, L., Paiz, V., Delle Donne, S., Zacagnini, R., & Errecalde, M. (2018). Quality flaw prediction in Spanish Wikipedia: A case of study with verifiability flaws. Information Processing & Management, 54(6), 1169-1181.
2019 Non-verifiability 22 Redi, M., Fetahu, B., Morgan, J., & Taraborelli, D. (2019, May). Citation needed: A taxonomy and algorithmic assessment of Wikipedia's verifiability. In The World Wide Web Conference (pp. 1567-1578).
2019 Non-verifiability 3 Bazán Pereyra, G., Cuello, C., Capodici, G., Jofré, V., Ferretti, E., & Errecalde, M. L. (2019). Automatically assessing the need of additional citations for information quality verification in Wikipedia articles. In XXV Congreso Argentino de Ciencias de la Computación (CACIC 2019, Universidad Nacional de Río Cuarto)..
2019 Non-verifiability 0 Murgu, C., & Ivings, K. (2019). “Blind Trust is Not Enough”: Considering Practical Verifiability and Open Referencing in Wikipedia. Journal of Critical Library and Information Studies, 2(2).
2020 Non-verifiability 13 Lewoniewski, W., Węcel, K., & Abramowicz, W. (2020). Modeling popularity and reliability of sources in multilingual Wikipedia. Information, 11(5), 263.
2020 Non-verifiability 2 Chou, A. J., Gonçalves, G., Walton, S., & Redi, M. (2020). Citation Detective: a Public Dataset to Improve and Quantify Wikipedia Citation Quality at Scale. Wiki-Workshop.
2021 Non-verifiability 6 Benjakob, O., Aviram, R., & Sobel, J. (2021). Citation needed? Wikipedia bibliometrics during the first wave of the COVID-19 pandemic. BioRxiv.
2022 Non-verifiability 0 Petroni, F., Broscheit, S., Piktus, A., Lewis, P., Izacard, G., Hosseini, L., ... & Riedel, S. (2022). Improving Wikipedia Verifiability with AI. arXiv preprint arXiv:2207.06220.
2022 Non-verifiability 0 Lewoniewski, W., Węcel, K., & Abramowicz, W. (2022). Reliability in Time: Evaluating the Web Sources of Information on COVID-19 in Wikipedia across Various Language Editions from the Beginning of the Pandemic. arXiv preprint arXiv:2204.14130.
2013 Sockpuppetry 90 Solorio, T., Hasan, R., & Mizan, M. (2013, June). A case study of sockpuppet detection in wikipedia. In Proceedings of the Workshop on Language Analysis in Social Media (pp. 59-68).
2013 Sockpuppetry 33 Solorio, T., Hasan, R., & Mizan, M. (2013). Sockpuppet detection in wikipedia: A corpus of real-world deceptive writing for linking identities. arXiv preprint arXiv:1310.6772.
2014 Sockpuppetry 113 Tsikerdekis, M., & Zeadally, S. (2014). Multiple account identity deception detection in social media using nonverbal behavior. IEEE Transactions on Information Forensics and Security, 9(8), 1311-1321.
2017 Sockpuppetry 148 Kumar, S., Cheng, J., Leskovec, J., & Subrahmanian, V. S. (2017, April). An army of me: Sockpuppets in online discussion communities. In Proceedings of the 26th International Conference on World Wide Web (pp. 857-866).
2022 Sockpuppetry 1 Niverthi, M., Verma, G., & Kumar, S. (2022, April). Characterizing, Detecting, and Predicting Online Ban Evasion. In Proceedings of the ACM Web Conference 2022 (pp. 2614-2623).
2008 Vandalism Detection 226 Potthast, M., Stein, B., & Gerling, R. (2008, March). Automatic vandalism detection in Wikipedia. In European conference on information retrieval (pp. 663-668). Springer, Berlin, Heidelberg.
2008 Vandalism Detection 121 Smets, K., Goethals, B., & Verdonk, B. (2008, July). Automatic vandalism detection in Wikipedia: Towards a machine learning approach. In AAAI workshop on Wikipedia and artificial intelligence: An Evolving Synergy (pp. 43-48). Antwerp, Belgium.: AAAI Press.
2009 Vandalism Detection 39 Itakura, K. Y., & Clarke, C. L. (2009, July). Using dynamic markov compression to detect vandalism in the wikipedia. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (pp. 822-823).
2010 Vandalism Characterization 409 Shachaf, P., & Hara, N. (2010). Beyond vandalism: Wikipedia trolls. Journal of Information Science, 36(3), 357-370.
2010 Vandalism Characterization 256 Geiger, R. S., & Ribes, D. (2010, February). The work of sustaining order in Wikipedia: The banning of a vandal. In Proceedings of the 2010 ACM conference on Computer supported cooperative work (pp. 117-126).
2010 Vandalism Characterization 83 Potthast, M. (2010, July). Crowdsourcing a Wikipedia vandalism corpus. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval (pp. 789-790).
2010 Vandalism Characterization 1 White, J., & Maessen, R. (2010). ZOT! to Wikipedia Vandalism. In CLEF (Notebook Papers/Labs/Workshops).(cited on page 68).
2010 Vandalism Detection 110 West, A. G., Kannan, S., & Lee, I. (2010, April). Detecting wikipedia vandalism via spatio-temporal analysis of revision metadata?. In Proceedings of the Third European Workshop on System Security (pp. 22-28).
2010 Vandalism Detection 89 Chin, S. C., Street, W. N., Srinivasan, P., & Eichmann, D. (2010, April). Detecting Wikipedia vandalism with active learning and statistical language models. In Proceedings of the 4th workshop on Information credibility (pp. 3-10).
2010 Vandalism Detection 81 Adler, B., De Alfaro, L., & Pye, I. (2010). Detecting wikipedia vandalism using wikitrust. Notebook papers of CLEF, 1, 22-23.
2010 Vandalism Detection 61 McKeown, K., & Wang, W. (2010). “Got You!”: Automatic vandalism detection in wikipedia with web-based shallow syntactic-semantic modeling.
2010 Vandalism Detection 30 Wu, Q., Irani, D., Pu, C., & Ramaswamy, L. (2010, October). Elusive vandalism detection in Wikipedia: a text stability-based approach. In Proceedings of the 19th ACM international conference on Information and knowledge management (pp. 1797-1800).
2010 Vandalism Detection 22 West, A. G., Kannan, S., & Lee, I. (2010, July). Stiki: an anti-vandalism tool for wikipedia using spatio-temporal analysis of revision metadata. In Proceedings of the 6th International Symposium on Wikis and Open Collaboration (pp. 1-2).
2010 Vandalism Detection 5 Harpalani, M. (2010). Wiki vandalysis—Wikipedia vandalism analysis (Doctoral dissertation, State University of New York at Stony Brook).
2010 Vandalism Detection 4 Velasco, S. M. M. (2010). Wikipedia Vandalism Detection Through Machine Learning: Feature Review and New Proposals∗ Lab Report for PAN at CLEF 2010.
2010 Vandalism Detection 3 West, A. G., Kannan, S., & Lee, I. (2010, July). Spatio-temporal analysis of Wikipedia metadata and the STiki anti-vandalism tool. In Proceedings of the 6th International Symposium on Wikis and Open Collaboration (pp. 1-2).
2010 Vandalism Detection 3 Hegedűs, I., Ormándi, R., Farkas, R., & Jelasity, M. (2010). Novel Balanced Feature Representation for Wikipedia Vandalism Detection Task⋆ Lab Report for PAN at CLEF 2010.
2011 Vandalism Characterization 1 Chin, S. C., & Street, W. N. (2011, July). Enriching Wikipedia Vandalism Taxonomy via Subclass Discovery. In LDH (pp. 19-24).
2011 Vandalism Characterization 1 Roessing, T. (2011). VANDALISM AND CONFLICT RESOLUTION IN WIKIPEDIA. AN EMPIRICAL ANALYSIS ON HOW A LARGE-SCALE WEB-BASED COMMUNITY DEALS WITH BREACHES OF THE ONLINE PEACE.
2011 Vandalism Detection 199 Adler, B. T., Alfaro, L. D., Mola-Velasco, S. M., Rosso, P., & West, A. G. (2011, February). Wikipedia vandalism detection: Combining natural language, metadata, and reputation features. In International Conference on Intelligent Text Processing and Computational Linguistics (pp. 277-288). Springer, Berlin, Heidelberg.
2011 Vandalism Detection 58 Javanmardi, S., McDonald, D. W., & Lopes, C. V. (2011, October). Vandalism detection in Wikipedia: a high-performing, feature-rich model and its reduction through Lasso. In Proceedings of the 7th International Symposium on Wikis and Open Collaboration (pp. 82-90).
2011 Vandalism Detection 47 Harpalani, M., Hart, M., Singh, S., Johnson, R., & Choi, Y. (2011, June). Language of vandalism: Improving wikipedia vandalism detection via stylometric analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 83-88).
2011 Vandalism Detection 33 Mola-Velasco, S. M. (2011, March). Wikipedia vandalism detection. In Proceedings of the 20th international conference companion on World wide web (pp. 391-396).
2011 Vandalism Detection 1 Aksit, F. G. (2011). An Empirical Research:“Wikipedia Vandalism Detection using VandalSense 2.0”: Notebook for PAN at CLEF 2011. In In Notebook Papers of CLEF 2011 LABs and Workshops.
2012 Vandalism Detection 41 Mola-Velasco, S. M. (2012). Wikipedia vandalism detection through machine learning: Feature review and new proposals: Lab report for pan at clef 2010. arXiv preprint arXiv:1210.5560.
2012 Vandalism Detection 5 Chin, S. C., & Street, W. N. (2012, June). Divide and transfer: an exploration of segmented transfer to detect wikipedia vandalism. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning (pp. 133-144). JMLR Workshop and Conference Proceedings.
2012 Vandalism Detection 5 Sumbana, M., Gonçalves, M. A., Silva, R., Almeida, J., & Veloso, A. (2012, September). Automatic vandalism detection in wikipedia with active associative classification. In International Conference on Theory and Practice of Digital Libraries (pp. 138-143). Springer, Berlin, Heidelberg.
2012 Vandalism Detection 1 Tang, X., Zhou, G., Fu, Y., Gan, L., Yu, W., & Li, S. (2012, November). Detecting wikipedia vandalism with a contributing efficiency-based approach. In International Conference on Web Information Systems Engineering (pp. 645-651). Springer, Berlin, Heidelberg.
2012 Vandalism Detection 0 Chang, T., Lin, H., & Lin, Y. (2012, December). Feature Transformation Method Enhanced Vandalism Detection in Wikipedia. In Asia Information Retrieval Symposium (pp. 187-198). Springer, Berlin, Heidelberg.
2013 Vandalism Detection 19 Tran, K. N., & Christen, P. (2013, April). Cross language prediction of vandalism on Wikipedia using article views and revisions. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 268-279). Springer, Berlin, Heidelberg.
2013 Vandalism Detection 5 Ramaswamy, L., Tummalapenta, R. S., Li, K., & Pu, C. (2013, October). A content-context-centric approach for detecting vandalism in Wikipedia. In 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing (pp. 115-122). IEEE.
2013 Vandalism Detection 2 Javanmardi, S., McDonald, D. W., Caruana, R., Forouzan, S., & Lopes, C. V. (2013). Learning to detect vandalism in social content systems: A Study on Wikipedia. In Mining Social Networks and Security Informatics (pp. 203-225). Springer, Dordrecht.
2013 Vandalism Detection 0 Lopes, C. V. (2013). Learning to Detect Vandalism in Social Content Systems: A Study on Wikipedia. Mining Social Networks and Security Informatics, 203.
2013 Vandalism Detection 0 Cioiu, D., & Rebedea, T. (2013, September). WikiDetect: Automatic Vandalism Detection for Wikipedia Using Linguistic Features. In International Conference on Computational Collective Intelligence (pp. 316-325). Springer, Berlin, Heidelberg.
2014 Vandalism Detection 14 Tran, K. N., & Christen, P. (2014). Cross-language learning from bots and users to detect vandalism on wikipedia. IEEE Transactions on Knowledge and Data Engineering, 27(3), 673-685.
2014 Vandalism Detection 0 Ramaswamy, L., Tummalapenta, R. S., Sethi, D., Li, K., & Pu, C. (2014). Harnessing Context for Vandalism Detection in Wikipedia. EAI Endorsed Transactions on Collaborative Computing, 1(1), e7.
2015 Vandalism Detection 87 Kumar, S., Spezzano, F., & Subrahmanian, V. S. (2015, August). Vews: A wikipedia vandal early warning system. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 607-616).
2015 Vandalism Detection 9 Tran, K. N., Christen, P., Sanner, S., & Xie, L. (2015, May). Context-aware detection of sneaky vandalism on wikipedia across multiple languages. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 380-391). Springer, Cham.
2015 Vandalism Detection 2 Spina, M., Rossi, D., Sozio, M., Maniu, S., & Cautis, B. (2015, June). Snooping Wikipedia vandals with mapreduce. In 2015 IEEE International Conference on Communications (ICC) (pp. 1146-1151). IEEE.
2016 Vandalism Characterization 8 de Laat, P. B. (2016). Profiling vandalism in Wikipedia: A Schauerian approach to justification. Ethics and Information Technology, 18(2), 131-148.
2016 Vandalism Detection 16 Tramullas, J., Garrido-Picazo, P., & Sánchez-Casabón, A. I. (2016, June). Research on Wikipedia Vandalism: a brief literature review. In Proceedings of the 4th Spanish Conference on Information Retrieval (pp. 1-4).
2016 Vandalism Detection 8 Susuri, A., Hamiti, M., & Dika, A. (2016, June). Machine learning based detection of vandalism in wikipedia across languages. In 2016 5th Mediterranean Conference on Embedded Computing (MECO) (pp. 446-451). IEEE.
2016 Vandalism Detection 7 Suyehira, K., & Spezzano, F. (2016, October). Depp: a system for detecting pages to protect in wikipedia. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (pp. 2081-2084).
2016 Vandalism Detection 1 Shulhan, M., & Widyantoro, D. H. (2016, August). Detecting vandalism on English Wikipedia using LNSMOTE resampling and Cascaded Random Forest classifier. In 2016 International Conference On Advanced Informatics: Concepts, Theory And Application (ICAICTA) (pp. 1-6). IEEE.
2017 Vandalism Detection 13 Yuan, S., Zheng, P., Wu, X., & Xiang, Y. (2017, September). Wikipedia vandal early detection: from user behavior to user embedding. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 832-846). Springer, Cham.
2018 Vandalism Characterization 4 Sierra, Á. O., & Castanedo, J. O. (2018, August). University students in the educational field and Wikipedia vandalism. In Proceedings of the 14th International Symposium on Open Collaboration (pp. 1-7).
2018 Vandalism Characterization 3 Alkharashi, A., & Jose, J. (2018, June). Vandalism on Collaborative Web Communities: An Exploration of Editorial Behaviour in Wikipedia. In Proceedings of the 5th Spanish Conference on Information Retrieval (pp. 1-4).
2019 Vandalism Characterization 2 Damas, C. A., & Mochetti, K. (2019, February). An analysis of homophobia on vandalism at wikipedia. In 2019 Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT) (pp. 1-2). IEEE.
2019 Vandalism Characterization 0 Subramanian, S. S., Pushparaj, P., Liu, Z., & Lu, A. (2019, October). Explainable Visualization of Collaborative Vandal Behaviors in Wikipedia. In 2019 IEEE Symposium on Visualization for Cyber Security (VizSec) (pp. 1-5). IEEE.
2019 Vandalism Characterization 0 Bolsarin, R., & Pinheiro, P. (2019). The practice of vandalism in the collaborative writing process of the Wikipedia entry “Simone de Beauvoir”. Educação & Sociedade, 40.
2019 Vandalism Detection 102 Zheng, P., Yuan, S., Wu, X., Li, J., & Lu, A. (2019, July). One-class adversarial nets for fraud detection. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 1286-1293).
2019 Vandalism Detection 14 Martinez-Rico, J. R., Martinez-Romo, J., & Araujo, L. (2019). Can deep learning techniques improve classification performance of vandalism detection in Wikipedia?. Engineering Applications of Artificial Intelligence, 78, 248-259.
2019 Vandalism Detection 5 Spezzano, F., Suyehira, K., & Gundala, L. A. (2019). Detecting pages to protect in Wikipedia across multiple languages. Social Network Analysis and Mining, 9(1), 1-16.
2019 Vandalism Detection 1 Liu, Z., & Lu, A. (2019, December). Explainable Visualization for Interactive Exploration of CNN on Wikipedia Vandal Detection. In 2019 IEEE International Conference on Big Data (Big Data) (pp. 2354-2363). IEEE.
2019 Vandalism Detection 0 Susuri, A., Hamiti, M., & Maraj, A. (2019). BOTS AND USERS IMPACT IN THE DETECTION OF VANDALISM IN LOW-RESOURCE LANGUAGE SECTIONS OF WIKIPEDIA. ISCBE 2019, 550.