Md. Abdur Rahman; Abu Nayem; Mahfida Amjad; Md. Saeed Siddik

How do Machine Learning Algorithms Effectively Classify Toxic Comments? An Empirical Analysis

Full Text (PDF, 449KB), PP.1-14

Views: 0 Downloads: 0

Author(s)

Md. Abdur Rahman ^1,* Abu Nayem ² Mahfida Amjad ³ Md. Saeed Siddik ⁴

1. Centre for advance Research in Sciences (CARS), University of Dhaka, Bangladesh

2. Department of Computer Science, Stamford University Bangladesh, Bangladesh

3. Department of Computer Science and Engineering, Stamford University Bangladesh, Dhaka, Bangladesh

4. Institute of Information Technology, University of Dhaka, Bangladesh

* Corresponding author.

DOI: https://doi.org/10.5815/ijisa.2023.04.01

Received: 6 Feb. 2023 / Revised: 11 Apr. 2023 / Accepted: 3 May 2023 / Published: 8 Aug. 2023

Index Terms

Toxic Comment Analysis, Text Classification, BoW, TF-IDF, Hashing, CHI2, Machine Learning

Abstract

Toxic comments on social media platforms, news portals, and online forums are impolite, insulting, or unreasonable that usually make other users leave a conversation. Due to the significant number of comments, it is impractical to moderate them manually. Therefore, online service providers use the automatic detection of toxicity using Machine Learning (ML) algorithms. However, the model's toxicity identification performance relies on the best combination of classifier and feature extraction techniques. In this empirical study, we set up a comparison environment for toxic comment classification using 15 frequently used supervised ML classifiers with the four most prominent feature extraction schemes. We considered the publicly available Jigsaw dataset on toxic comments written by human users. We tested, analyzed and compared with every pair of investigated classifiers and finally reported a conclusion. We used the accuracy and area under the ROC curve as the evaluation metrics. We revealed that Logistic Regression and AdaBoost are the best toxic comment classifiers. The average accuracy of Logistic Regression and AdaBoost is 0.895 and 0.893, respectively, where both achieved the same area under the ROC curve score (i.e., 0.828). Therefore, the primary takeaway of this study is that the Logistic Regression and Adaboost leveraging BoW, TF-IDF, or Hashing features can perform sufficiently for toxic comment classification.

Cite This Paper

Md. Abdur Rahman, Abu Nayem, Mahfida Amjad, Md. Saeed Siddik, "How do Machine Learning Algorithms Effectively Classify Toxic Comments? An Empirical Analysis", International Journal of Intelligent Systems and Applications(IJISA), Vol.15, No.4, pp.1-14, 2023. DOI:10.5815/ijisa.2023.04.01

Reference

[1]Jain E, Brown S, Chen J, Neaton E, Baidas M, Dong Z, Gu H, Artan NS. Adversarial text generation for google's perspective api. In 2018 international conference on computational science and computational intelligence (CSCI) 2018 Dec 12 (pp. 1136-1141). IEEE.
[2]Burnap P, Williams ML. Us and them: identifying cyber hate on Twitter across multiple protected characteristics. EPJ Data science. 2016 Dec; 5:1-5.
[3]Georgakopoulos SV, Tasoulis SK, Vrahatis AG, Plagianakos VP. Convolutional neural networks for toxic comment classification. In Proceedings of the 10th hellenic conference on artificial intelligence 2018 Jul 9 (pp. 1-6).
[4]Robinson D, Zhang Z, Tepper J. Hate speech detection on twitter: Feature engineering vs feature selection. In The Semantic Web: ESWC 2018 Satellite Events: ESWC 2018 Satellite Events, Heraklion, Crete, Greece, June 3-7, 2018, Revised Selected Papers 15 2018 (pp. 46-49). Springer International Publishing.
[5]Van Aken B, Risch J, Krestel R, Löser A. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572. 2018 Sep 20.
[6]Rybinski M, Miller W, Del Ser J, Bilbao MN, Aldana-Montes JF. On the design and tuning of machine learning models for language toxicity classification in online platforms. In Intelligent Distributed Computing Xii 2018 (pp. 329-343). Springer International Publishing.
[7]Saif MA, Medvedev AN, Medvedev MA, Atanasova T. Classification of online toxic comments using the logistic regression and neural networks models. In AIP conference proceedings 2018 Dec 10 (Vol. 2048, No. 1, p. 060011). AIP Publishing LLC.
[8]Ibrahim M, Torki M, El-Makky N. Imbalanced toxic comments classification using data augmentation and deep learning. In2018 17th IEEE international conference on machine learning and applications (ICMLA) 2018 Dec 17 (pp. 875-878). IEEE.
[9]Hosam O. Toxic comments identification in arabic social media. Int. J. Comput. Inf. Syst. Ind. Manag. Appl. 2019; 11:219-26.
[10]Zaheri S, Leath J, Stroud D. Toxic comment classification. SMU Data Science Review. 2020;3(1):13.
[11]Almerekhi H, Kwak H, Salminen J, Jansen BJ. Are these comments triggering? predicting triggers of toxicity in online discussions. In Proceedings of the web conference 2020 2020 Apr 20 (pp. 3033-3040).
[12]Vaidya A, Mai F, Ning Y. Empirical analysis of multi-task learning for reducing identity bias in toxic comment detection. InProceedings of the International AAAI Conference on Web and Social Media 2020 May 26 (Vol. 14, pp. 683-693).
[13]Reichert E, Qiu H, Bayrooti J. Reading between the demographic lines: Resolving sources of bias in toxicity classifiers. arXiv preprint arXiv:2006.16402. 2020 Jun 29.
[14]Rastogi C, Mofid N, Hsiao FI. Can we achieve more with less? exploring data augmentation for toxic comment classification. arXiv preprint arXiv:2007.00875. 2020 Jul 2.
[15]Alonso P, Saini R, Kovács G. Hate speech detection using transformer ensembles on the hasoc dataset. In Speech and Computer: 22nd International Conference, SPECOM 2020, St. Petersburg, Russia, October 7–9, 2020, Proceedings 2020 Sep 29 (pp. 13-21). Cham: Springer International Publishing.
[16]Boudjani N, Haralambous Y, Lyubareva I. Toxic Comment Classification for French Online Comments. In2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) 2020 Dec 14 (pp. 1010-1014). IEEE.
[17]Beniwal R, Maurya A. Toxic comment classification using hybrid deep learning model. In Sustainable Communication Networks and Application: Proceedings of ICSCN 2020 2021 (pp. 461-473). Springer Singapore.
[18]Husnain M, Khalid A, Shafi N. A novel preprocessing technique for toxic comment classification. In2021 International Conference on Artificial Intelligence (ICAI) 2021 Apr 5 (pp. 22-27). IEEE.
[19]https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data [Last Accessed: 05-02-2023].
[20]Carta S, Corriga A, Mulas R, Recupero DR, Saia R. A Supervised Multi-class Multi-label Word Embeddings Approach for Toxic Comment Classification. In KDIR 2019 Sep 17 (pp. 105-112).
[21]Risch J, Krestel R. Toxic comment detection in online discussions. Deep learning-based approaches for sentiment analysis. 2020:85-109.
[22]Cox DR. Two further applications of a model for binary regression. Biometrika. 1958 Dec 1;45(3/4):562-5.
[23]Nolan D, Lang DT. Data science in R: A case studies approach to computational reasoning and problem solving. CRC Press; 2015 Apr 21.
[24]He J, Ding L, Jiang L, Ma L. Kernel ridge regression classification. In 2014 International Joint Conference on Neural Networks (IJCNN) 2014 Jul 6 (pp. 2263-2267). IEEE.
[25]Boser BE, Guyon IM, Vapnik VN. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory 1992 Jul 1 (pp. 144-152).
[26]Crammer K, Dekel O, Keshet J, Shalev-Shwartz S, Singer Y. Online passive aggressive algorithms.
[27]Zhang T. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the twenty-first international conference on Machine learning 2004 Jul 4 (p. 116).
[28]Kotsiantis SB, Zaharakis I, Pintelas P. Supervised machine learning: A review of classification techniques. Emerging artificial intelligence applications in computer engineering. 2007 Jun 10;160(1):3-24.
[29]Apté C, Damerau F, Weiss SM. Automated learning of decision rules for text categorization. ACM Transactions on Information Systems (TOIS). 1994 Jul 1;12(3):233-51.
[30]Ho TK. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition 1995 Aug 14 (Vol. 1, pp. 278-282). IEEE.
[31]Quinlan JR. Bagging, boosting, and C4. 5. InAaai/Iaai, vol. 1 1996 Aug 4 (pp. 725-730).
[32]Friedman JH. Greedy function approximation: a gradient boosting machine. Annals of statistics. 2001 Oct 1:1189-232.
[33]Freund Y, Schapire R, Abe N. A short introduction to boosting. Journal-Japanese Society for Artificial Intelligence. 1999 Sep 1;14(771-780):1612.
[34]Nabiilah GZ, Prasetyo SY, Izdihar ZN, Girsang AS. BERT base model for toxic comment analysis on Indonesian social media. Procedia Computer Science. 2023 Jan 1; 216:714-21.
[35]Vatsya R, Ghose S, Singh N, Garg A. Toxic Comment Classification Using Bi-directional GRUs and CNN. In Proceedings of Data Analytics and Management: ICDAM 2021, Volume 2 2022 (pp. 665-672). Springer Singapore.

International Journal of Intelligent Systems and Applications (IJISA)