Amol C. Adamuthe; Sneha Jagtap

Comparative Study of Convolutional Neural Network with Word Embedding Technique for Text Classification

Full Text (PDF, 462KB), PP.56-67

Views: 0 Downloads: 0

Author(s)

Amol C. Adamuthe ^1,* Sneha Jagtap ¹

1. Department of Information Technology, Rajarambapu Institute of Technology, Rajaramnagar, MS, India

* Corresponding author.

DOI: https://doi.org/10.5815/ijisa.2019.08.06

Received: 29 Jan. 2019 / Revised: 5 Mar. 2019 / Accepted: 20 Mar. 2019 / Published: 8 Aug. 2019

Index Terms

Convolutional Neural Network, Text Classification, Text mining, Word2Vec

Abstract

This paper presents an investigation of the convolutional neural network (CNN) with Word2Vec word embedding technique for text classification. Performance of CNN is tested on seven benchmark datasets with a different number of classes, training and testing samples. Test classification results obtained from proposed CNN are compared with results of CNN models and other classifiers reported in the literature. Investigation shows that CNN models are better suitable for text classification than other techniques. The main objective of the paper is to identify best-fitted parameter values batch size, epochs, activation function, dropout rates and feature maps values. Results of proposed CNN are better than many other classification techniques reported in the literature for Yelp Review Polarity dataset and Amazon Review Polarity dataset. For all the seven datasets, accuracy obtained by proposed CNN is close to the best-known results from the literature.

Cite This Paper

Amol C. Adamuthe, Sneha Jagtap, "Comparative Study of Convolutional Neural Network with Word Embedding Technique for Text Classification", International Journal of Intelligent Systems and Applications(IJISA), Vol.11, No.8, pp.56-67, 2019. DOI:10.5815/ijisa.2019.08.06

Reference

[1]M. Ikonomakis, S. Kotsiantis and V. Tampakas, “Text Classification Using Machine Learning Techniques”, WSEAS transactions on computers, vol. 4, no. 8, pp. 966-974, 2005
[2]R. Talib, M. Kashif Hanif, S. Ayesha and F. Fatima, “Text Mining: Techniques, Applications and Issues”, International Journal of Advanced Computer Science and Applications, vol. 7, no. 11, 2016. http://dx.doi.org/10.14569/IJACSA.2016.071153
[3]A. Abbasi, H. Chen and A. Salem, “Sentiment analysis in multiple languages: Feature selection for opinion classification in Web forums”, ACM Transactions on Information Systems (TOIS), vol. 26, no. 3, p. 12, 2008. http://dx.doi.org/10.1145/1361684.1361685
[4]S. Günal, S. Ergin, M. Gülmezoğlu and Ö. Gerek,“On Feature Extraction for Spam E-Mail Detection“, International Workshop on Multimedia Content Representation, Classification and Security, 2006, pp. 635--642. http://dx.doi.org/10.1007/11848035_84
[5]A. Harb, M. Plantié, G. Dray, M. Roche, F. Trousset and P. Poncelet, “Web Opinion Mining: How to extract opinions from blogs?”, Proceedings of the 5th international conference on Soft computing as transdisciplinary science and technology, pp. 211--217, 2008. http://dx.doi.org/10.1145/1456223.1456269
[6]X. Zhu, J. Huang, Z. Zhou and Y. Han, “Chinese Article Classification Oriented to Social Network Based on Convolutional Neural Networks”, Data Science in Cyberspace (DSC), IEEE International Conference on, pp. 33--36, 2016. http://dx.doi.org/10.1109/DSC.2016.28
[7]S. Liaoa, J. Wangb, R. Yu, K. Sato and Z. Cheng, “CNN for situations understanding based on sentiment analysis of twitter data”, Procedia computer science, vol. 111, pp. 376--381, 2017. http://dx.doi.org/10.1016/j.procs.2017.06.037
[8]M. Hughes, I. Li, S. Kotoulas and T. Suzumura, “Medical text classification using convolutional neural networks”, Stud Health Technol Inform, vol. 235, pp. 246--50, 2017.
[9]M. Allahyari, S. Pouriyeh, M. Assefi, S. Safaei, E. Trippe, J. Gutierrez and K. Kochut, “A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques”, arXiv preprint arXiv:1707.02919, 2017.
[10]C. Liu, S. Zhao and M. Volkovs, “Learning Document Embeddings With CNNs.” arXiv preprint arXiv:1711.04168, 2017.
[11]R. Duda, P. Hart and D. Stork, “Pattern Classification”, 2012.
[12]Y. Yang and X. Liu, "A re-examination of text categorization methods", Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 42--49, 1999. http://dx.doi.org/10.1145/312624.312647
[13]P. Bednár, “Active learning of SVM and decision tree classifiers for text categorization”, Fourth Slovakian-Hungarian Joint Symposium on Applied Machine Intelligence, Herlany, Slovakia, 2006.
[14]W. Wang, D. Do and X. Lin,“Term graph model for text classification“, International Conference on Advanced Data Mining and Applications, pp. 19--30, 2005. http://dx.doi.org/10.1007/11527503_5
[15]V. Korde and C. Mahender, “Text classification and classifiers: A survey”, International Journal of Artificial Intelligence \& Applications, vol. 3, no. 2, p. 85, 2012.
[16]S. Kamruzzaman, F. Haider and A. Hasan, “Text classification using data mining”, arXiv preprint arXiv:1009.4987, 2010.
[17]B. Pang and L. Lee,”Opinion mining and sentiment analysis”, Foundations and Trends in Information Retrival, vol. 2, No. 1-2, pp. 1--135,2008. http://dx.doi.org/10.1561/1500000011
[18]V. Gupta, “Recent trends in text classification techniques”, International Journal of Computer Applications, vol. 35, no. 6, 2011.
[19]Y. Kim, “Convolutional neural networks for sentence classification”, arXiv preprint arXiv:1408.5882, 2014. http://dx.doi.org/10.3115/v1/D14-1181
[20]A. Hassan and A. Mahmood,“Deep learning for sentence classification.”, Systems, Applications and Technology Conference (LISAT), 2017 IEEE Long Island, 2017, pp. 1--5. http://dx.doi.org/10.1109/LISAT.2017.8001979
[21]C. dos Santos and M. Gatti, “Deep convolutional neural networks for sentiment analysis of short texts”, Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 69--78, 2014.
[22]Y. He and D. Zhou, “Self-training from labeled features for sentiment analysis”, Information Processing \& Management, vol. 47, no. 4, pp. 606--616, 2011. http://dx.doi.org/10.1016/j.ipm.2010.11.003
[23]B. Pang and L. Lee, “A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts”, Proceedings of the 42nd annual meeting on Association for Computational Linguistics, 2004, p. 271, http://dx.doi.org/10.3115/1218955.1218990
[24]A. Maas, R. Daly, P. Pham, D. Huang, A. Ng, and C. Potts, “Learning word vectors for sentiment analysis”, Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, 2011, pp. 142--150.
[25]Y. Zhang, M. Er, R. Venkatesan, N. Wang and M. Pratama, “Sentiment classification using comprehensive attention recurrent models”, Neural Networks (IJCNN), 2016 International Joint Conference on, pp. 1562--1569, 2016. http://dx.doi.org/10.1109/IJCNN.2016.7727384
[26]Y. Long, Q. Lu, R. Xiang, M. Li and C. Huang, “A cognition based attention model for sentiment analysis”, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 462--471, 2017.
[27]Y. Wen, W. Zhang, R. Luo and J. Wang, “Learning text representation using recurrent convolutional neural network with highway layers”, arXiv preprint arXiv:1606.06905, 2016.
[28]S. Wang and C. Manning, “Baselines and bigrams: Simple, good sentiment and topic classification”, Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, 2012, pp. 90--94.
[29]R. Johnson and T. Zhang, “Supervised and semi-supervised text categorization using LSTM for region embeddings”, arXiv preprint arXiv:1602.02373, 2016.
[30]S. Wang and C. Manning, “Fast dropout training”. international conference on machine learning, 2013, pp. 118--126.
[31]J. Du, L. Gui, R. Xu and Y. He, “A Convolutional Attention Model for Text Classification”, National CCF Conference on Natural Language Processing and Chinese Computing, pp. 183--195, 2017. http://dx.doi.org/10.1007/978-3-319-73618-1_16
[32]P. Liu, X. Qiu and X. Huang, “Recurrent neural network for text classification with multi-task learning”, arXiv preprint arXiv:1605.05101, 2016.
[33]Q. Le and T. Mikolov, “Distributed representations of sentences and documents”, International Conference on Machine Learning, pp. 1188--1196, 2014.
[34]T. Miyato, A. Dai and I. Goodfellow, “Adversarial training methods for semi-supervised text classification”, arXiv preprint arXiv:1605.07725, 2016.
[35]J. Hong and M. Fang, “Sentiment analysis with deeply learned distributed representations of variable length texts”, Technical report, Stanford University, 2015.
[36]D. Yogatama, C. Dyer, W. Ling, and P. Blunsom, “Generative and discriminative text classification with recurrent neural networks”, arXiv preprint arXiv:1703.01898, 2017.
[37]S. Mukherjee, S. Dutta, and G. Weikum, “Credible Review Detection with Limited Information using Consistency Analysis”, arXiv preprint arXiv:1705.02668, 2017.
[38]A. Salinca, “Business reviews classification using sentiment analysis”, 2015 17th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), 2015, pp. 247--250. http://dx.doi.org/10.1109/SYNASC.2015.46
[39]B. Nguy, “Evaluate Helpfulness in Amazon Reviews Using Deep Learning.”
[40]K. Jónsson and D. Platt, “CSE 255 Assignment 1: Helpfulness in Amazon Reviews.”
[41]M. Agarwal and M. Bhand, “Classification of Amazon Reviews.”, 2009.
[42]J. Vijayabhaskar, R. Sridhar and P. Vijayaragavan, “User Review Sentiment Classification and Aspect Extraction”, Imperial Journal of Interdisciplinary Research, vol. 3, no. 4, 2017.
[43]C. Qiao, B. Huang, G. Niu, D. Li, D. Dong, W. He, D. Yu and H. Wu, “A new method of region embedding for text classification.”, ICLR, 2018
[44]A. Joulin, E. Grave, P. Bojanowski and T. Mikolov, “Bag of tricks for efficient text classification.” arXiv preprint arXiv:1607.01759, 2016. http://dx.doi.org/10.18653/v1/E17-2068
[45]Y. Xiao and K. Cho, “Efficient character-level document classification by combining convolution and recurrent layers.” arXiv preprint arXiv:1602.00367, 2016.
[46]X. Zhang, J. Zhao and Y. LeCun, “Character-level convolutional networks for text classification”, Advances in neural information processing systems, 2015, pp. 649--657.
[47]G. Lee, J. Jeong, S. Sao, C. Kim and P. Kang, “Sentiment classification with word localization based on weakly supervised learning with a convolutional neural network.” Knowledge-Based Systems, vol. 152, pp.70-82, 2018. http://dx.doi.org/10.1016/j.knosys.2018.04.006
[48]G. Lee, J. Jeong, S. Seo, C. Kim and P. Kang1, “Sentiment Classification with Word Attention based on Weakly Supervised Leaning with a Convolutional Neural Network”, arXiv preprint arXiv:1709.09885, 2017.
[49]H. Bedi and N. Cheke, “Sentiment Analysis of Movie Reviews ”, 2012.
[50]S. Chintala, “Sentiment Analysis using neural architectures.” New York University, 2012.
[51]A. Mandelbaum and A. Shalev, “Word embeddings and their use in sentence classification tasks”, arXiv preprint arXiv:1610.08229, 2016.
[52]G. Chen, D. Ye, Z. Xing, J. Chen and E. Cambria, “Ensemble application of convolutional and recurrent neural networks for multi-label text categorization”, Neural Networks (IJCNN), 2017 International Joint Conference on, pp. 2377--2383, 2017. http://dx.doi.org/10.1109/IJCNN.2017.7966144
[53]https://drive.google.com/drive/u/0/folders/0Bz8a_Dbh9Qhbfll6bVpmNUtUcFdjYmF2SEpmZUZUcVNiMUw1TWN6RDV3a0JHT3kxLVhVR2M

International Journal of Intelligent Systems and Applications (IJISA)