Skip to main content
journal@msti-indonesia.com 081377790601
Register
Articles

Machine Learning-Based Sentiment Classification of Reviews from Indonesian Mobile Applications Using TF-IDF

  • 1*,2 Universitas Indraprasta PGRI

* Corresponding Author: Tuti Handayani

Open Access
DOI
https://doi.org/10.59431/jms.v4i2.972
Published
Pages
95–105

Issue

Vol. 4 No. 2 (2026)

Journal Mobile Technologies (JMS)
September

Abstract

The increasing volume of mobile application reviews has created a need for automated sentiment classification capable of handling large-scale and imbalanced user-generated data. This study aimed to compare classical machine learning algorithms and evaluate the effectiveness of class-weighted learning in improving minority-class recognition in Indonesian mobile application reviews. The dataset was cleaned, deduplicated, and controlled for conflicting labels and data leakage, resulting in 357,327 unique reviews across Negative, Neutral, and Positive sentiment classes. Textual features were represented using Term Frequency-Inverse Document Frequency (TF-IDF) with 50,000 features, and the data were divided using an 80:20 stratified train-test split. Logistic Regression, Multinomial Naive Bayes, Linear Support Vector Machine (SVM), and Random Forest were evaluated alongside class-weighted Logistic Regression and Linear SVM. Logistic Regression achieved the highest accuracy of 83.47%, whereas Balanced Linear SVM achieved the highest Macro F1-score of 62.87% and improved the Neutral F1-score from 8.70% to 21.18%, while maintaining an accuracy of 79.09%. The findings demonstrate that the highest overall accuracy does not necessarily indicate the most balanced classifier under class imbalance. Balanced Linear SVM provided the most favorable trade-off between overall performance and minority-class recognition, highlighting the importance of class-sensitive evaluation and class-weighted learning for imbalanced sentiment classification.

Keywords

Class Imbalance Class-Weighted Learning Indonesian Mobile Application Reviews Sentiment Analysis TF-IDF

How to Cite

Handayani, T., & Mardiyati, S. (2026). Machine Learning-Based Sentiment Classification of Reviews from Indonesian Mobile Applications Using TF-IDF. Journal Mobile Technologies (JMS), 4(2), 95–105. https://doi.org/10.59431/jms.v4i2.972

References

  • Abbas, S., Boulila, W., Driss, M., Sampedro, G. A., Abisado, M. B., & Almadhor, A. S. (2024). Active learning empowered sentiment analysis: An approach for optimizing smartphone customer's review sentiment classification. IEEE Transactions on Consumer Electronics. https://doi.org/10.1109/tce.2023.3328878
  • Allam, H., Makubvure, L., Gyamfi, B., Graham, K. N., & Akinwolere, K. (2025). Text classification: How machine learning is revolutionizing text categorization. Information, 16(2), 130. https://doi.org/10.3390/info16020130
  • Assi, M., Hassan, S., & Zou, Y. (2025). LLM-Cure: LLM-based competitor user review analysis for feature enhancement. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3744644
  • Birjali, M., Kasri, M., & Beni-Hssane, A. (2021). A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems, 226, 107134. https://doi.org/10.1016/j.knosys.2021.107134
  • Cunha, W., Canuto, S., Viegas, F., Salles, T., Gomes, C., Mangaravite, V., Resende, E., Rosa, T., Gonçalves, M. A., & Rocha, L. (2020). Extended pre-processing pipeline for text classification: On the role of meta-feature representations, sparsification and selective sampling. Information Processing & Management, 57(4), 102263.
  • Dabrowski, J., Letier, E., Perini, A., & Susi, A. (2022). Analysing app reviews for software engineering: A systematic literature review. Empirical Software Engineering, 27(2), 1–63. https://doi.org/10.1007/s10664-021-10065-7
  • Das, S., Mullick, S. S., & Zelinka, I. (2022). On supervised class-imbalanced learning: An updated perspective and some key challenges. IEEE Transactions on Artificial Intelligence, 3(6), 973–993. https://doi.org/10.1109/TAI.2022.3160658
  • Desuky, A. S., & Hussain, S. (2021). An improved hybrid approach for handling class imbalance problem. Arabian Journal for Science and Engineering, 46(4), 1–12. https://doi.org/10.1007/S13369-021-05347-7
  • Hardiansyah, D., Aziz, R. Z. A., & Hasibuan, M. A. (2024). The classification method is used for sentiment analysis in My Telkomsel. International Journal of Artificial Intelligence Research, 8(2), 169. https://doi.org/10.29099/ijair.v8i2.1229
  • Hassan, S. U., Ahamed, J., & Ahmad, K. (2022). Analytics of machine learning-based algorithms for text classification. Sustainable Operations and Computers, 3, 238–248. https://doi.org/10.1016/j.susoc.2022.03.001
  • Imran, M., Mahmood, A. M., & Qyser, A. A. M. (2014). An empirical experimental evaluation on imbalanced data sets with varied imbalance ratio. International Conference on Computer Communications, 1–7. https://doi.org/10.1109/ICCCT2.2014.7066742
  • Kaope, C., & Pristyanto, Y. (2023). The effect of class imbalance handling on datasets toward classification algorithm performance. MATRIK: Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer, 22(2), 227–238. https://doi.org/10.30812/matrik.v22i2.2515
  • Kholwal, R. (2023). Text-classify: A comprehensive comparative study of logistic regression, random forest, and KNN models for enhanced text classification performance. Zenodo. https://doi.org/10.5281/zenodo.10148008
  • Liu, T., Wang, C., Huang, K., Liang, P., Zhang, B., Daneva, M., & van Sinderen, M. (2023). RoseMatcher: Identifying the impact of user reviews on app updates. Information and Software Technology, 161, 107261. https://doi.org/10.1016/j.infsof.2023.107261
  • Liu, Y., Du, G., Yin, C., Zhang, H., & Wang, J. (2024). Clustering-based incremental learning for imbalanced data classification. Knowledge-Based Systems, 292, 111612. https://doi.org/10.1016/j.knosys.2024.111612
  • Mahmud, Md. S., Bonny, A. J., Saha, U., Jahan, M., Tuna, Z. F., & Marouf, A. Al. (2022). Sentiment analysis from user-generated reviews of ride-sharing mobile applications. International Conference Computing Methodologies and Communication, 738–744. https://doi.org/10.1109/ICCMC53470.2022.9753947
  • Memon, S. A., Mahar, M. A., Kehar, A., Oad, S., & Ahmed, I. (2025). User experience enhancement through sentiment analysis: A machine learning approach to app reviews. International Journal of Information Systems and Computer Technologies. https://doi.org/10.58325/ijisct.004.02.00131
  • Mondal, A. S., Zhu, Y., Bhagat, K. K., & Giacaman, N. (2022). Analysing user reviews of interactive educational apps: A sentiment analysis approach. Interactive Learning Environments, 1–18. https://doi.org/10.1080/10494820.2022.2086578
  • Mortaz, E. (2020). Imbalance accuracy metric for model selection in multi-class imbalance classification problems. Knowledge-Based Systems, 210, 106490. https://doi.org/10.1016/j.knosys.2020.106490
  • Niaz, N. U., Shahariar, K. M. N., & Patwary, M. J. A. (2022). Class imbalance problems in machine learning: A review of methods and future challenges. Proceedings of the 2nd International Conference on Computing Advancements, 485–490.
  • Rekha, G., & Tyagi, A. K. (2020). Necessary information to know to solve class imbalance problem: From a user's perspective (pp. 645–658). https://doi.org/10.1007/978-3-030-29407-6_46
  • Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5), 503–520. https://doi.org/10.1108/00220410410560582
  • Solovyeva, E. B., & Abdullah, A. S. (2022). Comparison of different machine learning approaches to text classification. 1427–1430. https://doi.org/10.1109/ElConRus54750.2022.9755806
  • Tang, Z., Li, W., & Li, Y. (2022). An improved supervised term weighting scheme for text representation and classification. Expert Systems with Applications, 189, 115985. https://doi.org/10.1016/j.eswa.2021.115985
  • Varghese, L., Pai, R. R., Savitha, G., Girisha, S., & Chetty, N. (2025). Manual annotation based sentiment analysis of user feedback in health and wellness app reviews. Scientific Reports, 15(1), 44766. https://doi.org/10.1038/s41598-025-28799-5
  • Vluymans, S. (2019). Dealing with imbalanced and weakly labelled data in machine learning using fuzzy and rough set methods (Vol. 807). Springer International Publishing. https://doi.org/10.1007/978-3-030-04663-7
  • Wang, L., Wang, H., & Fu, G. (2021). Multiple kernel learning with minority oversampling for classifying imbalanced data. IEEE Access, 9, 565–580. https://doi.org/10.1109/ACCESS.2020.3046604
  • Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731–5780. https://doi.org/10.1007/s10462-022-10144-1
  • Zhao, L., Han, F., Ling, Q., Han, H., Yao, Z., Liu, W., & Zhou, Z. (2025). A survey on class imbalance learning algorithms in complex scenarios. IEEE Access, 13, 180799–180833. https://doi.org/10.1109/access.2025.3618909

Author Biographies

  • Tuti Handayani, Universitas Indraprasta PGRI

    Department of Informatics Engineering, Universitas Indraprasta PGRI, South Jakarta City, Special Capital Region of Jakarta, Indonesia

  • Sri Mardiyati, Universitas Indraprasta PGRI

    Department of Informatics Engineering, Universitas Indraprasta PGRI, South Jakarta City, Special Capital Region of Jakarta, Indonesia

License

Copyright (c) 2026 Tuti Handayani, Sri Mardiyati

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

This is an open-access journal. All articles are distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Under this license, you are free to:

  • Share: Copy and redistribute the material in any medium or format.
  • Adapt: Remix, transform, and build upon the material for any purpose, even commercially.

Attribution Requirements:
You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

Copyright and Retention:
Authors retain copyright without restrictions and grant this journal the right of first publication under an open-access model. The journal retains non-exclusive publishing rights for archiving, indexing, and scholarly dissemination.

Usage Statistics