Enhancing Chatbot Intent Classification using Active Learning Pipeline for Optimized Data Preparation
This study presents a novel approach to enhancing chatbot intent classification through an optimized data preparation combined with Active Learning. We applied the clustering mechanism using a state-of-the-art sentence-transformers model with cosine similarity for cluster detection in order to categorize messages. This process was further refined through a dedicated Active Learning Pipeline, which focused on the most essential observations for labeling. Incorporating externally sourced labeled data from Scale AI, the labeling process was fine-tuned iteratively, until the model's performance stabilized. This approach shows promise for various datasets and tasks, suggesting a scalable solution for preparing data for supervised modeling and achieving optimal model performance in real-world commercial chatbot scenarios.
© 2024 The Author(s). This article is distributed under the terms of the license CC-BY 4.0., which permits any further distribution in any medium, provided the original work is properly cited.
Kuligowska, K., & Kowalczuk, B. (2024). Enhancing chatbot intent classification using active learning pipeline for optimized data preparation. Journal of Applied Economic Sciences, Volume XIX, Fall, 3(85), 317 – 325. https://doi.org/10.57017/jaes.v19.3(85).07
[1] Chandrakala, C. B., Bhardwaj, R., & Pujari, C. (2024). An intent recognition pipeline for conversational AI. International Journal of Information Technology, 16, 731-743. https://doi.org/10.1007/s41870-023-01642-8
[2] Cohn, D., Atlas, L. & Ladner, R. (1994). Improving generalization with active learning. Machine Learning, 15(2), 201–221. https://doi.org/10.1007/BF00993277
[3] Dligach, D., & Palmer, M. (2011). Reducing the Need for Double Annotation. [in:] Ide, N., Meyers, A., Pradhan, S., & Tomanek, K. (eds.). Proceedings of the 5th Linguistic Annotation Workshop (LAW-V). ACL SIGANN, USA, 65–73. https://aclanthology.org/W11-0408
[4] Druck, G., Settles, B., & McCallum, A. (2009). Active learning by labeling features. [in:] Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '09). ACL, Singapore, 81–90. https://dl.acm.org/doi/10.5555/1699510.1699522
[5] Ein-Dor, L., Halfon, A., Gera, A., Shnarch, E., Dankin, L., Choshen, L., Danilevsky, M., Aharonov, R., Katz, Y., & Slonim, N. (2020). Active Learning for BERT: An Empirical Study. [in:] Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 7949-7962. https://doi.org/10.18653/v1/2020.emnlp-main.638
[6] Finch, S. E., Paek, E. S., & Choi, J. D. (2023). Leveraging Large Language Models for Automated Dialogue Analysis. [in:] S. Stoyanchev, S. Joty, D. Schlangen, O. Dusek, C. Kennington, & M. Alikhani (eds.), Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue. ACL. 202-215. https://doi.org/10.18653/v1/2023.sigdial-1.20
[7] Grouin, C., Lavergne, T., & Névéol, A. (2014). Optimizing annotation efforts to build reliable annotated corpora for training statistical models. [in:] Levin L., Stede M. (eds.). Proceedings of the 8th Linguistic Annotation Workshop (LAW-VIII). ACL SIGANN, Ireland, 54–58. https://doi.org/10.3115/v1/W14-4907
[8] Hakkani-Tür, D., Ju, Y.-C., Zweig, G., & Tur, G. (2015). Clustering Novel Intents in a Conversational Interaction System with Semantic Parsing. [in:] Proceedings of Interspeech, Germany. 1854-1858. https://doi.org/10.21437/Interspeech.2015-70
[9] Huggingface (2019). https://huggingface.co/sentence-transformers/distilbert-base-nli-stsb-quora-ranking [accessed 08.2024]
[10] Lewis, D. D., Gale, W. A. (1994). A Sequential Algorithm for Training Text Classifiers. [in:] Croft, B.W., van Rijsbergen, C. J. (eds.), SIGIR ’94, Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. Springer, London. 3-12. https://doi.org/10.1007/978-1-4471-2099-5_1
[11] Meer, M. van der, Falk, N., Murukannaiah, P. K., & Liscio, E. (2024). Annotator-Centric Active Learning for Subjective NLP Tasks. ArXiv, 1-18. https://doi.org/10.48550/arXiv.2404.15720
[12] Mitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., Krishnamurthy, J., Lao, N., Mazaitis, K., Mohamed, T., Nakashole, N., Platanios, E., Ritter, A., Samadi, M., Settles, B., Wang, R., Wijaya, D., Gupta, A., Chen, X., Saparov, A., Greaves, M., & Welling, J. (2018). Never-ending learning. Communications of the ACM, 61(5), 103–115. https://doi.org/10.1145/3191513
[13] Nguyen, D. H. M., & Patrick, J. D. (2014). Supervised machine learning and active learning in classification of radiology reports. Journal of the American Medical Informatics Association, 21(5), 893-901. https://doi.org/10.1136/amiajnl-2013-002516
[14] Ouyang L., Wu J., Jiang X., Almeida D., Wainwright C. L., Mishkin P., Zhang C., Agarwal S., Slama K., Ray A., Schulman J., Hilton J., Kelton F., Miller L., Simens M., Askell A., Welinder P., Christiano P. F., Leike J., & Lowe R. (2022). Training language models to follow instructions with human feedback. [in:] Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 1-15.
[15] Pawlik, Ł., Płaza, M., Deniziak, S., & Boksa, E. (2022). A method for improving bot effectiveness by recognising implicit customer intent in contact centre conversations. Speech Communication, 143, 33-45. https://doi.org/10.1016/j.specom.2022.07.003
[16] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. [in:] Proceedings of the Conference on Empirical Methods in Natural Language Processing. ACL, 1-11. https://doi.org/10.48550/arXiv.1908.10084
[17] Setiaji B., Wibowo F. W. (2016). Chatbot using a knowledge in database: Human-to-machine conversation modeling. [in:] Proceedings of the 7th International Conference on Intelligent Systems, Modelling and Simulation (ISMS), IEEE, 72-77. https://doi.org/10.1109/ISMS.2016.53
[18] Settles, B. (2009). Active Learning Literature Survey. Computer Sciences Technical Report 1648. University of Wisconsin–Madison, 1-44.
[19] Settles, B. (2011). From Theories to Queries: Active Learning in Practice. [in:] Guyon, I., Cawley, G., Dror, G., Lemaire, V., & Statnikov, A. (eds.). Active Learning and Experimental Design Workshop in conjunction with AISTATS 2010, Proceedings of Machine Learning Research, 16, 1-18.
[20] Settles, B., Craven, M. W., & Friedland, L. A. (2008). Active Learning with Real Annotation Costs. [in:] Proceedings of the NIPS Workshop on Cost-Sensitive Learning, 1-10.
[21] Shafi, P. M., Jawalkar, G. S., Kadam, M. A., Ambawale, R. R., & Bankar, S. V. (2020). AI-Assisted Chatbot for E-Commerce to Address Selection of Products from Multiple Products. [in:] Dey, N., Mahalle, P., Shafi, P., Kimabahune, V., & Hassanien, A. (eds.). Internet of Things, Smart Computing and Technology: A Roadmap Ahead. Studies in Systems, Decision and Control, 266, 57-80. https://doi.org/10.1007/978-3-030-39047-1_3
[22] Steegh, E., & Sileno, G. (2023). No labels? No problem! Experiments with active learning strategies for multi-class classification in imbalanced low-resource settings. [in:] Proceedings of the 19th International Conference on Artificial Intelligence and Law (ICAIL '23). ACM, Portugal, 277-286. https://doi.org/10.1145/3594536.3595171
[23] Suryanto, T., Wibawa, A., Hariyono, H., & Nafalski, A. (2023). Evolving Conversations: A Review of Chatbots and Implications in Natural Language Processing for Cultural Heritage Ecosystems. International Journal of Robotics and Control Systems, 3(4), 955-1006. https://doi.org/10.31763/ijrcs.v3i4.1195
[24] Vasquez-Correa, J. C., Guerrero-Sierra, J. C., Pemberty-Tamayo, J. L., Jaramillo, J. E., & Tejada-Castro, A. F. (2021). One System to Rule Them All: A Universal Intent Recognition System for Customer Service Chatbots, 1-26. http://dx.doi.org/10.2139/ssrn.3986692
[25] Wissler, L., Almashraee, M., Monett Diaz, D., & Paschke, A. (2014). The Gold Standard in Corpus Annotation. [in:] Proceedings of the 5th IEEE Student Conference, IEEE, Germany, 1-4. https://doi.org/10.13140/2.1.4316.3523
[26] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., & Rush, A. M. (2020). Transformers: State-of-the-art natural language processing. [in:] Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP). ACL, 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6
[27] Xiao, Z., Zhou, M. X., Chen, W., Yang, H., & Chi, C. (2020). If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills. [in:] Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '20). ACM USA. 1-14. https://doi.org/10.1145/3313831.3376131
[28] Zhang, Z., Strubell, E., & Hovy, E. H. (2022). A Survey of Active Learning for Natural Language Processing. ArXiv. 1-26. https://doi.org/10.48550/arXiv.2210.10109