Evaluation of the Dialogue System Efficiency Based on the Application of Fuzzy Inference with Neural Network Settings
DOI:
https://doi.org/10.52575/2687-0932-2022-49-2-356-374Keywords:
dialogue system, virtual assistant, question-answer system, performance evaluation, fuzzy inference, neural network learningAbstract
The results of the study aimed at improving the process of evaluating the effectiveness of the functioning of dialogue systems based on the development of models of fuzzy inference and neural network learning are presented. The relevance of the development of tools for analyzing the performance of question-answer functions by software tools called virtual assistants is substantiated. As values for evaluating the effectiveness of the functioning of dialogue systems, it is proposed to use particular indicators that characterize the accuracy, conciseness and completeness of answers to the questions asked. The resulting performance evaluation is determined by the value of the generalized indicator, calculated taking into account the values of particular indicators. An algorithm for calculating a generalized indicator based on the use of fuzzy inference has been developed. The values of parameters of membership functions and individual conclusions of fuzzy rules necessary for its implementation are proposed to be calculated on the basis of a neural network learning algorithm. The decision to complete the neural network tuning of the fuzzy inference parameters is made based on the calculation and analysis of the current learning error. The results of experimental studies on evaluating the effectiveness of the functioning of dialogue systems based on the proposed algorithms are presented.
Acknowledgments
The reported study was funded by RFBR, project number 20-37-90083.
Downloads
References
Агузумцян Р.В., Великанова А.С., Польщиков К.А., Игитян Е.В., Лихошерстов Р.В. 2021. О применении интеллектуальных технологий обработки естественного языка и средств виртуальной реальности для поддержки принятия решений при подборе исполнителей проектов. Экономика. Информатика, 48 (2): 392–404. DOI 10.52575/2687-0932-2021-48-2-392-404.
Польщиков К.А., Польщикова О.Н., Игитян Е.В., Балакшин М.С. 2019. Алгоритм поддержки принятия решений по выбору средств обработки больших массивов естественно-языковых данных. Научные ведомости Белгородского государственного университета. Серия: Экономика. Информатика. 46 (3): 553–562. DOI 10.18413/2411-3808-2019-46-3-553-562.
Abu Daqar M.A.M., Smoudy A.K.A. 2019. The Role of Artificial Intelligence on Enhancing Customer Experience. International Review of Management and Marketing, 9(4): 22–31.
Balakrishnan J., Dwived Y. K. 2021. Conversational commerce: entering the next stage of AI-powered digital assistants. Annals of Operations Research. URL: https://doi.org/10.1007/s10479-021-04049-5 (accessed: 15.05.2022).
Bylieva D., Lobatyuk V., Kuznetsov D., Anosova N. 2021. How Human Communication Influences Virtual Personal Assistants. Lecture Notes in Networks and Systems, 184: 98–111.
Jannach D., Manzoor A., Cai W., Chen L. 2021. A Survey on Conversational Recommender Systems. ACM Computing Surveys, 54 (5): 1–36.
Karaboga D., Kaya E. 2019. Adaptive network based fuzzy inference system (ANFIS) training approaches: a comprehensive survey. Artificial Intelligence Review, 52: 2263–2293.
Konstantinov I.S., Lazarev S.A., Polshchykov K.O., Mihalev O.V. 2015. Theoretical aspects of evaluation of the corporative portal network traffic management. International Journal of Applied Engineering Research, 10(24): 45691–45696.
Lavie A., Agarwal A. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. Proceedings of the second workshop on statistical machine translation: 228–231.
Lin T.-E., Xu. H. 2019. A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier. Knowledge-Based Systems, 186: 104979.
Papineni K., Roukos S., Ward T., Zhu W.-J. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics: 311-318.
Polshchykov K.A., Lazarev S.A., Konstantinov I.S., Polshchykova O.N., Svoikina L.F., Igityan E.V., Balakshin M.S. 2020. Assessing the Efficiency of Robot Communication. Russian Engineering Research, 40: 936–938.
Polshchykov K., Lazarev S., Polshchykova O., Igityan E. 2019. The Algorithm for Decision-Making Supporting on the Selection of Processing Means for Big Arrays of Natural Language Data. Lobachevskii Journal of Mathematics, 40(11): 1831–1836.
Polshchykov K.O., Lazarev S.A., Zdorovtsov A.D. 2017. Neuro-Fuzzy Control of Data Sending in a Mobile Ad Hoc Network. Journal of Fundamental and Applied Sciences, 9(2S): 1494–1501.
Reis A., Paulino D., Paredes H., Barroso I., Monteiro M.J., Rodrigues V. 2018. Using intelligent personal assistants to assist the elderlies an evaluation of Amazon Alexa, Google Assistant, Microsoft Cortana, and Apple Siri. 2-nd International Conference on Technology and Innovation in Sports, Health and Wellbeing (TISHW): 1–5.
Sun Y., Hu Y., Xing L., Yu J., Xie Y. 2020. History-Adaption Knowledge Incorporation Mechanism for Multi-Turn Dialogue System. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05): 8944–8951.
Takagi T., Sugeno M. 1985. Fuzzy Identification of Systems and Its Applications to Modeling and Control. IEEE Transactions on Systems, Man, and Cybernetics, 15: 116–132.
Tulshan A.S., Dhage S.N. 2019. Survey on Virtual Assistant: Google Assistant, Siri, Cortana, Alexa. Communications in Computer and Information Science, 968: 190–201.
Velikanova A.S., Polshchykov K.A., Likhosherstov R.V., Polshchykova A.K. 2021. The use of virtual reality and fuzzy neural network tools to identify the focus on achieving project results. Journal of Physics: Conference Series. 2nd International Scientific Conference on Artificial Intelligence and Digital Technologies in Technical Systems 2021, Volgograd, 2060: 173707.
Xu H., Peng H., Xie H., Cambria E. 2020. End-to-End latent-variable task-oriented dialogue system with exact log-likelihood optimization. World Wide Web, 23: 1989–2002.
Abstract views: 3927
Share
Published
How to Cite
Issue
Section
Copyright (c) 2022 ECONOMICS. INFORMATION TECHNOLOGIES
This work is licensed under a Creative Commons Attribution 4.0 International License.
