SPEAKER’S BEHAVIOR IN VIRTUAL REALITY (Methodology of the Experiment and Description of Preliminary Results)
DOI:
https://doi.org/10.17072/2073-6681-2020-4-54-67Keywords:
deixis, verbal means, nonverbal means, virtual reality, coordinate system.Abstract
The article is a preliminary research that examines the methodology of the experiment devoted to the study of the speaker’s behavior in virtual reality (VR) and its results.The purpose of the experiment is to study the specific features of verbal and non-verbal implementation of the role and spatial deixis in VR. Based on the received reactions, the speaker’s coordinate system will be determined and models of the speaker’s communicative behavior in VR will be constructed.The article describes in detail the design of the experiment, which includes: stimulus material on the basis of which the VR scene is created; types of scenes used to determine the type of the speaker’s orientation; description of the tools by means of which VR is created; requirements for the selection of participants in the experiment; methods of recording material and the principles of its decoding. Special attention was paid to correlating the ‘external’ (gesture) and ‘internal’ (verbal) behavior of the speakers, to the principles of gesture fixation and their correlation with the subject’s verbal reactions to the presented scene.Based on the data obtained from the subjects, there was developed a classifier where the categories ‘Structure of the situation’, ‘Type of communication’ and ‘Means of communication’ and their subclasses are used. The data are entered into the Semograph information system and are visually presented in the form of a customizable interactive graph by using the SciVi software.At present, the reactions of two informants (a man and a woman) from the general sample have been processed. Based on the analysis of their reactions, preliminary results were obtained, which made it possible to demonstrate approaches to work with the classifier and to trace the representation of verbal and non-verbal means in the reactions of the subjects. Moreover, preliminary results help to determine the type of orientation and communicative behavior of the speaker depending on their personality and the type of scene. Preliminary analysis allows us to refine the principles of material processing, its annotation and entry into the Semograph system as well as working with the classifier.References
Список литературы
Боронникова Н. В., Талески А., Белоусов К. И., Рябинин К. В. Визуальное представление эгоцентрического поля говорящего в македонском языке (экспериментальное исследование) // Вестник Пермского университета. Российская и зарубежная филология. 2018. Т. 10, вып. 3. С. 13–27. doi 10.17072/2037-6681-2018-3-13-27
Григорьева С. А., Григорьев Н. В., Крейдлин Г. Е. Словарь языка русских жестов. Москва-Вена: Языки русской культуры; Венский славистический альманах. 2001. 256 с. (Язык. Семиотика. Культура).
Зинченко Ю. П., Меньшикова Г. Я., Баяковский Ю. М. Технологии виртуальной реальности: методологические аспекты, достижения и перспективы // Национальный психологический журнал. 2010. № 1(3). С. 54–62.
Каганов Ю. Т. Большой психологический словарь / под ред. Б. Г. Мещерякова, акад. В. П. Зинченко. М.: Прайм-ЕВРОЗНАК 2003. 672 с.
Кибрик А. А. Русский мультиканальный дискурс. Часть II. Разработка корпуса и направления исследований // Психологический журнал. 2018. Т. 39, № 2. С. 79–90.
Литвиненко А. О., Николаева Ю. В., Кибрик А. А. Аннотирование русских мануальных жестов: теоретические и практические вопросы // Компьютерная лингвистика и интеллектуальные технологии: материалы ежегодной Междунар. конф. «Диалог» (Москва, 31 мая – 4 июня 2017 г.). М.: Изд-во РГГУ. 2017. Вып. 16(23). С. 271–286.
Ростовцев-Попель А. А. Типология демонстративов: средние дейктики // Вопросы языкознания. 2009. № 2. С. 22–34.
Рябинин К. В., Баранов Д. А., Белоусов К. И. Интеграция информационной системы Семограф и визуализатора SciVi для решения задач экспертного анализа языкового контента // Научная визуализация. М.: Национальный исследовательский ядерный университет МИФИ. 2017. Т. 9, № 4. С. 67–77. doi 10.26583/sv.9.4.07
Рябинин К. В., Белоусов К. И., Чуприна С. И. Средства визуальной аналитики для комплексного исследования результатов многопараметрического описания пользователей социальных интернет-сервисов // Научная визуализация. 2018. Т. 10, № 4. С. 82–99. doi 10.26583/sv.10.4.07
Ahn, Sun Joo (Grace) Using Avatars and Agents to Promote Real-World Health Behavior Changes // Combs, Donald C., John A. Sokolowsky and Catherine M. Banks (Eds.), The Digital patient: Advancing Healthcare, Research, and Education. New York: John Wiley, 2015. P. 171–180.
Anderson S. R., Keenan Е. Deixis // T. Shopen [ed.] Language Typology and Syntactic Fieldwork. Cambridge: Cambridge University Press. 1985. Vol. III. Р. 259–308.
Atlas S. Inductive metanomics: economic experiments in virtual worlds // Journal of Virtual Worlds Research. 2008. № 1(1). P. 1–15.
Belousov K., Erofeeva E., Leshchenko Y., Baranov D. “Semograph” Information System as a Framework for Network-Based Science and Education // Smart Education and e-Learning 2017. Smart Innovation, Systems and Technologies. Vol. 75. P. 263–272. doi: 10.1007/978-3-319-59451-4_26
Biocca F., Levy M. R. Communication applications of virtual reality // F. Biocca & M. R. Levy (Eds.), Communication in the age of virtual reality. Hillsdale, NJ: Erlbaum, 1995. P. 127–157.
Blascovich J., Loomis J., Beall A. C. et al. Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology // Psychological Inquiry. 2002. Vol. 13(2). P. 103–124.
Diessel H. Demonstratives: Form, function, and grammaticalization. Philadelphia: John Benjamins Publishing Company. 1999. 205 p.
Dolgunsöz E., Yıldırım G., Yıldırım S. The effect of virtual reality on EFL writing performance // Journal of Language and Linguistic Studies. 2018. Vol. 14(1). P. 278–292.
Ekman P. Emotional and Conversational Nonverbal Signals // Messing L. S. & Campbell R. (Eds.), Gesture, Speech, and Sign. New York: Oxford University Press, 1999. P. 45–55.
Ekman F., Friesen W. V. The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding // Semiotica. 1969. № 1(1). P. 49–98.
Fillmore C. J. Santa Cruz lectures on deixis: 1971. Bloomington: Indiana University Linguistics Club, 1975. P. 217–306.
Fillmore C. J. Towards a Descriptive Framework for Spatial Deixis / Ch.Fillmore // Space, Place and Action: Studies in Deixis and Related Topics / Ed. by R. Jarvella and W. Klein. Chichester. New York. 1982. P. 31–59.
Fox J., Arena D., Bailenson J. N. Virtual Reality: A survival guide for the social scientist // Journal of Media Psychology. 2009. Vol. 21(3). P. 95–113.
Heyselaar E., Hagoort P., Segaert K. In dialogue with an avatar, language behavior is identical to dialogue with a human partner // Behav Res Methods. 2015. P. 1–15.
Innocenti A. Virtual Reality Experiments in Economics // Labsi Working Papers. 2015. № 49. P. 1–21. Доступ в SRNN: https://ssrn.com/abstract=2818550https://ssrn.com/abstract=2818550 or https://dx.doi.org/10.2139/ssrn.2818550http://dx.doi.org/10.2139/ssrn.2818550 (дата обращения: 12.05.2020)
Jungbluth K. Deictics in the Conversation Dyad: Findings in Spanish and Some Cross-linguistic Outlines // Deictic Conceptualization of Space, Time and Person / Friedrich Lenz (ed.). Amsterdam, 2003. P. 13–40.
McNeill D. Hand and mind: what gestures reveal about thought. Chicago: University of Chicago Press, 1992. 423 p.
Mariner L. Cleared for Takeoff: English for Pilots. Book 1 // AE Link Publications. 2007. P. 89–90.
Peeters D. Virtual Reality: A Game-Changing Method for the Language Sciences // Psychonomic Bulletin & Review. 2019. Vol. 26(3). P. 894–900. URL: https://link.springer.com/article/10.3758/s13-423-019-01571-3 or https://doi.org/10.3758/s13423-019-01571-3 (дата обращения: 22.04.2020).
Riva J. Virtual Reality in HealthCare: An Introduction // Cyber Therapy & Rehabilitation. 2008. Issue 1. P. 6−9.
Rizzo A., Kim G. J. A SWOT Analysis of the Field of Virtual Reality Rehabilitation and Therapy // Presence: Teleoperators and Virtual Environments. 2005. Vol. 14(2). P. 119–146. doi 10.1162/
Rothbaum B. O., Hodges L. F. The Use of Virtual Reality Exposure in the Treatment of Anxiety Disorders // Behavior Modification. 1990. Vol. 23(4). P. 507−525.
Tarr M. J., Warren W. H. Virtual reality in behavioral neuroscience and beyond // Nature Neuroscience. 2002. 5 (Suppl). P. 1089–92. doi 10.1038/nn948
Zinchenko Yu. P., Kovalev A. I., Menshikova G. Ya. Postnonclassical methodology and application of virtual reality technologies in social research // Psychology in Russia: State of the Art. 2015. Vol. 8(4). P. 60–71.
References
Boronnikova N. V., Taleski A., Belousov K. I., Ryabinin K. V. Vizual’noe predstavlenie egotsentricheskogo polya govoryashchego v makedonskom yazyke (eksperimental’noe issledovanie) [Visual representation of the egocentric field of the speaker in the Macedonian language (experimental study)]. Vestnik Permskogo universiteta. Rossiyskaya i zarubezhnaya filologiya [Perm University Herald. Russian and Foreign Philology], 2018, vol. 10, issue 3, pp. 13–27. doi 10.17072/2037-6681-2018-3-13-27 (In Russ.)
Grigor’eva S. A., Grigor’ev N. V., Kreydlin G. E. Slovar’ yazyka russkikh zhestov [Dictionary of the Russian sign language]. Moscow-Vienna, LRC Publishing House, Viennese Slavic Almanac Publ., 2001. 256 p. (In Russ.)
Zinchenko Yu. P., Menshikova G. Ya., Bayakovskiy Yu. M. Tekhnologii virtualnoy realnosti: metodologicheskie aspekty, dostizheniya i perspektivy [Virtual reality technologies: Methodological aspects, achievements and prospects]. Natsionalnyy psikhologicheskiy zhurnal [National Psychological Journal], 2010, issue 1(3), pp. 54–62. (In Russ.)
Kaganov Yu. T. Bol’shoy psikhologicheskiy
slovar’ [Large psychological dictionary]. Ed. by B. G. Meshcheryakov., acad. V. P. Zinchenko. Moscow, Praym-EVROZNAK Publ., 2003. 672 p. (In Russ.)
Kibrik A. A. Russkiy multikanal’nyy diskurs. Chast’ 2. Razrabotka korpusa i napravleniya issledovaniy [Russian multichannel discourse. Part 2. The corpus development and areas of research]. Psikhologicheskiy zhurnal [Psychological Journal], 2018, vol. 39(2), pp. 79–90. (In Russ.)
Litvinenko A. O., Nikolaeva Yu. V., Kibrik A. A. Annotirovanie russkikh manual’nykh zhestov: teoreticheskie i prakticheskie voprosy [Annotation of Russian manual gestures: theoretical and practical issues]. Kompyuternaya lingvistika i intellektualnye tekhnologii: Po materialam ezhegodnoy Mezhdunarodnoy konferentsii ‘Dialog’. (Moskva, 31 maya – 4 iyunya 2017) [Computational linguistics and intelligent technologies: proceedings of the annual international conference ‘Dialogue’. (Moscow, May 31 – June 4, 2017)]. Moscow, Russian State University for the Humanities Press, 2017, issue 16(23), pp. 271–286. (In Russ.)
Rostovtsev-Popel’ A. A. Tipologiya demonstrativov: srednie deyktiki [Typology of demonstrations: Average deictics]. Voprosy yazykoznaniya [Topics in the Study of Language], 2009, issue 2, pp. 22–34. (In Russ.)
Ryabinin K. V., Baranov D. A., Belousov K. I. Integratsiya informatsionnoy sistemy Semograf i vizualizatora SciVi dlya resheniya zadach ekspertnogo analiza yazykovogo kontenta [Integration of the Semograph information system and the SciVi Visualizer for solving the tasks of lingual content expert analysis]. Nauchnaya vizualizatsiya [Scientific Visualization]. Moscow, National Research Nuclear University ‛MEPhI’ Press, 2017, vol. 9, issue 4, pp. 67–77. doi 10.26583/sv.9.4.07 (In Russ.)
Ryabinin K. V., Belousov K. I., Chuprina S. I. Sredstva vizualnoy analitiki dlya kompleksnogo issledovaniya rezultatov mnogoparametricheskogo opisaniya polzovateley sotsialnykh internet-servisov [Visual analytics tools for systematic exploration of multi-parameter data of social web-based service users]. Nauchnaya vizualizatsiya [Scientific Visualization], 2018, vol. 10, issue 4, pp. 82–99. doi 10.26
/sv.10.4.07 (In Russ.)
Ahn Sun Joo (Grace) Using avatars and agents to promote real-world health behavior changes. The Digital Patient: Advancing Healthcare, Research, and Education. Ed. by Combs C. Donald, John A. Sokolowsky and Catherine M. Banks, New York, John Wiley, 2015, pp. 171–180. (In Eng.)
Anderson S. R., Keenan Е. Deixis. Language Typology and Syntactic Fieldwork. Ed. by T. Shopen. Cambridge, Cambridge University Press, 1985, vol. 3, pp. 259–308. (In Eng.)
Atlas S. Inductive metanomics: economic experiments in virtual worlds. Journal of Virtual Worlds Research, 2008, issue 1(1), pp. 1–15. (In Eng.)
Belousov K., Erofeeva E., Leshchenko Y., Baranov D. ‘Semograph’ information system as a framework for network-based science and education. Smart Education and e-Learning 2017. Smart Innovation, Systems and Technologies, 2017, vol. 75, pp. 263–272. doi 10.1007/978-3-319-59451-4_26 (In Eng.)
Biocca F., Levy M. R. Communication applications of virtual reality. Communication in the
Age of Virtual Reality. Ed. by F. Biocca & M. R. Levy. Hillsdale, NJ, Erlbaum, 1995, pp. 127–157. (In Eng.)
Blascovich J., Loomis J., Beall A. C. et al. Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychological Inquiry, 2002, vol. 13(2), pp. 103–124. (In Eng.)
Diessel H. Demonstratives: Form, Function, and Grammaticalization. Philadelphia, John Benjamins Publishing Company, 1999. 205 p. (In Eng.)
Dolgunsöz E., Yıldırım G., Yıldırım S. The effect of virtual reality on EFL writing performance, Journal of Language and Linguistic Studies, 2018, vol. 14(1), pp. 278–292. (In Eng.)
Ekman P. Emotional and Conversational Nonverbal Signals. Gesture, Speech, and Sign. Ed. by L. S. Messing & R. Campbell. New York, Oxford University Press, 1999, pp. 45–55. (In Eng.)
Ekman F., Friesen W. V. The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1969, issue 1(1), pp. 49–98. (In Eng.)
Fillmore C. J. Santa Cruz lectures on deixis: 1971. Bloomington: Indiana University Linguistics Club, 1975, pp. 217–306. (In Eng.)
Fillmore C. J. Towards a descriptive framework for spatial deixis. Space, Place and Action: Studies in Deixis and Related Topics. Ed. by R. Jarvella and W. Klein. Chichester, N. Y., 1982, pp. 31–59. (In Eng.)
Fox J., Arena D., Bailenson J. N. Virtual reality: A survival guide for the social scientist. Journal of Media Psychology, 2009, vol. 21(3), pp. 95–113. (In Eng.)
Heyselaar E., Hagoort P., Segaert K. In dialogue with an avatar, language behavior is identical to dialogue with a human partner. Behav Res Methods, 2015, pp. 1–15. (In Eng.)
Innocenti A. Virtual reality experiments in economics. Labsi Working Papers, 2015, issue 49, pp. 1–21. Available at SSRN: https://ssrn.com/abstract=2818550https://ssrn.com/abstract=2818550 or https://dx.doi.org/10.2139/ssrn.2818550 (accessed 12.05.2020) (In Eng.)
Jungbluth K. Deictics in the conversation dyad: Findings in Spanish and some cross-linguistic outlines. Deictic Conceptualization of Space, Time and Person. Ed. by Friedrich Lenz. Amsterdam, 2003, pp. 13–40. (In Eng.)
McNeill D. Hand and Mind: What Gestures Reveal About Thought. Chicago, University of Chicago Press, 1992. (In Eng.)
Mariner L. Cleared for Takeoff: English for Pilots, Book 1. AE Link Publications. 2007, pp. 89–90 (In Eng.)
Peeters D. Virtual reality: A game-changing method for the language sciences. Psychonomic Bulletin & Review, 2019, vol. 26(3), pp. 894–900. Available at: https://link.springer.com/article/10.3758/s13-423-019-01571-3 or https://doi.org/10.3758/s13423-019-01571-3 (accessed 22.04.2020) (In Eng.)
Riva J. Virtual reality in healthcare: An introduction. Cyber Therapy & Rehabilitation, 2008, issue 1, pp. 6−9. (In Eng.)
Rizzo A., Kim G. J. A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence: Teleoperators and Virtual Environments, 2005, vol. 14(2), pp. 119–146. doi 10.1162/105474605396709. (In Eng.)
Rothbaum B. O., Hodges L. F. The use of virtual reality exposure in the treatment of anxiety disorders. Behavior Modification, 1990, vol. 23(4), pp. 507−525. (In Eng.)
Tarr M. J., Warren W. H. Virtual reality in behavioral neuroscience and beyond. Nature Neuroscience, 5 (Suppl.), 2002, pp. 1089–1092. doi 10.1038/nn948 (In Eng.)
Zinchenko Yu. P., Kovalev A. I., Menshikova G. Ya. Postnonclassical methodology and application of virtual reality technologies in social research. Psychology in Russia: State of the Art, 2015, vol. 8(4), pp. 60–71. (In Eng.)
