The Application of Natural Language Processing to VR for Anti-Bullying Education


Objective:

Applying Natural Language Processing into a virtual reality world to allow a bully and a victim to interact in a natural way.

Features:

A functional Natural Language Processing system, using Python and C#, to allow for a virtual bully and a user to have a verbal interaction.


Demo Video



Materials


Quest Pro

Meta Quest 2

Unity

Unity Engine

paper

Language

paper

Research Paper

Email Me!»


SDKS and Libraries


Gensim and NLTK. Natural Language Processing.

    Gensim and NLTK were used to drive the bully-victim dialogue.
    • The bully initiates the dialogue. After the user says their first sentence, the user's sentence is processed along with the sentences it is being compared to (positive, negative, and neutral).
    • Stop words are removed from all sentences using NLTK Stop Words
    • The dot product of the vectors is given by Sparse Term Similarity Matrix using Gensim.Similarities.
    • The bully sentence with the largest similarity (smaller dot product signifies smaller distance) is then played. Essentially, a user's positive sentence is most similar to the pre-defined positive sentence, etc.

Quest Pro

Python's SpeechRecognition Library. Speech-to-Text.

  • After the bully says its first line, the user must speak. These words must be captured and turned into text to then be classified and processed with Gensim and NLTK.
  • Microphone input is taken using the PyAudio Library and then SpeechRecognition converts it into text.

speech

Other Python Libraries. PyAudio, Playsound, Importlib.

  • PyAudio used to capture microphone input from the user.
  • PlaySound used to play the pre-recorded bully lines.
  • Importlib used to create individual instances of all the python libraries we used, using its name and path of its location in the computer, to be tracked down by Unity Engine.

python