The problem Virtual GPT solves

Interact with time travelers Ada Lovelace, Charles Babbage, Marie Curie, and Leonardo Da Vinci through text-to-text conversations in-game!

Virtual GPT is a gamified interactive experience aimed towards increased immersion in virtual social settings as well as interesting and possibly infinite NPC experience generation in role-playing games. ChatGPT integration along with creative prompt engineering can result in interesting and vastly variable encounters with game characters which enriches overall game experience.

Characters inspired by famous GPT LLM models.

Challenges we ran into

• Time constraint has been the biggest challenge that we faced considering we had decided to work on XR media with GenAI capabilities.
• Another challenge was checking the feasibility of ideas that we thought of along with various software we wanted to utilize. Following are some ideas and software which we tried and had to drop eventually -
  • Creating XR assets using Nvidia Omniverse (The lack of a strong GPU was a problem but we'll admit it was a bit ambitious :) ).
  • We also experimented with 3D storyboards using movie scripts. One particular challenge was to define the scene with modular assets such as characters, buildings (if any), camera movements, etc.
  • GPT-Actor/Gymnast, an XR GPT agent capable of following your commands to do tasks like performing gymnastics or acting out a scene from the user's direction.
  • Loading new environments into the Unity engine was quite harrowing because of rendering pipeline issues, so we ended up using a default map.

We intend to execute the sleuth of ideas we generated during the hackathon and overcome these challenges in our own time.