making of webvr experience, three body
Kirill and I made our second WebVR experience, Three Body. It’s a story experience based on the sci-fi novel Three Body Problem by Cixin Liu.
Three Body Problem tells a really amazing story about humanity and interplanetary contact. In the book, the main character plays a WebVR simulation where he experiences life on Trisolaris — a planet in the Alpha Centauri galaxy that lives in the chaotic orbit of three suns. The visual imagery of this planet was captivating, and left me inspired and dreaming of all the shaders I could write to recreate this world…
This project was quite different than most of our previous work, since our goal was to create an immersive experience with a story — instead of making another technical demo.
.. and we learned that this step from a technical demo to an experience with a story was a hard one that really adds to the scope and complexity of the project. All of a sudden we were asking questions such as “Will the user understand what to do here?” or “Is this speech too long for the user to stay focused?” instead of our usual aesthetically focused demo questions about shaders or performance.
So let me start from the beginning. The project took around 4 weeks, and in the first week of starting this project, we had written about 80% of the code. We had an apocalyptic environment with a procedural sky shade that reflected the position of the three procedurally animated suns.
Our main character was a nice floating #FF00FF color box ^^ — character design was very new for us :) We had to find a balance between what would fit the aesthetics of the story but also stay within our technical / time limitations. We decided that character animation would be out of the scope of this project… meaning no limbs. We wanted to have speech but did not want to show a talking animation… so no mouth either.
That left us with a mouthless, limbless creature… sounds rough but we were able to brainstorm some cool masked, hovering designs :)
After some iterations between Houdini and my sketchbook, we ended up with a scholar.
Now we had a mystical scholar hovering around our scene, and a Google Doc of his monologues. We thought about recording his voice ourselves, applying some strong filters in adobe audition, or maybe getting a friend to be a voice actor…
To be honest, I thought that this step was just way out of our expertise. I thought it would be an impossible hurdle and we would just end the project here with a purely visual demo where you just experience the beauty of the planet 0____o
But then, plot twist! Kirill checked on the state of the art text to speech algorithms — and it’s better than Siri! We found a website where you can paste in your text, select your lead voice (we picked Daniel, a nice British man with a deep coarse voice), and out comes a convincingly realistic MP3!
We bypassed some restrictions in order to make requests directly to the API from the terminal with unlimited word count (ask Kirill for details), and started converting our script to speech.
This website accepted Speech Synthesis Markup Language which allowed us to modify the generated speech. We used markups such as break time and prosody rate, but that’s pretty much it. We found the markup language pretty limited in general — it’s hard to markup an emotional statement such as “ quick hurry to the pyramid!!” We took this technical limitation into consideration and built it into the persona of our character, making him a more stoic, mumbling, and sarcastic scholar :)
After sharing with some friends, some were surprised to discover it was AI!
At this point, the experience was more or less functional. The character comes to you from a distance and starts telling you about the wonders of his civilization... until three suns come out over the horizon and the world burns into flames. While the scene had some beautiful visual and dialogue moments, something felt wrong — it was missing interaction.
The scholar had long monologues, and you play a wallflower patiently listening… Even though the monologues were beautifully written, we needed a way to engage the user so that they feel like they are actually part of the scene.
After some brainstorming, we came up with a few ideas. We incorporated a “tour” element, where the scholar guides you to some key spots of the environment, saying lines like “I want to show you something, come, follow me”. This seems like a very simple touch but it was actually quite effective — it broke up the monologues with periods of walking and also made it feel like the scholar acknowledges your presence.
We also added a win state, so you can save yourself from the heat of the three suns by completing certain actions. To complement the story, we added some giant pendulums that the scholar believes “have a hypnotic effect on the sun god”. The scholar asks you to swing the pendulums to help keep the planet in a stable harmony.
These small additions helped the overall engagement level of the experience, yet it still feels slightly lacking in this regard. It would be awesome to have the characters speech actually depend on your actions… but this would take the project much farther than originally planned.
To conclude, making an interactive story base experience was fun but challenging — we spent a lot of time thinking about the usability and design, instead of just technical details. It was hard to strike a balance between our technical abilities and requirements of the story we want to tell. However we are quite happy with the result and encourage you to check it out here, and github for the devs.
Also can check out other projects @ https://codercat.tk
And renders / screenshots @ https://www.instagram.com/snayss/
❤
