VR is a lot more fun with friends. As TiltBrush multiplayer doesn’t seem to be anywhere close to release, I decided I would spend a few days adding Oculus Quest support and multiplayer functionality to a vr painting code base I created a while back.
You can download the app from here to try with a friend: https://github.com/eman-insilico/Virtual-Studio/releases
THE PROCESS
Porting to Quest
The first step was to add Oculus Quest support to the painting tools I already had from Virtual Studio (link to repository). Since the original code was using SteamVR, I had to replace all the buttons (trigger, grip and touchpad) API with the Oculus SDK api.
This was fairly straightforward. I deleted the SteamVR plugin folder so that I get all the errors that will directly lead me to where the SteamVR code was used. Then I added the Oculus Utilities plugin from the assets store and replaced each call to the SteamVR buttons api with the OVR api. Besides the touchpad and joystick difference in vive and oculus controllers which took a bit of code re-writing, everything else just worked.
Then I had to change the target build to ‘Android’, remove realtime lighting and change some of the materials to get better performance on mobile.
Networking
Once I had the painting tools working on the Quest it was time to start working on the networking. As I only had a few days to make it all happen, I needed a networking system that is very simple to use and very robust. For this reason, I chose to go with OSC. There are many Unity OSC plugins out there. I used this one which worked fine for what I needed. OSC is based on UDP so it is very fast and most importantly very flexible. All you need is the receiver IP and a port number to send a message. The Unity OSC script allows you to send ints and floats which the scripts converts to byte arrays and sends as UDP packets. This was all I needed for sending the paint data over.
The painting in Virtual Studio is done by extruding a mesh along the direction of movement. You can see this better and learn more about it here. So to network the painting, all I needed was to sync the movement of the paintbrush from one quest to another and I would be able to perfectly recreate each brushstroke.
When a user pulls on the trigger to start painting, I capture the position and rotation and send that over to the other quest. On the receiving end, I use this information to instantiate a brush at that position and rotation. Then, as the user moves their hand I keep sending the position and rotation of the brush to the other side until the user lets go of the trigger. Along with position and rotation data, I also send brush type information, colour, size, etc.
“Matchmaking”
Once I had the painting networked, I needed some avatars and some kind of matchmaking, so that users will be able to just connect to their wifi, jump in the app and would automatically see each other. At the moment, the IP’s were fixed, so I needed to find a way to discover the IP’s on the local network. Luckily this was easily done. By sending a message to the address 255.255.255.255 you can send UDP packets to all the addresses in the local network. So at the start of the app, I send a ‘discovery message’ which contains the local IP address and a request to join. If any response is received it joins as the second player. If not, it joins as the first player, waits for someone to join and uses the IP information contained in the received message to know where to send information from now on.
For syncing the avatars, I wrote a simple script that sends position and rotation data similar to the networked painting. Below you can see the player setup in Unity.
End notes
There is a lot more work to be done to turn this demo into a useful collaborative painting app. However, the simplicity of this networking design pattern makes me positive about this project and developing collaborative applications in general.
You can download the final result from here: https://github.com/eman-insilico/Virtual-Studio/releases
And here is me practicing my life draing skills in VR:
Emanuel Tomozei 