Oculus Connect 2 : Pixar v. Epic : Life in 11ms

Posted on 2015-10-24 by g2

There was so much to absorb at Oculus Connect 2, and now coming on the heels of Digital Hollywood, my brain is completely full. So instead of making my massive report as I did last year, we’re going to let the knowledge and wisdom trickle down in little pieces. Here’s the first one, from Max Planck, Sascha Unseld, and the team at Oculus Story Studio that produced Henry:

“At Pixar, with our rendering farm, we accepted the truism that it would take roughly 11 minutes to render each frame of animation, producing 24 frames per second.

With Henry, we have a hard wall of rendering each frame, in real time, at 90 frames per second, which translates to 11 milliseconds of rendering time per frame. We spent 6 months optimizing every single aspect of the models, lighting, renderers and animations to assure that we met that 11ms threshold for each and every frame, without compromise.”

11 minutes to 11 milliseconds.

Thank you Moore’s Law.
And Thank You Epic.

We accept this Truth…

Posted on 2015-10-15 by g2

…to be self-evident:

If it performs wonderfully on GearVR:
then it will be a dream on the Oculus Rift,
and take little or no effort to port, other than input harness*

If it performs adequately on the Rift + PC,
it may or may not work well on GearVR.
In fact, it may take both a massive re-factoring as well as a
total re-thinking of graphics, models, textures and code.

Conclusion?

Be smart : Mobile First.

Develop for the Gear. Port to the Rift. WIN.

* a few words on input harnesses:

Designing input mechanisms for the GearVR touchpad is a tricky business… its a VERY limited input surface, and we tend to use both gestures AND a lot of “gaze detection” combined with taps…

for the Rift, we often take the easy way out : keyboard input. At dSky, we are especially fond of the following, which are easy to “find” for “blind” VR users:

space bar (easiest to find… big and central and edge)
cursor arrows at lower right (2nd easiest to find blind)
ESC key (far upper left, also “feelable”)

Truth be told, we should ALL be designing for
a) gamepad, and
b) natural hand tracking devices,
c) with keyboard as a “fallback”
d) oh, did i fail to mention the venerable mouse? ooops!

as the long-term goal for natural and intuitive VR input streams.

Into the Deep : Water, Physics, Bubbles, Sunlight & such

Posted on 2015-08-21 by g2

We’re doing a rapid prototype of how an underwater VR cinematic journey might play out. A primary dynamic visual element is the bubbles. Refraction, expansion, sound FX… all of it! Right after bubbles, come rays of light, caustics, and the miraculous mirror reflection of the surface from below. It should be a fun design and rendering challenge, esp. when we port to the resource-challenged mobile headsets.

Here are some of our sourcebook visual inspirations for the math & physics ahead:

Vivian Spiral dancing beneath the waves. photo credit: Orion Fredericks

Got ideas about VR underwater?

Do you have experience with shaders and rendering and particle animation of underwater scenarios? Post your links below and / or send us your resume!

VR Design : Best Practices

Posted on 2015-07-30 by g2

At GDC 2015, I had some informal conversations with some of the best videogame designers and engineers in the world, and inevitably they all centered around: “we know all about videogames, what do we need to look out for when creating VR?”

Across the course of the conference, I synthesized these key points, which together represent what we feel are the guiding principles of VR design, circa 2015.

From the trenches, to your eyes. Here’s your free guidance:

Best Practices in VR Design

1. Its got to be pure 3d
— 2d tricks no longer work. billboards, masks, overlays etc…
— unless you want to make a stylistic choice
— even your UI is now 100% situated in 3space

2. your geometry, and physics, must be seamless, waterproof, and tight.
— when a player sees the world stereoscopically, small details stand out
— for instance, props that float 1cm above a surface
— and 2mm cracks at wall joins
— these were overlooked in frame games, but are unforgivable in VR

3. really consider detail in textures / normals
— VR has a way of inviting players to inspect objects, props, surfaces and characters…
— up close. really close.
— in a much more intimate level than traditional games
— so be prepared for close inspection
— and make sure that your textures are tight
— along with your collision hulls

4. your collisions for near field objects must be perfect
— fingers can’t penetrate walls
— create detailed high resolution collision shells
. . . for all near sets pieces, props, and characters

5. positional audio is paramount
— audio now has true perceptive 3d positioning, 360° sphere
— you can really effectively guide the users attention and direction with audio prompts
— they will generally turn and look at audio calls for attention.

6. locomotion is key. and hard.
— swivel chair seated experiences are currently optimal
— near-instant high velocity teleports are optimal
— strafing is out, completely : generates total nausea
— 2 primary metaphors are
. . . a) cockpits — cars, planes, ships
. . . b) suited helmets — space suit, scuba mask, ski mask
— cockpits allow physical grounding and help support hard / fast movements
— helmets support HUDs for UI, maps, messaging

7. flying is fun
— a near optimal form of locomotion
— no concerns with ground contact, head bob
— good way to cover large geographies at moderate speed
— managing in-flight collisions:
— a whole ‘nother conversation : force fields and the skillful flying illusion
— speaking of collisions:

8. consider where to place UI
— fixed GUIs suggest a helmet
— local / natural GUIs are more optimal
— consider point of attachment : primaries are:
—— head attachment, which is like a helmet
—— abdomen attachment, which is something you can look down and view

9. graphics performance & frame rate is absolutely key
— the difference between 75fps and 30fps is night and day…
— you MUST deliver 75 fps at a minimum
— don’t ship until you hit this bar
— this isn’t an average, its a floor

10. consider the frustum / tracking volume
— generally, depending on the specific hardware, the positional tracking is in a limited volume
— design your game to optimize performance while in the volume
— and don’t do things that lead players outside the volume
— and gracefully handle what happens when they exit, and then re-enter, the tracking space
— this is similar to the “follow-cam” challenge in trad 3D videogames

11. pacing
— when designing the play experience, consider:
— VR currently favors exploratory experiences above fast paced combat
— this is an absolutely new medium, with its own conventions and rules
— this is a KEY design principle
— be considerate of a users comfort and joy

11+. test test test
— VR experiences are very subjective
— find out what works for your intended audience
— reward your players for their commitment

That’s your high level design direction.

There’s also some great, more detailed technical docs on the web regarding the dirty details of VR dev & design, from the creators themselves. Here they are:

Unreal Engine : VR Dev Best Practices
Oculus VR : Best Practices Guide
LeapMotion : VR Best Practices

Got experience with VR dev / design?
Think we missed something? Want a job?
Comment below:

mobile : the final destination

Posted on 2015-03-29 by g2

Witness : the future. a truly mobile VR picnic.

Way back in 2005, I was in the business of creating massive multiplayer augmented reality systems. My team created playspaces which could read up to 200 players simultaneously, using high-powered projectors to paint the space with light, complex multi-camera arrays to sense the people and their movements… and very highly tuned algorithms to transform those raw camera feeds into usable structured data, in real time, with near-zero latency. This was called markerless motion capture, or markerless mocap. It was before Microsoft Kinect, before time-of-flight, and was considered one of the holy grails of computer vision.

We were able to package all this equipment: CPU, GPU, projector, cameras… all into a single piece of unified hardware. Our first build weighed in at 110 lbs.

The Gizmo v1 was a massive 89 pound steampunk joy machine

I lugged that beast all around North America, paying the steep fees to airlines and hurting my back all the while. Due to the physical stresses, I demanded that we bring the weight in under 50 lbs, the top limit of airlines, and indeed, with some clever mechanical engineering, we were able to accomplish that goal.

Gizmo v3 weighed in at a svelte, travel-ready 49.9 lbs. including high-lumen projector, camera-array, power supply, CPU, GPU, and fans.

Nonetheless, 49.9 lbs was still a hell of a lot to haul around, especially given my 13th story walk-up apartment in Lower East Side, Manhattan where I was based at the time. On the 20th time I climbed those stairs, I swore to the gods above that never again would I haul heavy hardware around the planet.

That promise held true for many years. Until now. Now, in 2015, somehow we find ourselves again in need of high powered GPUs, with the accompanying massive power supplies and cases. Thank the gods, I was able to engineer this thing to less than 20 lbs this time: the cameras are featherlight, and the projectors are replaced by goggles. Instead of projecting to an outerworld, we are creating rich innerworlds. However, its still a massive amount of heavy iron.

which brings us to the key event:
a seminal board meeting of my former company.

Matt, Suzanne, and I were sitting at the massive mahogany conference table, alongside all of our Board of Advisors: brilliant businessmen, financiers, and researchers. We presented our new ultralight 49 lb. unit, the PlayBox Mark IV. My father was in attendance; he played a key role in ushering in the modern era of VR, having launched the military’s SimNet initiative waaay back in 1980. He simply looked at the schematics, and said:

“You do realise, that all that hardware is going to sit inside a cellphone, inside of 5 years?”

At the time, I scoffed:

“A cellphone? That’s ridiculous! Do you realise the graphics supercomputing power we are harnessing to make this a real-time, responsive, computer vision AR system?”

But as the days, months, and years went on, I realised the wisdom of my father’s words. First came the pico-projectors, medium-lumen LED-powered HD projectors that were the size of a matchbox. Next came the low-powered, high-resolution stereoscopic camera arrays, these the size of a dime. And finally came nVidia’s Tegra line of GPUs, ulttra-fast graphics supercomputers that were purpose-designed for smartphones and tablets.

Before I knew it, all the parts were in place.

Which brings us to the present moment.

Once again, we have built graphics supercomputers to ease our entry into real-time, high-performance VR. We tweak and optimize every component to maintain the floor 75fps required for genuine presence.

the engine of our current VR-PC, the venerable Radeon 7990. 400 watts, 4 gigaflops of graphics supercomputing horsepower, and 75fps on our Oculus Rift.

the engine of our current VR-PC, the venerable Radeon 7990. 400 watts of energy draw, 4 gigaflops of graphics supercomputing horsepower, and 75fps on our Oculus Rift.

And then, I got a Samsung Note4 and the GearVR peripheral, a hardware/software combo lovingly hand-architected by none other than John Carmack, designed to deliver high-performance VR in a truly mobile form factor.

Samsung GearVR : the harbinger of the final form factor of VR : light, wireless, fast, mobile.

The shocker? To date, my GearVR has outperformed all desktop solutions we’ve created.

Let me say that again:

A $900 battery-powered 6-ounce smartphone currently outperforms my $2500, 1-kilowatt, 21-pound desktop beast…

AND, the added element of freedom of physical movement is not even factored in here. The ability to bring your GearVR on a picnic in an Adidas sport bag, as opposed to bringing people into your studio and holding the cords out of their way… that alone justifies the Gear.

In short: as soon as possible, dSky will be focusing all efforts on mobile as our lead platform. No worries, Oculus, Sony, and HTC: our apps and experiences will still perform insanely wonderfully on your platforms. Its just, as with the world:

dSky is Mobile First.

dSky VR Studios

VR Experiences to the People

Category Archives: design