mobile : the final destination

Witness : the future. a truly mobile VR picnic.

Witness : the future. a truly mobile VR picnic.

Way back in 2005, I was in the business of creating massive multiplayer augmented reality systems. My team created playspaces which could read up to 200 players simultaneously, using high-powered projectors to paint the space with light, complex multi-camera arrays to sense the people and their movements… and very highly tuned algorithms to transform those raw camera feeds into usable structured data, in real time, with near-zero latency. This was called markerless motion capture, or markerless mocap. It was before Microsoft Kinect, before time-of-flight, and was considered one of the holy grails of computer vision.

We were able to package all this equipment: CPU, GPU, projector, cameras… all into a single piece of unified hardware. Our first build weighed in at 110 lbs.

The Gizmo v1 was a massive 89 pound steampunk joy machine

The Gizmo v1 was a massive 89 pound steampunk joy machine

I lugged that beast all around North America, paying the steep fees to airlines and hurting my back all the while. Due to the physical stresses, I demanded that we bring the weight in under 50 lbs, the top limit of airlines, and indeed, with some clever mechanical engineering, we were able to accomplish that goal.

Gizmo v3 weighed in at a svelte, travel-ready 49.9 lbs. including high-lumen projector, camera-array, power supply, CPU, GPU, and fans.

Gizmo v3 weighed in at a svelte, travel-ready 49.9 lbs. including high-lumen projector, camera-array, power supply, CPU, GPU, and fans.

Nonetheless, 49.9 lbs was still a hell of a lot to haul around, especially given my 13th story walk-up apartment in Lower East Side, Manhattan where I was based at the time. On the 20th time I climbed those stairs, I swore to the gods above that never again would I haul heavy hardware around the planet.

That promise held true for many years. Until now. Now, in 2015, somehow we find ourselves again in need of high powered GPUs, with the accompanying massive power supplies and cases. Thank the gods, I was able to engineer this thing to less than 20 lbs this time: the cameras are featherlight, and the projectors are replaced by goggles. Instead of projecting to an outerworld, we are creating rich innerworlds. However, its still a massive amount of heavy iron.

which brings us to the key event:
a seminal board meeting of my former company.

Matt, Suzanne, and I were sitting at the massive mahogany conference table, alongside all of our Board of Advisors: brilliant businessmen, financiers, and researchers. We presented our new ultralight 49 lb. unit, the PlayBox Mark IV. My father was in attendance; he played a key role in ushering in the modern era of VR, having launched the military’s SimNet initiative waaay back in 1980. He simply looked at the schematics, and said:

“You do realise, that all that hardware is going to sit inside a cellphone, inside of 5 years?”

At the time, I scoffed:

“A cellphone? That’s ridiculous! Do you realise the graphics supercomputing power we are harnessing to make this a real-time, responsive, computer vision AR system?”

But as the days, months, and years went on, I realised the wisdom of my father’s words. First came the pico-projectors, medium-lumen LED-powered HD projectors that were the size of a matchbox. Next came the low-powered, high-resolution stereoscopic camera arrays, these the size of a dime. And finally came nVidia’s Tegra line of GPUs, ulttra-fast graphics supercomputers that were purpose-designed for smartphones and tablets.

Before I knew it, all the parts were in place.

Which brings us to the present moment.

Once again, we have built graphics supercomputers to ease our entry into real-time, high-performance VR. We tweak and optimize every component to maintain the floor 75fps required for genuine presence.

the engine of our current VR-PC, the venerable Radeon 7990. 400 watts, 4 gigaflops of graphics supercomputing horsepower, and 75fps on our Oculus Rift.

the engine of our current VR-PC, the venerable Radeon 7990. 400 watts of energy draw, 4 gigaflops of graphics supercomputing horsepower, and 75fps on our Oculus Rift.

And then, I got a Samsung Note4 and the GearVR peripheral, a hardware/software combo lovingly hand-architected by none other than John Carmack, designed to deliver high-performance VR in a truly mobile form factor.

Samsung GearVR : the harbinger of the final form factor of VR : light, wireless, fast, mobile.

Samsung GearVR : the harbinger of the final form factor of VR : light, wireless, fast, mobile.

The shocker? To date, my GearVR has outperformed all desktop solutions we’ve created.

Let me say that again:

A $900 battery-powered 6-ounce smartphone currently outperforms my $2500, 1-kilowatt, 21-pound desktop beast…

AND, the added element of freedom of physical movement is not even factored in here. The ability to bring your GearVR on a picnic in an Adidas sport bag, as opposed to bringing people into your studio and holding the cords out of their way… that alone justifies the Gear.

In short: as soon as possible, dSky will be focusing all efforts on mobile as our lead platform. No worries, Oculus, Sony, and HTC: our apps and experiences will still perform insanely wonderfully on your platforms. Its just, as with the world:

dSky is Mobile First.

Unity 5 port complete

Well, the port to Unity 5 took a bit longer than expected. Then again, what port doesn’t? Overall, we’re very happy with the more robust namespace support in code, and the physically based shader model. It took quite some time to re-tool all our custom shaders into a PBR model, but once done, the results are spectacular, no pun intended.

R2-PBR-shaders

R2D2 with the new PBR in Unity5. We’re loving that blue-alloy metal look!

And, we finally solved the mascara issue with all our character models, which we created in Mixamo’s excellent Fuse product. For those techies / artists out there: the trick is to duplicate the existing Legacy/Diffuse-Bump shader for each character, keep the textures and normals, and set the shader model to “Standard / Specular / Fade” with a smoothness of 1.0. Do the same with the eyes, and you’ll have that beautiful “twinkle in the eyes” that all pseudo-living avatars should properly exhibit.

Luke finally drops the mascara and gets real eyebrows -- and a spark

Luke finally drops the mascara and gets real eyebrows — and a spark

In other news, our friends at Magic Leap released their first actual concept video. Just single-player for now, but fun stuff nonetheless.

That’s all for today.

Onward.

The lightsaber… that is all.

killer-best-lightsaber

I realised many months ago that the make or break of our VR experience will be the veracity and feel of the lightsaber. To this end, I’ve developed a wishlist of features:

  1. accurate saber model with metal and buttons
    1. — getting there.
    2. need a really good reflective metal shader
    3. need it to be all solid faces :)
  2. perfect glowing blade
    1. — COMPLETE with alpha textures AND point light source
  3. perfect control by hands
    1. — COMPLETE with Razer Hydra
  4. dynamic motion-driven sound FX
    1. halfway there
    2. need some velocity based flange / distortion
    3. OpenAL or other, transform tip velocity vector to dynamic sFX
  5. hands interaction
    1. switch saber power-on/off from keyboard control to Hydra control
  6. multiple on/of cycles
    1. once lightsaber is off:
    2. blade needs Prefab re-instantiation to be tied to proper place on base
    3. and finally…
  7. interaction with architecture
    1. calculate the intersection of the blade with solids
    2. leaving decals on wall where point hits
    3. showing a fire where it is actively intersection
    4. throwing sparks

Yeah. That should about do it.

lightsaber-cutting-wall

From here forward… Optimization

We recently migrated primary build development from my trusty MacBook Air to our high-performance Windows PC rig… while FPS was churning along at around 15fps on the MBA, I figured we’d easily be hitting the 75fps required for presence on the PC. Lo and behold, that was simply not to be the case… yet.

Shockingly, the frame-rate on the PC is about the same, and the stutters are even worse. This makes me angry. That PC has what was, a year or so ago, the best graphics card that money could buy. A monster Sapphire ATI R9 HD7900, with 3GB of fast DDR graphics RAM, pulling 400w of juice. How could this be doing similar performance to my humble 2012 MacBook Air with Intel 4000 graphics on a tiny chip welded to the motherboard?

Time to run some benchmarks. I used the beautiful Heaven and Valley benchmarks, both free from Unigine (not to be confused with Unreal and/or Unity, ha!). My PC rig scored a respectable(?) 901 on Heaven, average fps of 36, with a max of 66 and a low of 9. Same settings on the Mac, drove a humbling score of 87, average 3.4fps, max 5.3, low 2.4. This would imply a theoretical 10x performance enhancement on the PC… if we were only talking about pure graphics. Good… at least the benchmarks showed as much. Conclusion:  the raw horsepower is there.

1680x1050-4xaa-heaven--MBA--087

Now I just have to do the hard work of getting Unity, and Oculus, and Sixense, to all perform similarly, and to optimize, optimize, optimize until we arrive at that fabled 75fps required for solid presence.

May the force be with me.


UPDATE: BRAINS!

It appears that my CPU was totally the bottleneck. Since the machine was re-purposed from its past life as a CryptoCurrency miner, we invested heavily in the GPU and scantily in the CPU, currently an AMD Sempron 145 @2.6gHz. After sorting through the so-often misinformed reddits on this matter…

OctopusRift has a most excellent resource to build the ultimate VR PC:
http://www.octopusrift.com/building-a-vr-pc/

Tom’s Hardware also appears to be a great resource, specifically:
• http://www.tomshardware.com/charts/ and
Tom’s Gaming CPU Hierarchy Chart March 2015


UPDATE 2: UPGRADES

We’ve gone ahead and purchased a new brain for Anakin. Moving up from the lil Sempron. Here are the new specs: (and yes, 8GB of RAM comes next)

CPU : AMD FX-8350 8-Core Black
GPU : Sapphire Radeon R9 280X Dual-X 3GB GDDR5
HDD : 500gb SSD Samsung 850 EVO
RAM : 4GB Kingston 2GB x2
MBO : ASRock 970 Extreme4 — full ATX socket AM3+
PSU : 1200w Cooler Master Silent Pro Gold BEAST
Stay tuned for the new benchmark results. 

All Hands on Deck

So last night, for the first time, I got my calibrated hands into the VR world. First I had only viewed them on a 2d screen, which was a sort of detached amusement… it felt like operating a robot through telemetry.

seeing your properly calibrated hands appear in VR is a consciousness expanding experience.

seeing your properly calibrated hands appear in VR is a consciousness expanding experience.

Last night, though, I got them in real 3d space, in VR, with perfect calibration… so that I perceived my hands to be precisely where they were in real space, in a virtual environment. I brought my hands right up to my face, and there they were… The effect was similar to my first dive into modern VR… utterly profound.

All night I dreamed about hands, and virtual hands, and virtual keyboards and virtual iPhones. Today, I place the hands into the worlds I’ve been building for the past year, and we see what we can do…

 

finally… HANDS

and finally, we have hands… in VR land. Tuscany, to be precise. coming soon to an HMD near you…

the first time seeing my hands in VR on our own development rig.

the first time seeing my hands in VR on our own development rig.

Thank you so much to Sixense, and Razer Hydra, for making this possible. And to BitCoin, without which we wouldn’t have obtained the hardcore PC to drive this sim.