Why OTOY will change our 3D world: A conversation with Jules Urbach

Everyone once in a while a very bright and tremendously driven young person comes up within the computer software industry—in time introducing something that will change our world. Jules Urbach may just be one of those people. Except, he’s not entirely that young anymore. Fortunately for me my iPad’s note taking app has a digital recorder, which I completely had to rely on to keep up with a 39 year-old Urbach who—if you didn’t know any better—you would assume is another fast-talking, techno-skater wunderkind happily enjoying the glory of Silicon Valley.

Yet, Jules Urbach isn’t in Silicon Valley—and for good reason. And he’s not a young Bill Gates or Mark Zuckerberg—though like both of those tech stars he too was once a Harvard bound kid who bailed on the comfort and retreat of cozy, ivy-layered quads for self-induced, caffeine enriched late-night solo coding sessions. Those sessions of course led the young Jules to create the hit game “Hell Cab” for Time Warner while in his teens. He later pushed gaming tech towards more realism and onto the Net into what we now refer as the cloud.

But for the past ten or so years Jules Urbach has been perfecting a digital recipe for both leveraging the massive computational power of the standard GPUs inside servers on the cloud as well as devising a way to compress intensely realistic imagery—which is typically massive in file size—down to something that can literally stream over the Net just like your favorite YouTube video. And with all of this comes several sets of almost magical promises for the future, including—and quite importantly for our readers—the ability to run just about any complex 3D application on any platform, on any device and remotely over the Internet. If this happens—and there is a very good chance it will—every industry from architecture, industrial design to computer gaming and Hollywood Blockbusters will be affected.

Sound exciting? How about interesting?

We think this is very important. And so I sat down virtually with Jules Urbach a few weeks ago to chat about all of this transformative technology his company OTOY is up to. What follows are the highlights of our conversation.

The Emergence of OTOY

AFR (Anthony Frausto-Robledo):  So tell me, what are you focused on, what keeps you up at night and what do you love? 

JU (Jules Urbach):  I love this work. OTOY was really built around graphics—even the streaming stuff we do with the cloud—it all stems from the GPU. Literally every product we’ve done has some connection to the GPU. The coder and decoders all run on pixel shaders. We want to be a part of making graphics for the better!

AFR: How did you personally get started with computer graphics?

JU:  I always wanted to work on films and games. My first job was working for Time Warner on “Hell Cab” in ’93.  I got into Harvard and I just dropped out. I never went back—so I could do that game. I was focused on video games up until around ’96, then I started to focus on streaming games and then improving the rendering in game engines. Then I realized you could get raytracing to run on GPUs—and also the whole complex shader pipeline for “final renders” to also work on GPUs—and that’s when things got really exciting.

Okay, so I want to do a game engine and a final render engine that basically have a shared core.  And I want to do it on the cloud where compute power isn’t even an issue. And that really was the genesis of OTOY today; those vectors led me there.

So as we’ve been building and testing various pieces we have also done work on films, like Transformers, and games for Hasbro and others—testing and improving both. Soon the streaming became a really big part of what we were doing.

AFR: And where are we in time when this sequence emerged?

JU: That came together around 2009. And that’s when OTOY, as a company, really started to form and when we got our first investment in.

AFR: When did you begin partnering with Autodesk?

JU: Well our largest partner on the streaming side is Autodesk, who came to us about four years ago—ideally to help them build a web-based framework to allow them to deliver their applications and forward-looking services, remotely, and to have us work on those pieces. And shortly after that Octane came to market as the first “final render” quality GPU renderer. And since then we have basically been involved on all those various pieces.

And…yeah! That’s basically how this all came to be, but there is more on the business side.

Harvard, You’re Not “New” Enough

AFR: Okay, we can do the business side later. But first I want to know about your earlier roots, as I understand that you dropped out of Harvard. Where did you learn programming if not in college? You are a programmer, right?

JU: Yeah, I’m self-taught. I took some courses in high-school but I wanted to do things a totally different way. They weren’t focused on graphics really, where I was taking classes; and at Harvard I was interested in the cross-enroll with MIT because—of course—they have the Media Lab. But none of that stuff was new.

The point was if they actually taught it it wasn’t going to be new and innovative. And I wanted to just totally create stuff from scratch. And a lot of the stuff that I did, that led up to OTOY, really even started back in the video game world. I tried to do streaming and getting games to run in the cloud—and all that stuff happened in the early and mid ’90’s.

So yeah, I taught myself everything. I taught myself how to program GPUs, program in C++. And that is just how I learned to progress. I don’t do nearly as much programming now. Thankfully there is a team that has been able to pickup a lot of the pieces, but pretty much up until 2010, a huge amount of the work of everything we did was done by me. And it was good, it helped set a lot of the frameworks we have in place, now that we are a larger company.

AFR: So I think I got this right, help me out here now…but I think it was around 2010 or 2011 that Octane Render had its coming out party of sorts…that’s when OTOY emerged.

JU: It ended up being that way but it was almost an accident—because Octane was a product that we brought into OTOY.  It was created by the guy who created LuxRender, Terrance Vergauwen; and I think he literally spent months alone building the product and basically when we acquired it we put a team around it. But he put it out there on a website around 2010 and one of our guys immediately saw it and we were pretty excited by it and I immediately wanted it to be a part of OTOY.

AFR: Why?

JU: It is a very different way of rendering. I had already been doing GPU-based raytracing in a “biased” way; Terrance had done it in an “un-biased” manner and that’s why I felt we needed it—it was a great complement to what we were doing, especially to finally crack the “final render” piece on the GPU side. I think we announced it [the acquisition] officially in 2011 and it became part of OTOY in 2012.

next page: Move Over OpenGL and DirectX—Meet the New “Brigade”

[title image: Jules Urbach inside OTOY’s Light Stage technology. Image courtesy of OTOY Inc., All rights reserved.]

Move Over OpenGL and DirectX—Meet the New “Brigade”

AFR: So tell me about Brigade. What is it?

JU: OTOY is starting to focus on the bigger eco-system and Brigade is one of those pieces. Brigade is going back to my desire to see cloud gaming being taken to the next level. I think there is definitely something to be said for just putting a game in the cloud, and not have to worry about the hardware; but we think that if you are going to game on the cloud you have to offer people way more GPU power—an order of magnitude more than they get on their device. And of course that is becoming more challenging now where you actually have the integrated graphics which are becoming pretty good. We now have OpenGL ES 3.1 on phones where we have compute shaders, right? So today Brigade takes it to the next level providing Octane-level realism with enough shortcuts to make 50 hz game play viable.

And with Amazon providing so many GPUs and doing that worldwide, we can actually launch a cloud gaming platform around that and they have been very encouraging in that respect. OTOY has been putting all the technology pieces together for that.

So the pipeline for Brigade will be basically Octane. Octane exports this package file format, which we are open-sourcing as ORBX, which can be video like we see in streaming online today plus contain all the data used to render in Octane. So Brigade can consume that and then you basically have a ready to go, physically correct rendering pipeline, for cloud gaming.  And of course with the Octane ecosystem touching just about every 3D app out there—and now getting into 2D apps like After Effects (AE) and Photoshop—the idea is you should be able to export and import an ORBX file as simply as you do a JPEG. And that’s what we are trying to accomplish with the interchange format.

AFR: And do the two feed each other?

JU: There is definitely some cross-pollination between Octane and Brigade. Some of the things that made Octane better came from the research that went into Brigade.

AFR: So how does Brigade work on an API level? Is it working above OpenGL and DirectX or instead of?

JU: It’s totally separate, it uses compute to do all the rendering; it doesn’t use any OpenGL or DirectX to do the 3D rendering. It’s all done in compute shaders—just like Octane is.

OpenGL and DirectX are rasterizers, you send triangles and they are rendered to the screen, back to front and all that stuff—and there’s fixed function hardware that does that stuff very quickly. However, you can’t do reflections, you can’t do physically correct light and shades so you have to fake all that with pixel shaders and vertex shaders, so Brigade is very much like Octane where it is doing the full path raytracing  calculation. You get basically perfect lighting. It doesn’t have the full material system that Octane does but you end up with the speed of OpenGL or DirectX with dynamic environments and characters.

So Brigade is basically a replacement for DirectX and OpenGL. In fact, the way we see it being used is as a component in a game engine. So if we look at something like Unity or the Source Engine or Unreal, those things have multiple backends, right, so they all support various versions of DirectX, OpenGL, OpenGL ES, et cetera, and Brigade is basically just another backend target. And since some of those have physically correct materials, you just plugin into Brigade and suddenly all your games are raytraced or path traced. And you are not limited to two GPUs to do frame interleaving—you can have a whole cloud of them and do that on demand but remotely.

AFR: So Brigade is doing all the geometry too or just a part of it, in tandem with OpenGL or DirectX?

JU: No, Brigade is doing all of the geometry—direct triangle calls, et cetera. We can do total procedural shapes and things like that—that don’t use a tessellation system. We can render a perfect sphere; we can render a strand. And none of them need to be Direct drawn or OpenGL primitives, they can be completely novel, and that gives us this massive amount of flexibility.

So Brigade is a full raytracing system, geometry to shading to final output, and there is no DirectX or OpenGL in any of those parts.

It All Runs on Octane

AFR: You have a lot of plugins for so many 3D applications out there, just about everybody. Who is coming to who in these deals and arrangements? And the second part of this question is, obviously, you have a major partner in Autodesk. Can you step back and explain where you see that going in the big picture sense?

JU: I guess I’ll start talking about the first question first and then get to the second one. So with Octane we really don’t reach out to that many other companies. Some of the plugins are just in-house, like Maya, Max and Blender—those are obvious ones—some of them like Poser are being led by advanced users who ask to get the SDK, work with us to build it, and we split the revenue with them. And we do that. And they get most of the revenue and credit for doing that work. We just help them create it. And that is why we have so many plugins. We just get a reasonable minor percentage.

I think we have done every 3D app. I don’t think there is a 3D app we haven’t yet done. That’s why we are onto the 2D apps now.  Anyone who makes an app that supports plugins, we are able to get an Octane render plugin done in a matter of months. We think down the road we will likely split Octane and it will become a web service. So just like you can upload or embed YouTube videos—you will be able to upload Octane renders.

Eventually we see Octane just being available as a web service and that way others can even more easily embed it into their apps and it becomes just another service out there. I think that is the future, where you will be able to compute on demand. I think that is a big part of it. That is where the future is!

AFR: And what goes on the client…

JU: And intelligently putting some stuff on the client. I think the Web today is powerful. HTML-5 is very powerful. You look at what Famous just launched and look at how good that looks!—you have enough there to build a really good front end for any application, you really don’t need that anymore. And if you do it correctly your app will run everywhere, every phone OS and every platform.

AFR: This is adding color to the whole Autodesk thing…thanks for explaining this.

JU: I think Autodesk is looking at this very correctly. They are building out Autodesk 360 as this hub and if you look at what we just launched in November, we are just one step away from delivering Max or Maya as part of that.

What we did with Autodesk was prove the technology—that it will stream—that the apps can run in the cloud and the next version or iteration of all this will be greatly improved. That’s why we say we are probably at the mid-point of even what we are doing with Autodesk. I think the next steps are to sit down and use the technology more. We took the first steps.

Now that you saw that phase done and very successfully, you will probably see that stuff adopted within Autodesk. You may find that every team at Autodesk finds a way to do the web-based version of an app that used to require a desktop piece. And I don’t think it’s going to be pure loading. I think WebGL will be done locally and only the hardest stuff will be done in the cloud. And ultimately what we are trying to build is the technology to fuel this. You will still have the UI/UX stuff being powered at the desktop, so you are not going to be viewing a simple steam, but the power of the cloud will take over the hard number crunching parts.

AFR: Those are very large and transformational changes. 

JU: That’s where I think we are going. That’s where companies like Lagoa are—that are purely browser based—and I think that is really cool too . . . But the idea is that the technology we are bringing to market you can do that without any compromise and not lose any of the features that you are used to having on a high-end desktop experience. So we are always working on pushing that and part of our partnership with Mozilla is to say: this is what we really need to make these apps for Autodesk work—to really make that seamless.

So we have a nice circle of partners, with Mozilla and us and with Autodesk and its apps, I see this being the way things are going and the hardware to run this is getting simpler and the barrier to entry is falling–which is what I think people want.

To learn more about OTOY visit them on the web.