Marley Rafson

Transcript

Thank you. And hello. Today I'll be talking to you about the case for augmented reality on the web. I'm Marley Rafson.

I'm a software engineer at Google on the web XR team. My Twitter handle is mprafson.

Love to talk letter. The two things today with the web and augmented reality. I believe that the combination of these two things, the whole is greater than the sum of its parts situation. I think that the web will gain from augmented reality, and things that augmented reality will certainly gain from the web.

So, to begin, let's start by getting everyone on the same page and let's define augmented reality. It's the mixing of computergenerated content with the real world. To me, we can now add contextualized information into the world around us. So, for example, you're trying to make a coffee at the coffee machine, and you have the instructions overlaid on top of the coffee machine.

And implication of augmented reality is you can interact with 3D content in a 3D space. So, the canonical example would be you're trying to buy a piece of furniture and you're able to put it into your room before you buy it.

Let's look at what augmented reality looks like today. Today this is smartphone AR. This is run using a smart phone and basically has some understanding of the world around it allowing virtual content to interact with that world too. We have augmented reality on headsets.

Your hands would be free, and you would be free to interact with the content using your hands. And the third example that I think is less talked about, but I find compelling is audiobased augmented reality.

You're seeing Bose frames. It's a pair of sunglasses that have speakers that shoot audio into your ears. With this example, with audio AR, you would have that experience where you're at a coffee machine and you don't know what to do. But instead of seeing the information, it's read off to you.

And it I would also like to say if you hear me say AR, that's just short for augmented reality. So, this talk, as I mentioned, it's also about the web. Let's talk about where those two things intersect. That's what we'll call the immersive web.

And the immersive web is the use of augmented reality, virtual reality and other 3D technologies on the web. It's this whole spectrum of things.

It can be more augmented reality than virtual reality and doesn't quite matter. And you'll hear me say XR. And that really just stands for wild card reality. So, it could be virtual, it could be augmented reality and some combination of the two.

So, today what the immersive web looks like is something like this. We have seen progressive enhancements of sites we use every day. The left two examples, Facebook 3D posts and they're incorporated directly into the newsfeed. So, that first example is a 3D model that you can interact with on the newsfeed.

The second is using portrait mode depth information to delightfully play with these photos with a bit of depth added. And the third example is from Wikipedia who about a year ago started supporting 3D models in their articles.

So, if you're not a large company and you don't have a whole engineering team dedicated to building this model viewing experience, there's a solution to that now. So, recently Google, a team at Google open sourced a web component for viewing models. It's called Model viewer. It's fantastic. Using HTML, you can add into your side around it.

That's an example with the astronaut which we know and love. And something that's great about the modelviewer team is they're thinking about accessibility early. So, they've already incorporated alt text into the component.

So, let's say you have these 3D models on your website. How do you actually get it into augmented reality? So, right now what we've seen is implementations of native system viewers that allow for augmented reality. This is not running in a web browser, but this is slick and tight integration with the web. So, you would have this search result, so it's integrated into search with Google. So, you can tap the button and then it intents into this native viewer and then you can place it in augmented reality is the same exists on iOS and it's similarly intense.

And so, what I just said there was that we're using native viewers to do augmented reality. But we would like to do that in the browser itself. That's where the WebXR device API comes in. It exposes lowlevel sensory information. So, the camera pose, if you can place an object on the floor.

And that's under development. It's open so we can all contribute to it. And you can find that on GitHub.

And with those examples that I just shared, that's all talking about bringing immersive technologies into the web. But we can also bring the web into immersive technologies. Two examples of XR browsers. And you will see the Helio browser placing images into the world around you and then Firefox Reality, which is a VR browser.

So, I think the immersive web today is fantastic. I think it's adding information, it's making it more delightful to browse the web. But when we're developing these technologies, it's important to see where we're going next and to start thinking about that earlier rather than later. So, when I think about a future of augmented reality, I start by thinking about how can this assist me? So, I think about maybe I'm leaving work one day and I'm wearing these lightweight AR goggles, I'm walking down the street.

I'm looking for dinner. So, as I look at the restaurants on the street, I'm seeing Yelp data populating with the stars rating and the types of menu.

And maybe I also want to see photos of the food. So, Instagram information's coming too. And I like Pokemon. So, maybe while I'm doing all of this, I'm catching Pokemon, playing Pokemon Go. And when I get to the train station, I'll have information populated for me telling me which train to take so I can get home fastest.

And as I mentioned earlier, I probably don't want to be restricted to doing this on just one headset. Maybe I want headphones another day from a different company. Who knows? Maybe I just want to use my smartphone. So, when I'm thinking of this vision of augmented reality going forward, I start to see some interesting things and patterns that I would like to call out here.

So, the first is that is inherently uses web content. I want access to all the things that I already use when I'm browsing the web. Another thing is it's this really interesting composition of different types of information. So, that could be models or 2D pages. And we're going to have to interact with them in a way that's esthetic and delightful and still makes me want to use it.

And onethird trend that I start to see it we're really interacting with the world around us. I could be places information on to a building or on to a tree or on passing by cars. And so, when I start to think about this vision that I want and where we are today, which is mostly placing 3D models, I think there is a lot of work to go.

And where I see a lot of that work coming into play is with a user agent. So, the definition from W3C of a user agent is a user agent is any software that retrieves, renders and facilitates end user interaction with web content. So, today that's usually just your browser. So, you get a lot of things when you're browsing using any of the modern browsers.

It could be Chrome, it could be Firefox. It helps you do safe browsing. It renders the HTML pages for you and things like that.

So, to understand how a user agent will play out with this added dimension for augmented reality, let's start to break down this definition of user agent. So, I'll go in reverse order. Start with a user agent facilitates end user interaction with web content.

So, what I believe is that the user should always be in control of how information is presented to them. So, on a modern web browser, that's preferences like default text sizing. That's contrast ratios. And Chrome extensions and things like that.

But when you think about augmented reality, we have this whole new dimension. It's really immersive. It's, you know? And so, we need to start to think about other ways that we want to have safe browsing and we want the user to be in control.

One example, think about physical proximity. You have content that's being rendered possibly close to you and I should be able to say I'm not comfortable with anything rendering closer than 5 meters away from me. Or I think it's compelling to think about this in terms of sound. I probably would never want to allow someone to whisper into my ear as I'm walking down the street.

so, a lot of these considerations come into play when we're talking about augmented reality. And in order to accommodate these things for the years, we're going to need an intervention point for the information.

So, the user agent knows what's being asked of it to do this. If we think of declarative content that we know and love, that would be HTML and CSS. I see this future where there's declarative content that you're able to add into the world around you so you could specify things like I would like float left and pin to a building. Or you could say, I want the depth to be 5 meters from the user.

But it's up to the user agent to decide whether or not it can actually honor that request. And it's in that way that it can advocate for the user. So, another thing that declarative content offers is that is provides semantic understanding for the user agent. So, what that means is that the user agent has a view into what's being asking to rendered or presented to the use. I find this example extremely lie lighting of what I'm talking about.

If we look at the left side, you're seeing the web. If given this canvas to render the object like on the web today, given an XY location, you can only see the color. I see a black pixel, a white pixel, a gray pixel. But nothing more than that.

But with semantic understanding of what's actually in the scene, we have a much more rich understanding of the model the and the content that we're placing. So, if you look at the right side, you can see I'm looking at the windshield. And it has a material that's class. And it's transparent.

You can also see things like the tire and stuff like that. And so, you can imagine with screenreaders or accessible or just translating this into soundbased AR, you can actually talk about what the user is seeing. So, you could say I see a van with black tires and white car metal paint.

So, the next part of the user agent that I'll talk about is rendering. So, the user agent is response for rendering to the browser. So, modern web rendering emergencies are amazing. They're fantastic.

That's blank, echo, WebKit, things like that. When you think about what they asked to do, it's mostly 2D and it's textbased.

They're fantastic at rendering pages like the verge at 60 frames per second. And some of you might know and some of you might not, but the web already does performant 3D rendering. That's great and provided to the user through a Canvas element and uses WebGL to create those graphics. What that might look like today is this, this is not augmented or virtual reality. But it's just rendering in a web page using a canvas and WebGL.

So, for the rendering needs of augmented reality, thinking back to the vision is this complex composition and layout dynamics of content. So, we're gonna probably need to build a rendering engine that can handle that from the ground up. Whereas the modern rendering engines were optimized for 2D.

And so, even though I just said that we probably want this authorized from the ground up and that we don't want to use Canvas elements, we tried it. So, my team experimented on this internal prototype, air web view. It's a set of libraries that enable AR through native apps and web view. It's a lightweight browse their you can embed into a native app.

The technical understanding of what we've done is that we've web rendered and using that to do than we have tracking, AR Core and ARKit. And we combine that with a natively rendered camera feed.

Let's look at what that looks like. You look at the example, we have a stormtrooper and it's convincingly rendered into the space around you. You need to make sure that the content that's rendered moves at the same time as a native camera feed. Because if it's off, it completely breaks the illusion.

So, we have this transparent web view, it's rendering the content, and this becomes a difficult technical problem. Because if you're familiar, and don't worry if you're not. You have a render loop that's running from the browser. But then you also have a render loop that's running natively, and you also have the update loop for the native tracking.

So, it's figuring out a way to get all of these things working together in a performant way. And the advantages of AR web view is it's lightweight.

It's only 60 kilobytes added to your app. It's crossplatform. So, you write code once and it works on both iOS and Android. It's embeddable into native apps and you can pull down content and code on demand.

And that last part, to highlight the advantages of that is that you can make changes to this augmented reality experience that's independent from your native app code. You don't have to wait for an app release to change your AR experience.

So, one use of AR web view in action was at Google IO this year. If you were there, you would have seen this experience that allowed you to orient yourself once you're on site at the conference. Using augmented reality, you could see which way you might want to go next. And so, this experience is literally just a web page. And these white text that say water station and eats market are just actually HTML elements styled with 3D transforms.

So, again, AR web view, it's a prototype, it's internal. We will a lot of fun and learned a lot with it. So, this last aspect of the user agent that I'll talk about is that the user agent retrieves content on behalf of the user. And this part is so fascinating to me.

So, if you think back to this vision of augmented reality, you're walking down the street. And you look at a restaurant and it knows what you want is the menu information.

And it knows what you want is the Instagram photos. And so, if I was doing this today on my smartphone, I would go to the restaurant. I would see the name. I would type it into maybe Google search, who knows. And then I would access the information that I wanted.

But what we have with augmented reality is this fundamental paradigm shift. Now the user agent is querying on my behalf and should know what I want to see. That's very interesting if in a camerafirst browser environment, the user receives content rather than querying for it. That means that the user gets it over the surfacing. This is an interesting question off the bat.

We can think about modern questions like echo chambers and stuff like that. Manage imagine you're in an augmented reality immersive world and you don't know when there's content that you're not seeing. Or you should have a way to know and have ways to set preferences on what you do want to see.

And I think that there is kind of this pessimistic view of that. But I think there's a lot of advantages of having this intermediary between you and the information. An example that I like to think

Imagine you're a coffee drinker. You wake up, you want to wear your AR glasses on the way to work. You haven't had your glass of coffee, you want the bare minimum. Until you get to the coffee shop, then you are ready to get the other things coming. In that way, the user agent having discretion is an advantage.

And so, as you start to hear these things about augmented reality and maybe this future where everybody is wearing headsets. You might be like me yesterday in Conrad electronics thinking, that's a lot of cameras. So, it's true. A camerafirst future requires a lot of cameras.

And it's not just augmented reality that's facing these things. It would be things like selfdriving cars where they have six censors, eight, a ton of sensors looking at the world around it as they're processing. What that means is we're capturing basic life everything that happens on the sidewalk. Anything that someone wearing one of these headsets can see. And so, as we try to build towards this augmented reality feature, we need to have privacy and surveillance top of mind when thinking about worldscale AR.

So, at face value, I think one of the first things you would think about in the context of privacy is that these cameras are capturing RGB data about the world around you. Which is just a picture. So, what happens with that picture data? Is your device computing it on device? Is it going up to the cloud? Who owns that data on the cloud? Is it going to your private cloud? And then when you think about secondary data. So, that's information that's extracted from this camera image which is how augmented reality works.

You can imagine if you're building meshes around you. You live in an apartment building with a distinct statue outside. You walk out.

Your headset creates a mesh of this statue and someone could know exactly where that statue is and locate you. Even if they don't have the camera data, they can understand that from the derivative information that's used for augmented reality. And what I think, though, is that user  user agents being more powerful is actually an advantage in this scenario. Because if we can move that information away from the developer and closer to a trusted user agent, that's a positive thing. But we'll need something like open source code to actually know that we're being  that we can trust our user agents.

So, to me, augmented reality and the web is an amazing partnership. The web will gain a whole new access for understanding information. It will be more contextualized. It will be ergonomic.

And for augmented reality, the web for sure is an important piece. It will provide the content and I love the webby principles that we use today.

I think as we look forward to a future with augmented reality, we need to keep those webby principles in line. If you are interested in using the resources today, check out the resources. Look at the model viewer and see if adding 3D models increases use and context to your information. And also, start talking to us about the WebXR device API. Everyone's opinion matters and it will be great for the ecosystem overall.

So, to me if the web plus augmented reality is an open ecosystem of rich, contextualized information, then I'm excited and I hope that you all are too.

[ Applause ]