Joyee Cheung

Transcript

Hi, everybody. Nice to see you. You may have opinion at JSConf Europe last year. And this morning, there was a great introduction to workers.

It is exciting times again for animations in serverside JavaScript run times. So, here I am to talk about our recent effort to bring a new set of APIs into Node Core.

This may be an opportunity for us to really think about the JavaScript API service in the ecosystem and create a more universal developer experience across the different platforms. So, let's finish this that is right, I'm Joyee, I live in Hangzhou. I worked on Gaia, and I'm a member of the Node steering committee. On Twitter or GitHub, Joyee Cheung handle. Some of you may be aware of this effort of bringing more web APIs into Node Core.

But maybe some folks are just curious because they hear things that are interesting. So, in case I am using some lingoes that only a very few people understood. So, what does it mean when we talk about bringing web APIs into Node Core? of well, we are basically talking about adding browser APIs to the Node.js runtime as buildings. So, you can run them on the server side without using npm install.

So, in this talk I will try to be objective and summarize how far we have come in this journey, where we are, and where we are going. On a teeny tiny side note, Node.js is silent. So, like many others, I will try to use the correct pronunciation in this talk.

Here is a mental picture of how JavaScript in the wild came to be. In the beginning before Node came along there were two kinds of APIs that came with the runtime for JavaScript developers. There were some JavaScript builtins like date, regular submissions, errors that were part of the language. It was created by system 9 and implemented by engines.

There was a host API. And at that time hosts were basically just browsers. These APIs were implemented by the browsers and are more or less run by organizations like W3C.

And then Node was created a started to take off on the server side. Initially it included some of the web APIs that were already present in the browsers. Or like at least some APIs that look like what you'll find in browsers. But Node also introduced several different APIs for doing certain things on the server side like  instead of even target in the browsers. Because the semantics of the equivalent APIs on the browser side may not make sense for servers.

Like, for example, there isn't one DOM tree in this execution context for events to bubble through on the server side. So, it does not really make sense to implement part of the event target specification in Node. So, as time goes by, browsers have developed and synthesized more APIs in the browser to empower web developers. While Node also developed several equivalent APIs on the server side with a didn't design. Because of the popularity of the Node.js runtime, these home-grown APIs gradually became the defacto factors for the server side run times.

But obviously no one likes to remember two sets of APIs for doing basically the same thing. So, over the past few years people have been sending feature requests to Node to bring the two platforms closer together. For now, it usually means making the runtime more compatible with browsers. So, Node did evaluate and implement some of the requests and it is looking into more APIs from the web.

So, we currently have different types of web APIs implemented in Node Core. So, some of them are more localize the with the browsers. And, for instance, we have methods in the browsers, and this is the living standard of the timers in the HTML specification.

In Node we also have a set of timer APIs, that's similar, but the implementation does not strictly follow what the browsers do. For example, it returns an object while browsers usually return a number instead. The other type of web APIs we have in Node Core are the ones implemented with an existing specification in mind and behave mostly the same as the APIs that you will find in browsers. For instance, the new URL implementation in Node Core was developed specifically according to the URL standard.

This means we did look at the spec test when we implemented them. And we also have tests.

So, how do we know how close our implementations are to the ones in the browser? We run a subset of the web platform tests which are tests which are on browsers and other implementations. So, to see the current status of the performance in Node, you can look at the status files under tests, the status folder of the Node project. There are a few pretty selfexplanatory JSON files documenting the web standards implemented in Node.

So, this one is not technically a web API addition, but I wanted to mention that there is now another type of web standards implemented in the core. We have collaborated with the what is the HTML specification and the first implementation of JSON modules has landed in the master branch as part of our experimental ECMAScript module. So, Node implemented first.

So, here we have the web APIs implemented in Node Core that can be alternatives to certain home-grown Node build teams. Legacy methods under the URL and the string built in module can be replaced by the new URL and the URL search params classes. And searching and coding and decoding methods of buffer and string decoder can be replaced by text decoder and text decoder as well.

And there is the performance API which, for example, can be used to replace certain timing methods in the process object. In addition, we also have some APIs that are not exactly replacements for existing Node APIs. But are similar enough in many use cases that it is reasonable to consider them as alternatives. Be sure to read the documentation if you want to rewrite your code with these APIs. For example, cue microtask can be used to cue a microtask which will be run asynchronously which will be used to replace the process if you do not have strict requirements about the timing you want your task to be run.

There is also web worker which spawns threads in Node. And it may be used to replace child processes if you are only looking for a way to upload and you don't necessarily need processes.

So, as of Node 12, we have several web API implementations that are now stable and can be used in production. Those are listed on the left here. And these are available as globals. So, you don't have to require a builtin to get hold of them. These are also covered by the web platform task in Node Core.

There may be some additional extensions in these APIs. There's some minor unspecified behavior differences. But at least those are fairly limited, and we are aware of them.

We also have a few APIs that are still experimental which means there may still be breaking changes in the future. These are currently placed under builtin modules and not on the global object yet. The current experimental web APIs that we have are workers and performance time being API.

But they also differ significantly from what you would get from the browsers. And we do not run web platform tests for them yet. So, watch out. So, there is also an implementation of WebAssembly, the JavaScript API of WebAssembly that we get for free from V8. But the web API of it has not been implemented yet.

Other than existing APIs, there are a bunch of others that are hunting the issue tracker. Oops. There are still under active discussion. They're web strings which are the foundation of web specifications. But then we already have Node strings in Core which are also the foundation of many existing Node Core APIs.

In case you didn't know, there are many different types of strings in Node. There's string number one. Which has several issues.

So, they now introduce string number two. But then they also had some issues. And then actually introduce string number three which still had issues. So, there's now a new implementation of strings called Bob which is under development for some time. And will hopefully solve all our problems.

Anyway, this is not a string talk. You can read the documentation if you want to learn more about them. So, with all the strings, you can imagine how complicated it would be to bring yet another string into Node Core.

And there is also Fetch. Which is probably the most requested web API in Node Core. We just had a session about it this week at the Open JS Coverage Summit in Berlin and there is now a new work in progress pull request to bring it fresh into Node Core.

Yay!

[ Applause ]

So, we have to talk about the history of these web APIs in Node and what may be coming next. So, why exactly are we doing this? One obvious run is with a common API surface, there is less cognitive burden for developers. We could their documentation, tutorials, tooling. Instead of developing and maintaining a separate set of educational resources.

This is especially important for just beginners. At the moment they have to choose between Web APIs and Node APIs when they are just getting started. With a common API service, beginners can be less distracted learning about basics. There are still differences between the two platforms.

But it will be less intimidating when they already learned a bit more about these APIs. So, another reason for web APIs into Node core, we have more containers compared to more npm modules.

This is open source and it's natural that contributors come and go. In Node, even when the existing maintainers of a specific builtin module shop less often, we have an open governance and an effective nomination and onboarding process to bring new contributors into the team. Compared to regular npm modules, this kind of maintenance story fits better with the Web APIs that are designed as builtins for the host environment. So, if you have been paying some attention to this topic, you may be aware that it takes an extraordinary amount of time for this API additions to be accepted into Node Core.

Here I will lay out some of the reasons why there are several requests that keep showing up in the issue checker but have never really gone anywhere. So, everything I'm going to talk about later are in the context of the consensusseeking model of Node Core. So, Node Core is operated under the consensus from over 100 core collaborators. These are contributors who have write access to the repository. For every technical decision, any one of these 100  more than 100 Node Core collaborators creates an objection.

If consensus cannot be reached within the collaborators, it may come down to a vote among the members of the tech committee who are a subset of the collaborators. But we usually try to avoid voting. We also take community feedback into account even if it's not from someone who has committed into Node before.

So, here are the common arguments against adding Web APIs into Node Core. There is still, to some extent, a small core philosophy within Node Core. It's about providing only the basics functionalities in Core and empower users to implement user modules instead of building our own opinionated APIs that may become a compatibility or maintenance burden.

This philosophy has been broken several times in the past. But at least as far as Web API goes, we are still mostly just trying to expose the existing functionalities through a different API services. This is not exactly an idea that's welcomed by everybody either. Especially when the web API may also lack features that do not make sense for browsers.

Oh, yeah. And like sometimes they may be necessary for servers.

So, one alternative to adding these APIs in Node is to release them as official modules. Theoretically for modules that are maintained under the Node organization, it will be possible to have a maintenance story similar to the one that Node query has. But some may also argue that it is easier to optimize if it's done in Core because it can use certain internal APIs. Or sometimes it may just not be technically possible to implement something without access to internals.

A part of the philosophical concerns, apart from those, there are concerns about the design of the web APIs. They're more than just a bunch of interfaces. Behind the Design of these APIs, there is just a very different context. For instance, the browser has a very different security model.

When you fetch an API endpoint, for example, at a script and you want to send credentials like cookies along with the request. Fetch as implemented according to the specifications should follow the crossorigin resource protocol and check the access control allowed header in the HTTP response before invoking that with the data. So, that scripts cannot  untrusted scripts cannot steal your cookies when the server is not aware of them.

Somewhere in Node, there isn't really a concept of origins. At least for now. So, these are loaded from your local file system and are just trusted by default within current security model of Node. If you perform the request using the existing HTTP request method in Node, this security policy would have to be implemented by the users. Well, if they do want them.

It is not impossible to implement something like this in Node. But this may just be confusing for most users because then we'll have two conflicting security models in Node. So, when we look at the Fetch specification, the interface itself is just the tip of the iceberg. There are many implications under the surface of the API like course origin, consent and security policy, caching interop with service workers and like potential management.

So, some of this may make sense for now. Some of them don't. If we only implement part of the API that we think makes sense for Node, we may confuse our users more because this will bring another kind of platform compatibility headache to everyone.

There is also another type of concern. The ecosystem has come to depend on a lot of existing infrastructure in Node. And this may differ significantly from their web equivalence. For example, we have different interfaces to do stringing and to emit events. As usual, there are even more differences in the underlying design of these infrastructures.

When implementing web APIs in Node, we also need to decide whether we want to introduce the web infrastructure into Node Core or base the higher-level APIs on the existing Node infrastructure or just use some instruction layer instead. We will also need to figure out the interop between these abstractions for other existing modules. And this work just takes a lot of time.

So, there are a lot of open questions to answer. A lot of decisions to be made. And this, you know, just takes time in the current  this is the model in Node Core. But how do new Web API actually ended up being added to Node Core these days? It usually starts with a feature request opened in the Node.js/node repository.

To actually make progress on the request, someone, or some group of people, need to step up and start a prototype. They do not need to be Node collaborators. It could be anyone who are willing to invest their time in the work. Typically, they would create a fork.

Either as a personal fork or as a fork under the Node organization. And they will hack together an initial invitation and then send a pull request back to the main repository against the master branch.

At these stages there may be objections coming from collaborators or the community. Sometimes the proposal just gets stalled and closed. For example, this is the current status of the feature request for web crypto. Sometimes, but rarely, there are no objections.

Or these objections get resolved either through discussion or through voting. Either way, someone must be interested enough in this feature to get it through the consensusseeking model. Then the initial implementation may get merged into the master branch.

So, once the feature is merged in master, we'll start iterating on this. Fixing bugs, optimizing. At a certain point, depending how visible it is, we may start with the subset of web platform tests with it and collaborate with the web platform tests upstring as well as the authors to improve this spec and the test suite. It is also typical to eventually expose these interfaces to the global object. But this is depending on various complexities.

At this phase, the feature is still in experimental status. Depending on the release schedule, this feature may be released to users while it is experimental. And it may get updated in the release branch with patches back ported and the master branch. So, this is the current status of web workers and the performance timing API.

Eventually this feature would be, or it is supposed to be, moved out of experiment and becomes a stable feature. This is the current status of the URL and encoding implementation. So, here's a quick summary. Node query APIs have diverged from the web APIs because they were designed for very different use cases.

But more and more web APIs have now been added into Node Core. We have work would through the existing web APIs in Node Core and their status.

Then we looked into the challenges and the workflow of bringing for APIs into Node Core in the future. So, finally, we are starting an open standards initiative in Node to collaborate more with senders and other implementations. As we said earlier, there are many questions to answer and it takes a lot of energy to find the answers. If you are interested, please get involved. Thank you.

[ Applause ]