Maël Nison

Transcript

Maël Nison - Yarn - 2019 and beyond

>> Hello, again. We have a new speaker on the stage, and it's Maël. Maël worked in various projects from compilers to simulators - emulators, even to toy games engines. Wow.

It's like #jackofalltrades. He is French and prepares amazing pancakes.

We are here for Yarn 2019. And beyond. Please a big round of applause for Maël. [Applause].

>> So you might have seen the talk from Kat just before. It's really interesting, and you should keep it in mind while we make this one as there are a few things that might overlap a bit. First, who am I? My name is Maël. I used to work at Facebook as part of the JavaScript foundation team.

My job there was to en - do the infrastructure, and I came to contribute to Yarn. As time passed, I started to take more and more responsibilities on the project, and here I am today talking about the work we've been doing for the V2 which is in two small part based on my observation after using Yarn at scale for a few years. Something I like to remember when doing this talk is why does Yarn exist in the first place? How did it get created? It all started with three companies - Facebook, Google, and Expo - which felt something was missing in their infrastructure.

They figured the existing solution didn't meet all their needs in terms of stability, user experience, and so they decided to try something different, to try something new. A different solution with different architectural choices, different trade-offs that would still be compatible with the ecosystem at large. That was a huge bet but we succeeded, and that is how Yarn got created. Since then, we keep this philosophy, think of Yarn as an a laboratory.

Speaking of feels, before talking about what Yarn 2 will bring to the table in 2019 and beyond, we must first go quickly over what it contributed and more importantly what are the things they did right. It's the important to know because that's how we can ensure that we won't lose what makes our core values when we make the transition from the v1 to the v2.

This is an open question, by the way. I would like to hear what you think Yarn did right. Please come and see me after the talk and share the stories. Ever since Yarn got created, something we deem very important was stability.

We want JavaScript project to be stable. We want you to trust your project, and we are ready to go the extra mile to achieve this goal. This is about teaching good practices and letting you know when something is wrong before it can cause any article.

We work to remove unsafe patterns from the equation as much as possible. Our log files ensure that the install dependencies are always the same, our install strategy, plug 'n' play, prevent your application breaking at deploy time, and our offline mirror ensures that the network will never be a problem in deployment. Speaking of the network, this is one thing in particular we worked really well for us. At Facebook, we barely even noticed when the registry was down. In fact, we often learned about it through Twitter, really.

Despite the offline mirror being one of our less-known features, it might be one of the most important ones in my mind. You should definitely check it out. If you don't, it's fine, because it will soon become unmissable, starting from the v2 where it will be enabled by default. Going back to the things we did right, work spaces are high on the list.

If you don't know them pretty well, they allow you to split your applications into sub projects that can live independently from each other while style being able to cross-reference each other. Ing see them as a declarative Yarn link. For example, you can see here we've been able to split our huge blob of dependencies into smaller pieces, one for the back end and one for the front-end.

Spaces came to the community, but as we noticed how much they improve our own product, we decided to make it a core feature in Yarn. That was a great idea. Given the success they got, we will be double ing down our efforts to make them first-class citizens, as you will soon see. Finally, the last core value that we definitely should mention is the overall user experience. It doesn't seem of, but it's actually a big reason why our users say they use Yarn.

The small things, like the run key while not being native, or the merge conflicts automatically being resolved. Those things aren't the big shiny things you typically that are on our side, but they are the comments that you use every day, the ones that shape your work flow, and that is something we keep in mind when working on the v2. Okay, so now we have to talk a bit about the elephant in the room: why are we even talking about a new major Yarn release today? About half of you use Yarn in your life with great success, so how would a new major release affect you? Do we really have something significant to bring to the table? If so, what? Something to realise is that Yarn got created in 2016. Since then, a lot of things happened.

For one, we released a lot of major features. Work spaces in 2017, plug 'n' play in 2018, zip loading is coming in 2019, and we will go to that later.

Our architecture needed a refresh to stay up to date with all those new concepts that didn't even exist when Yarn got created, and to support the new work flow they unlock. Additionally, Yarn got initially created as an npm clone. As such, we ... values behaviours that won't mesh well with the user experience we provide which is something we really care about. For those technical reasons, we needed to build something new, to build a new modern foundation more suited to what we had in mind.

That's not the only reason. The main reason in my mind is that we wanted to give you more power. We wanted to make it easier for you to contribute to Yarn. To write your own features, to fix what blocks you, not to have to rely on us.

After all, Yarn is a community project. We don't want to work for you, we want to work with you. So our architecture needed to reflect that, and for this reason, we decided to make it very modular so that you would be able to write your own plugins.

In fact, most of the features in Yarn 2 are actually plugins that got aggregated together in a single binary. In the same vein, we moved our codebase to a more familiar stack-typescript, and we hoped that, by doing this, it will decrease the time needed for you to understand the pieces, how they fit together in such a way that you will be able to help us maintain this critical part of your infrastructure, and it's scope to ... enough about the abstract talk, let's delve a bit into what the next major version of Yarn will concretely bringing to the table. The first thing you will notice is that we revamped the output.

What Yarn prints is how it communicates with you, and we wanted to make it easier for you to figure out what happens and highlight the most important information. This translates into important way: first, we know print message codes that can be Googled for more detail on what a specific message means, and, if if the in the case of errors, how to solve them.

Secondly, we are now using colours with a specific purpose to highlight the dynamic parts of our messages. It's one of those very tiny things that I mention that happened to matter a lot when you work with Yarn every day. By using the colours at the right place, we aim to decrease the - you won't have to fight to understand it. The v2 also comes with new comments.

You might have heard about Yarn creating the v1 which allowed you to download and run any package prefixed by the word "create". We decided to expand it and build Yarn DLX.

That very similar to its sibling and PX can install packages from the registry. It allows you to run 1&1-off strips without having to revert your changes after the fact. Let's see a quick demonstration of it. So here, we are a JavaScript file which is webpack configuration, and we will want to minify it. Usually, we would have to add the minimum fire filter into our package at JSON and then remove it - but by using DLX, you just have to run Yarn DLX and the tame of the file, and Yarn will download everything's that is needed and run it locally.

But even better, since we are using plug 'n' play, it means that we don't have to create ... more, and DLX at the second invocation will be just as fast as it can be because we don't have to fetch the package again. If I run it again, it will be super fast. So that is for newcomers.

We have other ones. We've also improved the existing ones. Various changes were made to make it easier to work with mono repos or fix long-standing issues peeve.

Yarn hub can upgrade a package across all your at once. Yarn Link is able to properly resolve peer dependencies even when you link an external package into your current product. Yarn Ad accept an interactive flag that be instructs Yarn to ask you for advice in various situations. And it's not only a toy. The interactive mode, for example, allows you to quickly reuse a version of a package that another one of your work spaces is already using.

You don't need to grab into your packager JSON any more to find that out. Speaking about - we are introducing constraints. Wouldn't it be nice if you could just check that your project obeys some rules? Like linting, some kind of ES lint, but for package.json. We thought so too.

One of our newest conditions are constraints which are basically lints rules and written in prologue. Prologue makes them extremely small and powerful.

In the example, you can see our work spaces forbids Babel listed as an a regular dependency. In fact, with only two more lines, we could extend this to automatically move the dependency within the dev dependency field. Constraints are very powerful, and they have many usages. You can see some of the screen, and we would provide some recipes in our documentation to get started, but you will quickly see that it is super easy to write new ones that are directly adapted to your use case.

We can see them in practice a bit. So here, I have project with two work spaces: my front-end and my back-end. Both my front-end and back-end are depending on low dash except both are TypeScript and should depend on - it would be nice if there was a way to detect all of those problems and fix them as best as we can. Yarn as a newcomer, Yarn constraints check, when Yarn constraints check is called, it will validate the project against the defined rules, so, in this case, you can see that it has been able to detect that back-end and front-end must depend on TS ...

and they depend on conflicting versions of lodash. If I run Yarn constraints fix, Yarn will automatically try to fix all the problems that it detects and can fix. In this case, for example, you can see that it is able to add TSLIB to the dependencies.

It hasn't been able to correct the mismatch in lodash because it's ambiguous. It doesn't know which one you want to use. In order to fix those kind of issues, you have the good old Yarn Add. If you run that with interactive mode, Yarn will ask you which version of Yarn you want to use amongst multiple possible ones.

In this situation, it is smart enough to tell you that lodash is already used in two work spaces. Which one do you want to use? By using this feature, you don't have to think about how should I fix the problem? You just have to validate and Yarn will automatically continue and do its stuff.

You will see that all the ... have been properly fixed and everything has been applied. Now, something else we can do with constraint, because of how powerful they are, is using them as a query. By using Yarn constraints Query, you get to ask Yarn what do you know about my project? So, for example, in this case, by using one simple query, I'm able to ask Yarn to tell me all the dependencies and all the ranges in all of my work spaces.

But I want - I can be more precise if I want to. I have to specify one of the parameters and Yarn will be able to filter those.

That is the power of the constraints. Another feature that we are working on, and that is maybe the biggest feature that we will ship, is zero install. Zero install is not actually a feeling, it's more of a concept, and it's more aimed for applications than libraries, even though they work for both. The idea is never run Yarn Install again. That might remind you of something! The idea behind this is that if a code needs to run, this code will fail, even though we try our best to make a package manager that never fails, from time to time, we're not free of making a bug, or making a mistake, and is it will crash at some point, or if it is not us, maybe it's - maybe your file system will be run - that actually happened to me before.

So, in order for something to work every time, it doesn't - it must not run. So, with the zero install, the concept is simple. You just add all your Yarn files into the repository. And then you don't need to run Yarn install because everything will be there.

I know what you think that committing the directory bad. With plug 'n' play, we don't have a directory any more and plug 'n' play is the default in v2. The problem can committing non-models was that non-models contain 20,000 text files.

You may remember that I mentioned zip loading before. With Yarn 2, instead of generating an unpacked cache, we just download each package that you use, and we store them within your project, or elsewhere if you don't want to. Which means that, for each package that you use, you will have exactly one archive. And that is peanuts for it to support. We discussed it with GitHub before breaking their infra, and they told us it would be fine.

So what does zero install bring to the table? Kat mentioned it improves the developer experience by a large factor. We don't have to run Yarn style between git pull and checkout. Peer reviewers know which packet has been - they look at the file's change and see what has been added into the project. It is faster, simpler, and more stable for deployments, because again you don't have to run Yarn at all, so there is no risk of Yarn crashing in production.

Additionally, you will always have exactly the same state on production as you have on development, because that is the guarantee we provide. If it works locally, it will work in production.

Additionally, it allows you to skip the CI configuration step. So, in this case, we can see a quick demonstration of zero install, so you can see that we have a small project that is using webpack 3. We would like to update to webpack 4, so we create a new branch, and here, we will just run Yarn add webpack four. Yarn is now running, downloading be the dependencies as it always did. If we do after that git status, we will see something very interesting.

But first, yes, you can see that webpack got updated correctly, of course. And now if we run GIT status, you can see something interesting. Instead of having ... we have as I mentioned one zip for each package.

If we commit all this, then we have everything we need in order to run our applications. Let's say that suddenly a project manager comes into your room and says we need to fix this problem on master, and it needs to be done now, so here you need to give a switch quickly into the webpack 3 branch, make your child, and commit it. But just after switching, you don't have to run Yarn install because everything is like it should be.

Your project is always in a good state. Once you're done, you just have to switch back on your webpack 4 and everything will work just the same. By having this work flow and the simplifying context switches, it makes you more productive because you can work on multiple features at any point in time. In fact, it's super useful for me and a Yarn maintainer, because I can quickly check out the pulls submitted to Yarn and fix them myself, something that was before very painful because I had to be careful in which state my project - it's now super easy, in fact, Yarn to itself is using zero install, so, for example, here, you can find that I'm - so it's a bit slower than it would be if you don't have those tar balls inside your repository but it's faster than it would be if you were to clone your repository and then run Yarn install. In the end, there's a kind of balance here.

And, yes, there is a balance here. So the clone is about to finish. And starting from now, you can use any comment that you would typically use. For example, here, I can run the unit test for Yarn.

I don't have to run Yarn install. Everything is already there.

It makes it much easier for our contributors to jump into the project. They just have to clone and run the counter. They don't think more about it. Let's recap a bit. We have an improved login.

We have new comments. We have improved comments. We have added packages and linting. We have removed the need to run Yarn Install.

That has to be worth a major bump, right? That is the tip of the iceberg. This is only a 30-minute talk, and there is only so much that we can discuss here. Fortunately, we have a more comprehensive change log on our GitHub repository.

I encourage you to look at it to see what is coming. The v2 is flagged as experimental but it's starting to become stable enough for your future projects. The official release is still to be determined but we are thinking somewhere in the second half of 2019 once we've ensured the transition path is as smooth as possible. A big thanks should go to our contributors, of course, because we have had the pleasure to welcome various you in people who simple stepped into the project and improved what the feature they liked. I said Yarn is a community project at the beginning of my talk.

And I really meant it. We are always looking for new contributors. More than that, we are ready to go the extra mile and help you get started. Yarn is a huge project with a huge reach, a huge impact but we still aim to make it approachable by anyone.

Plus we have cool T-shirts for the v2 if you contribute to the project, of course! So I will now leave you. I hope you enjoyed this talk.

Please come ask your questions when you see me in the hall. I will be thrilled to share more about the work we do. Thank you, and have a good evening. [Cheering and applause].

>> Thank you so much, Maël. The next talk will be starting in ten minutes.