Getting rid of Lerna

Notesnook was not always a monorepo with 10s of subprojects. Everything used to be in its own Git repository. While that often became tiresome, it was also simpler in a way.

In October 2022 when we went open source, our beast of choice for the seemingly complicated task of monorepo management was Lerna (and Nx). We had to make a few concessions, learn some new patterns, give up old habits, but it all worked out...eventually.

What we had not foreseen was the impact it would have on our productivity. The only reason we decided to use Lerna was for its package management in a monorepo structure. At the time it solved the problem of, "How do I install X subproject in Y?" but every day we had to face some new problem requiring us to either work around Lerna or deal with it by changing something else. A few examples:

1. It is not a trivial task to use Lerna with flatpak-builder.
2. Sometimes Lerna would refuse to update the package lockfiles. The solution? Delete all the node_modules directories and run lerna bootstrap.
3. CI would fail because Lerna automatically switched to npm ci which requires the lockfile & package.json to correlate. The solution? Force a lockfile update, either by clearing all the node_modules or deleting the lockfiles.
4. Switching git branches and bootstrapping again was a nightmare. Sometimes dependencies wouldn't install or Lerna would install old versions due to outdated package lockfiles. The solution? Delete node_modules directory and run lerna bootstrap.
5. There's no way to do npm i in a subproject when you are using Lerna. You have to use lerna add.
6. lerna add doesn't work well with git: or file: packages requiring you to unnecessarily publish everything to NPM.
7. There's no lerna remove so the only way to uninstall a package is by removing it from the package.json, but most of the time that also requires deleting the whole node_modules directory.
8. Lerna takes 100 MB after install.

Lerna has forced us to download gigabytes over and over again. Okay, I admit, it's not that bad - there's caching and all but still I shouldn't have to clear out node_modules repeatedly just to get package management to work.

I am aware that Lerna is not only meant to be used for package management. In fact, even Lerna had to acknowledge their broken package management, calling it "Legacy", and encourage users to use the package managers' Workspaces feature instead. From v7 onwards, Lerna no longer provides its users with the ability to bootstrap or install a package through it.

It's good advice when you are starting a new project, but in an old project that's "working" there's no time to deal with all the issues that come with using Workspaces:

1. npm workspaces defaults to hoisting all the packages in a workspace to the root node_modules directory. I have found no way to turn it off. Hoisting creates a boatload of issues which require more tools to fix, which create a new set of issues, ad infinitum.
2. yarn workspaces supports turning off hoisting and was almost perfect, but it failed to install some native Node.js dependencies. I had no time to look into why it broke.
3. pnpm almost worked, but it isn't supported by flatpak-builder and also requires some nasty hacks to work with Metro Bundler — the de facto bundler for React Native projects.
4. bun install doesn't work on Windows.

We needed a tried and tested solution that "just worked". Our requirements:

1. It should be possible to do npm i or yarn add or pnpm install wherever we want.
2. Bootstrapping should be quick and responsive (i.e., we shouldn't have to see dead progress bars for ages).
3. No learning curve. Everything should be obvious, boring, and simple.
4. No hoisting. All packages should be placed in each project's node_modules directory. This is widely supported and very stable.
5. flatpak-builder support.
6. Small & hackable, requiring minimum maintenance.
7. Easy to bootstrap specific scopes (e.g. only bootstrap the desktop app).
8. Reliable cross-platform support.
9. Zero config files.
10. It shouldn't be 100mb in size.

You must be thinking (with a groan), "Here comes another monorepo management tool". You'd be 100% wrong.

Working on Notesnook, I have realized no one wants a simple solution. A 10-page-long feature list is preferred over something that does one thing and does it well — the Unix Philosophy, so to say.

No. I had no time to deal with a new tool that'd break in 100 different ways. I wanted a solution that worked with our existing codebase without too many changes, so I did what I should have done from the beginning:

I wrote a script.

144 lines (including blank lines & comments) and only 3 package dependencies (yargs, fast-glob & listr) for nicer DX, doing only one thing: bootstrapping. Here's how it looks:

What's the benefit? Why go to all this effort? Did it really pay off?

1. Our CI run time decreased by 1 minute per job across the board. That sounds little, but it all adds up when that job runs 60 times a day.
2. We have yet to delete the node_modules directory even once, and it has been a month, almost.
3. We can do npm i anywhere we want without breaking anything.
4. The perception of speed when running npm run bootstrap makes us feel 10x faster.
5. Lockfiles never get outdated or suddenly updated for no reason.
6. It's just npm i doing all the work, so everything works without changes.

Best of all, there's not been one occasion where any of us has screamed with frustration over Lerna getting stuck.

Life is simple again.

Bootstrapping a monorepo is a surprisingly simple task consisting of only 2 steps:

1. Find all the locally linked dependencies of all the subprojects recursively.
2. Go into each linked dependency and run npm i or yarn install.

A bash wizard could probably do this in 3 lines. I won't put here all the code because it's too obvious to warrant a discussion, but you can have a look at the code here.

Conclusion

This is our story. Don't take it as a pitch against using Lerna or any other tool. Lerna serves many other usecases which I haven't (yet) found a need for.

In the end, use whatever works best for you and when that breaks look for the simplest solution instead of the "industry solution" or "trends". After all, the only thing a tool is meant to do is save your time.