MonoRepository build tools

Mono-Repository -> M.R. -> Mr. -> Mister

mister is a set of commands that can super-simplify running tasks on a monorepository.

Mister...

• is a set of commands to simplify running tasks on packages in a monorepository.
• is not a build system or toolchain.
• does not require initialization, changes to your package.json.
• does not create symlinks, or magic fake relative packages with relative path imports, or anything else.
• may create one single file, .mister/build.json, to help prevent re-building packages that have not changed.

Finally, mister is not necessary. Everything mister does, you could do yourself, albeit much more tediously, if you use a similar folder structure.

mister may not be the right tool for you. There are plenty of monorepo systems available, from yarn workspaces, to lerna, and bit. Consult your programmers if mister runs longer than 4 hours.

What's so great about it

Mister tries to determine whether or not your task or command needs to be run. In general, if you build a package successfully, make no changes, and attempt to build it again, mister will skip that process. Tasks prefixed with an exclamation mark will skip this cache check and be run anyways- good for things like uni tests, integration tests, or acceptance testing.

mister do-all build !test
+[@scope/packace1:task:build] is up to date
+[@scope/package2:task:build] is up to date
+[@scope/package3:task:build] is up to date
-[@scope/package4:task:build] is out to date

mister do-all build !test
+[@scope/packace1:task:build] is up to date
+[@scope/package2:task:build] is up to date
+[@scope/package3:task:build] is up to date
+[@scope/package4:task:build] is up to date

This currently works by testing the timestamps of all the files in a package, minus those that are filtered through .gitignore (ideally, your source files). If you have a build task that generates files, and then you delete them yourself, mister won't try to rebuild them.

So how do I use it?

Mister's only real requirement is that your monorepository packages are in the folder packages/node_modules (although you can change it with the command line switch --package-prefix)

.
├── .mister             # You should .gitignore this if it gets created.
│   └── cache.json      # This is the file MR uses to keep track of build timestamps and dependencies.
├── node_modules        # This is where you put your external dependencies.
│   ├── express
│   └── underscore
└── packages
    └── node_modules    # These are the packages of your monorepository.
        ├── @scope
        │   ├── package1
        │   └── package2
        ├── package3
        └── package4

Now, you can have mister do some things for you:

mister do @scope/package4 --tasks clean !test build

Or just do some things on all your packages, if you keep common task names:

mister do-all build !test

How do I make package bundles? (TGZ)

mister comes with a pack command, which honors bundledDependencies.

mister pack @scope/package2 package1

Note that due to npm pack not following symlinks or doing any module resolution at all (it assumes everything is in relative node_modules), mister will perform npm install --production for all local package dependencies, and rename package-local dependency versions in package.json with the relative file path to the dependency tarball. Upon completion, all changes are reverted (the only side effects are tarballs in ./dist).

This process can be time-consuming.

mister pack is a primitive that only creates packages. You will need to build them first e.g. mister do packageName --tasks build && mister pack packageName

I want to share code between AWS functions easily!

mister comes with a zip command, which honors bundledDependencies. This is why I made mister:

tree
.
├── .mister
│   └── cache.json
├── node_modules
│   ├── redis
│   └── pg-promise
└── packages
    └── node_modules    # These are the packages of your monorepository.
        ├── @lambda
        │   ├── someLambdaFunction
        │   └── someOtherLambdaFunction
        ├── @shared
        │   ├── some-useful-function
        │   └── some-shared-utility

mister do-all build !test && mister zip

tree
.
├── .mister
│   └── cache.json
├── dist
│   ├── lambda-some-lambda-function-1.0.0.zip # Here is my first function
│   └── some-shared-utility-1.0.0.zip         # here is my second function
├── node_modules
( continues... )

Both functions have redis, pg-promise, @shared/some-useful-function, and @shared/some-shared-utility in bundledDependencies with all necessary dependencies, and are bundled in a zip with package.json at the top level, ready for direct deployment to AWS using your method of choice.

Reference

mister do [packages...] --tasks [tasks...] 

mister do-all [tasks...] 

mister pack [packages...] 

mister zip [packages...] 

Any other recommendations?

To take advantage of not having multiple dependency versions, you'll want to install all your dependencies at the top level (but you will still need to reference those dependencies in the package.json files of your packages). But you don't have to do that if you don't want to. Managing your dependencies is up to you 😉

How does this even work?

It helps to read up on how node resolves module names. The TLDR version is that, given require('module-name'), node will essentially walk up the filesystem until it discoveres a path matching node_modules/module-name.

Using package/node_modules to structure your monorepository means you can leverage this to require('your-monorepository-package') from inside of any other monorepository package, and also let you require('external-dependency') installed to your top-level node_modules folder.

Caveats

node_modules is Ignored

Several tools, like ts-node, by default ignore node_modules, which means they do not work as expected in packages/node_modules. If you have written tests in typescript, you will need to pass an updated TS_NODE_IGNORE environment variable either through your top-level script or your package-level script:

{
    "scripts": {
        "test": "cross-env TS_NODE_IGNORE=\"/(?