Don't use Leiningen to run shell-scripts

Working with various Clojure projects, I noticed one thing that really worries me. I’m talking about developers who add more and more entries into :prep-tasks vector in project.clj file. Please stop it. Every time you want to put there a new task to compile CSS or build a Docker image, take a minute or two to think on that. Let’s discuss the problem.

Lein is a great tool of course, really a piece of art. But its abilities are not unlimited. Remember, its main purpose is to manage a Clojure-driven project. I think, everybody agree with that statement.

The Clojure project you are working on is only a part of a top-level project in business terms. Besides both server and UI sides, your application probably sends emails, pushes notifications, interacts with Blockchain network and does further more things.

According to GitHub statistics, you may have even 99% percent of code written with Clojure. But still, there is definitely something that is beyond it. Thus, please do not use Lein for those tasks that do no have any relation to Clojure code.

Recently, I faced a situation when running REPL caused building Ethereum smart contracts first. That step assumes you have installed software of proper versions and configured paths, text configs, etc. Compiling them was a really resource- and time-consuming duty.

The next step was to compile CSS sources. Again, it required installing Ruby, less and wait for some time.

Remember, I did’n want all of this to happen. All I wanted to do is to connect to REPL from Emacs and debug one tiny function. Needless to say, I just commented those tasks in project file.

Again: your Clojure project should know nothing about compiling CSS, building email templates, fetching anything from the network, querying Ethereum, building Docker image or whatever else. Especially when we talking about running REPL. It should run without any troubles at any time.

That’s why I’m strictly against using lein-shell plugin or something similar that lets you run shell commands from Lein.

Of course, building a project requires passing through a list of particular steps that were partially mentioned above. What should we use for that? Shell-scripts? Any modern Javascript task runner? No, just Make utility that has been here for ages.

“Put that gun down and let me explain.” (c)

I know Make is an ancient tool that came from rough C/C++ times. Those days, developers knew quite few about a pipeline, CI or methodologies in general. Almost every single programming language nowadays offers a task runner that takes modern requirements into account. But still, for a set of tasks that might be run upon your project, there is nothing better then a Makefile at the top of file structure. And here is why.

Make utility is a binary tool that does not force you to install a new version of Node.js, Python or Ruby. Probably, it’s already installed on your computer since most of Linux distributions have it out from the box. It works perfectly of various systems. I have never faced any troubles with it switching between Linux and Mac.

Such modern shells as zsh support auto-completing make targets when pressing <TAB> character after “make”. That really saves time especially when you use prefixes. Say, in my pet project, besides Clojure-related targets, I’ve got a bunch of commands to operate on Docker images. These are docker-run, docker-build, docker-compose-up/down and more. Each of them takes 60 to 100 characters so it’s impossible to remember them. So in terminal, I type make doc<TAB> and see a short subset of Docker-related stuff.

I consider any Makefile as not just a list of commands but rather a knowledge base of your project. The more the project develops, the more operations you need to perform over it. Where to store all the those commands? In your wiki? Nobody reads it. A better solution would be to keep them as close to the code as it possible.

Make utility runs extremely fast whereas lein needs about five seconds to boot up. That’s pretty long time, really. Imagine each command line tool hangs for a couple of seconds before it runs. That would be a hell on your computer. A situation that makes me angry is when I mistype in a long lein command like lein migratus craete user-updated, wait for five seconds and see an error message saying there is no craete subcommand in migratus. With make utility, there is no an option for such things.

One of my favorite features is to ask a user for prompt when typing names, passwords or any other sensitive data. Here is an example of how usually I create a new SQL migration:

create-migration:
    @read -p "Enter migration name: " migration \
    && lein migratus create $$migration

When I type make crea<TAB><RET> (remember, auto-complete magic works here), the system asks for a new migration name. I enter a string that I need and a new migration named properly appears.

Makefiles may include other ones so probably you can maintain separate files for both production/developer modes or for developers/ops teams. Since those files support comments, you are welcome to put doc hints there.

The utility allows to chain targets. When I start working on a new task, at least I need to perform three steps: 1) migrate the database; 2) run tests to ensure everything works before I change something; 3) launch REPL. In my make file, I’ve got a separate target for each step. But I can call them in chain as well:

make mig<TAB> te<TAB> re<TAB>

that expands into:

make migrate test repl

and finally becomes:

lein migratus migrate
lein test
lein repl

Inside a makefile, you can call for another makefile what may become a powerful feature of your build process. Here is how it works.

Say, in our project, on the top level of it there is a “email” folder that brings Node.js project for building email templates. Our Clojure application use those compiled templates for further processing with Selmer to send final emails to our clients. It’s obvious, I need those templates only when I work with our email subsystem and never else. So it would be madness to force all our developers to install Node.js with tons of packages and compile the templates each time they launch REPL.

Inside “email” folder, there is a separate Makefile that knows how to deal with that sub-project. It installs all the requirements and has some useful targets, say, to run compiler in debug mode or open a browser window for preview.

A small fragment of that make file:

all: install build dist

install:
	yarn install

build:
	node_modules/.bin/gulp build --production

dist:
    cp ...

The default target performs all the targets that are required to get final templates. Now, in the main make file that is on the root of the project, I put:

EMAILDIR := $(CURDIR)/email

.PHONY: email
email:
    make -C $(EMAILDIR)

uberjar-build: email
    lein uberjar

So this configuration gives me freedom of choice. When I launch REPL, I don’t need all that stuff to deal with compiling emails. Probably, I may work in a company for years without touching them. But those guys who do, they run make email and get the full and ready email installation. Finally, no one be able to build an Uberjar without compiling fresh emails.

Now take into account that those emails were just a small part of a project. Remember, for successful production build you might need fetching huge JSON declarations from 3rd-party services; compiling smart contracts; building Docker images; building CSS and much more. Now answer, whey lein utility that even doesn’t have any default capabilities for that, should do it instead of special tools designed for exactly those things?

I think it’s obvious now that lein only should be used to manage your Clojure code. Never configure it to run non-Clojure-related stuff.