Vendoring

Carson Gross

“Vendoring” software is a technique where you copy the source of another project directly into your own project.

It is an old technique that has been used for time immemorial in software development, but the term “vendoring” to describe it appears to have originated in the ruby community.

Vendoring can be and is still used today. You can vendor htmx, for example, quite easily.

Assuming you have a /js/vendor directory in your project, you can just download the source into your own project like so:

curl https://raw.githubusercontent.com/bigskysoftware/htmx/refs/tags/v2.0.4/dist/htmx.min.js > /js/vendor/htmx-2.0.4.min.js

You then include the library in your head tag:

<script src="/js/vendor/htmx-2.0.4.min.js"></script>

And then you check the htmx source into your own source control repository. (I would even recommend considering using the non-minimized version, so you can better understand and debug the code.)

That’s it, that’s vendoring.

Vendoring Strengths

OK, great, so what are some strengths of vendoring libraries like this?

It turns out there are quite a few:

On the other hand, vendoring also has one massive drawback: there typically isn’t a good way to deal with what is called the transitive dependency problem.

If htmx had sub-dependencies, that is, other libraries that it depended on, then to vendor it properly you would have to start vendoring all those libraries as well. And if those dependencies had further dependencies, you’d need to install them as well… And on and on.

Worse, two dependencies might depend on the same library, and you’ll need to make sure you get the correct version of that library for everything to work.

This can get pretty difficult to deal with, but I want to make a paradoxical claim that this weakness (and, again, it’s a real one) is actually a strength in some way:

Because dealing with large numbers of dependencies is difficult, vendoring encourages a culture of independence.

You get more of what you make easy, and if you make dependencies easy, you get more of them. Making dependencies, especially transitive dependencies, more difficult would make them less common.

And, as we will see in a bit, maybe fewer dependencies isn’t such a bad thing.

Dependency Managers

That’s great and all, but there are significant drawbacks to vendoring, particular the transitive dependency problem.

Modern software engineering uses dependency managers to deal with the dependencies of software projects. These tools allow you to specify your projects dependencies, typically via some sort of file. They then they will install those dependencies and resolve and manage all the other dependencies that are necessary for those dependencies to work.

One of the most widely used package managers is NPM: The Node Package Manager. Despite having no runtime dependencies, htmx uses NPM to specify 16 development dependencies. Development dependencies are dependencies that are necessary for development of htmx, but not for running it. You can see the dependencies at the bottom of the NPM package.json file for the project.

Dependency managers are a crucial part of modern software development and many developers today couldn’t imagine writing software without them.

The Trouble with Dependency Managers

So dependency managers solve the transitive dependency problem that vendoring has. But, as with everything in software engineering, there are tradeoffs associated with them. To see some of these tradeoffs, let’s take a look at the package-lock.json file in htmx.

NPM generates a package-lock.json file that contains the resolved transitive closure of dependencies for a project, with the concrete versions of those dependencies. This helps ensure that the same dependencies are used unless a user explicitly updates them.

If you take a look at the package-lock.json for htmx, you will find that the original 13 development dependencies have ballooned into a total of 411 dependencies when all is said and done.

htmx, it turns out, relies on a huge number of packages, despite priding itself on being a relatively lean. In fact, the node_modules folder in htmx is a whopping 110 megabytes!

But, beyond this bloat there are deeper problems lurking in that mass of dependencies.

While writing this essay I found that htmx apparently depends on the array.prototype.findlastindex, a polyfill for a JavaScript feature introduced in 2022.

Now, htmx 1.x is IE compatible, and I don’t want polyfills for anything: I want to write code that will work in IE without any additional library support. And yet a polyfill has snuck in via a chain of dependencies (htmx does not directly rely on it) that introduces a dangerous polyfill that would let me write code that would break in IE, as well as other older browsers.

This polyfill may or may not be available when I run the htmx test suite (it’s hard to tell) but that’s the point: some dangerous code has snuck into my project without me even knowing it, due to the number and complexity of the (development) dependencies it has.

This demonstrates significant cultural problem with dependency managers:

They tend to foster a culture of, well, dependency.

A spectacular example of this was the infamous left-pad incident, in which an engineer took down a widely used package and broke the build at companies like Facebook, PayPal, Netflix, etc.

That was a relatively innocuous, although splashy, issue, but a more serious concern is supply chain attacks, where a hostile entity is able to compromise a company via code injected unwittingly via dependencies.

The larger our dependency graph gets, the worse these problems get.

Dependencies Reconsidered

I’m not the only person thinking about our culture of dependency. Here’s what some other, smarter folks have to say about it:

Armin Ronacher, creator of flask recently said this on the ol’twits:

The more I build software, the more I despise dependencies. I greatly prefer people copy/pasting stuff into their own code bases or re-implement it. Unfortunately the vibe of the time does not embrace that idea much. I need that vibe shift.

He also wrote a great blog post about his experience with package management in the Rust ecosystem:

It’s time to have a new perspective: we should give kudos to engineers who write a small function themselves instead of hooking in a transitive web of crates. We should be suspicious of big crate graphs. Celebrated are the minimal dependencies, the humble function that just quietly does the job, the code that doesn’t need to be touched for years because it was done right once.

Please go read it in full.

Back in 2021, Tom Macwright wrote this in Vendor by default

But one thing that I do think is sort of unusual is: I’m vendoring a lot of stuff.

Vendoring, in the programming sense, means “copying the source code of another project into your project.” It’s in contrast to the practice of using dependencies, which would be adding another project’s name to your package.json file and having npm or yarn download and link it up for you.

I highly recommend reading his take on vendoring as well.

Software Designed To Be Vendored

Some good news, if you are an open source developer and like the idea of vendoring, is that there is a simple way to make your software vendor-friendly: remove as many dependencies as you can.

DaisyUI, for example, has been in the process of removing their dependencies, going from 100 dependencies in version 3 to 0 in version 5.

There is also a set htmx-adjacent projects that are taking vendoring seriously:

None of these JavaScript projects are available in NPM, and all of them recommend vendoring the software into your own project as the primary installation mechanism.

Vendor First Dependency Managers?

The last thing I want to briefly mention is a technology that combines both vendoring and dependency management: vendor-first dependency managers. I have never worked with one before, but I have been pointed to vend, a common lisp vendor oriented package manager (with a great README), as well as go’s vendoring option.

In writing this essay, I also came across vendorpull and git-vendor, both of which are small but interesting projects.

These all look like excellent tools, and it seems to me that there is an opportunity for some of them (and tools like them) to add additional functionality to address the traditional weaknesses of vendoring, for example:

With these additional features I wonder if vendor-first dependency managers could compete with “normal” dependency managers in modern software development, perhaps combining some of the benefits of both approaches.

Regardless, I hope that this essay has helped you think a bit more about dependencies and perhaps planted the idea that maybe your software could be a little less, well, dependent on dependencies.

</>