Npm, package.json, package-lock.json, node_modules. What does it all mean? Create your first npm package.

Okay, for this blog post I want to talk about NPM. How does it work? How does it relate to package.json, package-lock.json, and the node_modules directory?

I will also be going over “what is a package and what is a module?” Also, we will create our own very simple npm package and publish it.

Introduction

I’ve been working with Javascript libraries like React, and Vue for almost 2 years now — also working with NodeJS for APIs. I understand a little bit about installing packages and importing them into my projects. Being able to import a library of code and plug & play makes building applications so much easier and quicker. However, when things go wrong with a package, it’s so frustrating because, one, I don’t have a deep understanding of NPM, and two, I’m not familiar with the code of the package (the imported code is not my code).

I can’t really do much about packages not being my code, but I think by familiarizing myself with npm I can become more adept at working with packages and debugging any errors.

Just yesterday, I was trying to build a Docker image of a Vue project and I was running into problems with npm. During the npm run build command there was a problem with the dependency I was using called v-calendar . I was importing a CSS file from the v-calendar module that was no longer available. Also, the usage of the package had been updated and the code that was using v-calendar component was breaking.

After I fixed these two things, the npm run build command worked, but it took so much Googling and hours of unnecessary code-fidgeting to figure out. If instead, I understood how npm worked, I could’ve easily seen the npm error message and solved my problem.

So with all that said, today, I want to understand the basics of npm, and therefore, be easily able to solve dependency or package related problems.

Also, package managers are involved in any modern programming language today. Ruby bundler, Python PiP, NodeJS npm. The knowledge I learn about npm would be easily transferrable to other package managers so npm would be a useful thing to learn.

Note: I think I should also note that I won’t be focusing too much on npm commands because you can easily look them up in the documentation. Rather, I’ll be focusing on the “what” and also the “how” of these different npm components.

What is NPM exactly?

So I think the first thing we should talk about is NPM. What is NPM exactly?

It’s mainly two different (or three) things.

1. The CLI (command-line interface) we use to install packages e.g. npm install some_module
2. The registry => a public database where JS packages are uploaded to.
3. (The Website)

Important definitions: Module vs Package

Package => a file or directory that is described by a package.json.

Module => anything in the node_modules directory that can be loaded with require() in a Node.js program.

What can be loaded by Nodejs require()?

• a folder with a package.json containing main
• a folder with an index.js file in it
• a js file

What is the difference?

Most npm packages are modules. However, some npm packages can’t be loaded e.g. cli packages that only contain an executable command-line interface and don’t provide a main field for use in NodeJS programs.

An example:

For example, if you create a file at node_modules/foo.js and then had a program that did var f = require('foo.js'), it would load the module. However, foo.js is not a "package" in this case, because it does not have a package.json.

Alternatively, if you create a package which does not have an index.js or a "main" field in the package.json file, then it is not a module. Even if it's installed in node_modules, it can't be an argument to require().

In the context of a NodeJS/npm application, when we run npm install some-package we are installing a PACKAGE from the npm registry. The package is then added to our node_modules directory. The package is now also a MODULE of our application and we can load it into our app using const somePackage = require(‘some-package’);.

What is package.json?

Now that we know that a package is simply a directory containing a package.json file, the next thing we want to know is what is the package.json file exactly??

Package.json is a JSON file that describes the package for npm. I think creating and publishing our own package will easily demonstrate how a package.json relates to the package, so let’s do that here.

1. First, create an npm account on the website. https://www.npmjs.com/
2. Next, create your package
   a. Create your directory mkdir arthur-test-package && cd arthur-test-package
b. Create your package.json file npm init
The log will prompt you to describe your package, i.e. the name and other information. You can just hit enter for all the prompts and use the default fields. The important thing to note is name and the main field. The name will be used as the name when published to the registry and the main field is the file that gets imported when the user does require(‘your-module’).
You can also use npm init -y to create a package.json with default fields

c. In your “main” file (default is index.js) create the code. Mine looks like this (this goes without saying but make sure you npm install package-name any dependencies you need in your package so that your packge.json file will list the dependency.)

4. Now we have our package and we just need to log in before we can publish it to the registry, npm login and enter your username and password

5. Publish to the npm registry. npm publish

That’s it! It’s surprisingly easy to create and publish an npm package. We can easily check out our package on the npm website by navigating to the “packages” section under our profile.

See the documentation for more details about publishing your package to npm!

We can also our package in a separate NodeJS project.

In a different NodeJS project with npm initialized, we can run npm install arthur-test-package and use our arthur-test-package in our code.

Now, when we run this file node someotherfile.js , it logs the day string using the function that we created in arthur-test-package.

So after creating our own package and its accompanying package.json file, we know that a package.json file is just a JSON file that describes the package to NPM.

• the name of the package
• any dependencies the package needs
• the main field describes what file gets imported when this package is loaded using require

How npm commands work with the package.json file

Some other things to know are how npm commands work with the package.json file.

What happens when you run npm install?

npm install => will install any packages listed under “dependencies” in your package.json file into the node_modules directory so that you can begin using the modules in your application!

npm install some-package => will install the specific package into node_modules directory and ALSO will list the package under “dependencies” in package.json so that npm knows this package requires some-package to run correctly.

Finally, what is package-lock.json?

When you run npm install for the first time, npm automatically creates a package-lock.json file?

If we have the package.json that describes any dependencies that we need, why do we need a package-lock.json??

It says in the docs:

package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.

If we read more it says that package-lock.json:

Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.

In other words, package-lock.json ensures that anyone running npm install will generate an identical dependency tree. This is important because your project may require several different versions of the same package. In order to ensure that these dependency requirements are resolved we need dependency trees, and anybody else that wants to run your program also needs to generate the exact dependency tree.

Resolving different dependency versions is known as “dependency hell”. You can read more about what dependency hell is and how npm resolves dependency hell in their documentation here. https://npm.github.io/how-npm-works-docs/npm2/how-npm2-works.html

Why should we commit the package-lock.json file to GitHub?

I’m always a little unsure as to whether the package-lock.json file should be committed to GitHub, but after understanding what they do it makes a little more sense why they should be committed.

From earlier:

[packge-lock.json] describe[s] a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.

• Also, package-lock.json provides a history of our dependency tree without having to commit the entire node_modules directory.

So committing our package-lock.json provides a history of the dependency tree in our GitHub repository and also allows other people to generate the exact dependency tree in node_modules.

Final thoughts…

Alright so with that, I think we have a pretty solid understanding of how npm works and what package.json, package-lock.json, and node_modules are.

• npm is a public registry where packages are published to
• npm is also the CLI we use to download packages
• package.json is a JSON file that describes our package to npm
• node_modules is the directory that npm installs our packages to and also where NodeJS looks for modules
• and finally, package-lock.json is a file that describes the exact dependency tree that was generated by npm; it ensures that other people using the application create the identical dependency tree when installing packages!

Initially, I also wanted to get into npm scripts , specifically, React and Vue run and build scripts and how exactly they work, but I think this post has gotten long enough. Also, breaking down what those scripts do seem extensive enough on their own, so hopefully, I can cover that another time.