仓库源文，站点原文

layout: "../layouts/BlogPost.astro" title: "Understand npm concepts" slug: understand-npm-concepts description: "" added: "Dec 14 2022" tags: [web]

updatedDate: "Aug 26 2024"

package.json and package-lock.json

package-lock.json (called package locks, or lockfiles) is automatically generated for any operations where npm modifies either the node_modules tree or package.json. This file is intended to be committed into source repositories. The purpose of the package-lock.json is to avoid the situation where installing modules from the same package.json results in two different installs. package-lock.json is a large list of each dependency listed in your package.json, the specific version that should be installed, the location (URI) of the module, a hash that verifies the integrity of the module, the list of packages it requires.

  1. If you run npm i against the package.json and package-lock.json, the latter will never be updated, even if the package.json would be happy with newer versions.
  2. If you manually edit your package.json to have different ranges and run npm i and those ranges aren't compatible with your package-lock.json, then the latter will be updated with version that are compatible with your package.json.
  3. Listed dependencies in package-lock.json file have mixed (sha1/sha512) integrity checksum. npm changed the integrity checksum from sha1 to sha512. Only packages published with npm@5 or later will include a sha512 integrity hash.

Two fields are mandatory in package.json:

Package code entry points:

<img alt="package-code-entries" src="https://raw.gitmirror.com/kexiZeroing/blog-images/main/package-code-entries.png" width="700" />

Read How To Create An NPM Package by Total TypeScript

Create a package.json with:

@arethetypeswrong/cli is a tool that checks if your package exports are correct. Add a script "check-exports": "attw --pack ." to check if all exports from your package are correct.

Add a main field to your package.json with "main": "dist/index.js", and our package is compatible with systems running ESM.

npm run check-exports
┌───────────────────┬──────────────────────────────┐
│                   │ "tt-package-demo"            │
├───────────────────┼──────────────────────────────┤
│ node10            │ 🟢                           │
├───────────────────┼──────────────────────────────┤
│ node16 (from CJS) │ ⚠️ ESM (dynamic import only) │
├───────────────────┼──────────────────────────────┤
│ node16 (from ESM) │ 🟢 (ESM)                     │
├───────────────────┼──────────────────────────────┤
│ bundler           │ 🟢                           │
└───────────────────┴──────────────────────────────┘

If you want to publish both CJS and ESM code, you can use tsup. This is a tool built on top of esbuild that compiles your TypeScript code into both formats. We'll now be running tsup to compile our code instead of tsc.

// tsup.config.ts
import { defineConfig } from "tsup";

export default defineConfig({
  entryPoints: ["src/index.ts"],
  format: ["cjs", "esm"],
  dts: true,
  outDir: "dist",
  clean: true,
});

This will create a dist/index.js (for ESM) and a dist/index.cjs (for CJS). Add an exports field to your package.json, which tells programs consuming your package how to find the CJS and ESM versions of your package. In this case, we're pointing folks using import to dist/index.js and folks using require to dist/index.cjs. Run check-exports again, everything is green.

{
  "exports": {
    ".": {
      "import": "./dist/index.js",
      "require": "./dist/index.cjs"
    }
  }
}

tsup also creates declaration files for each of your outputs. index.d.ts for ESM and index.d.cts for CJS. This means you don't need to specify types in your package.json. TypeScript can automatically find the declaration file it needs.

npm install and npm ci

npm install reads package.json to create a list of dependencies and uses package-lock.json to inform which versions of these dependencies to install. If a dependency is not in package-lock.json it will be added by npm install.

npm ci (named after Continuous Integration) installs dependencies directly from package-lock.json and uses package.json only to validate that there are no mismatched versions. If any dependencies are missing or have incompatible versions, it will throw an error. It will delete any existing node_modules folder to ensure a clean state. It never writes to package.json or package-lock.json. It does however expect a package-lock.json file in your project — if you do not have this file, npm ci will not work and you have to use npm install instead.

npm audit automatically runs when you install a package with npm install. It checks direct dependencies and devDependencies, but does not check peerDependencies. Read more about npm audit: Broken by Design by Dan Abramov.

npm outdated, a built-in npm command, will check the registry to see if any installed packages are currently outdated. By default, only the direct dependencies of the root project are shown. Use --all to find all outdated meta-dependencies as well.

npm ls

npm ls (aliases: list, la, ll) list dependencies that have been installed to node_modules. It throws an error for discrepancies between package.json and its lock file.

const cp = require("child_process");
const verify = () => cp.exec("npm ls", error => {
  if (error) {
    console.error("Dependency mismatch between package.json and lock. Run: npm install");
    throw error;
  }
  console.log("Dependencies verified =)");
});

verify();

What do "idealTree" and "reify" mean in the context of npm?
An idealTree is the tree of package data that we intend to install. actualTree is the representation of the actual packages on disk.

During lockfile validation, npm compares the inventory of package items in the tree that is about to be installed (idealTree) with the inventory of items stored in the package-lock file (virtualTree).

During reification, the idealTree is diffed against the actual tree, and then the nodes from the ideal tree are extracted onto disk. At the end of reify(), the ideal tree is copied to actualTree, since then it reflects the actual state of the node_modules folder.

dependencies, devDependencies and peerDependencies

Dependencies are required at runtime, like a library that provides functions that you call from your code. If you are deploying your application, dependencies has to be installed, or your app will not work. They are installed transitively (if A depends on B depends on C, npm install on A will install B and C). Example: lodash, your project calls some lodash functions.

devDependencies are dependencies you only need during development, like compilers that take your code and compile it into javascript, test frameworks or documentation generators. They are not installed transitively (if A depends on B dev-depends on C, npm install on A will install B only). Example: grunt, your project uses grunt to build itself.

peerDependencies are dependencies that your project hooks into, or modifies, in the parent project, usually a plugin for some other library. It is just intended to be a check, making sure that the project that will depend on your project has a dependency on the project you hook into. So if you make a plugin C that adds functionality to library B, then someone making a project A will need to have a dependency on B if they have a dependency on C. Example: your project adds functionality to grunt and can only be used on projects that use grunt.

In npm versions 3 through 6, peerDependencies were not automatically installed, and would raise a warning if an invalid version of the peer dependency was found in the tree. As of npm v7, peerDependencies are installed by default. (npm has a shortcut where it automatically install mandatory peer dependencies even if the parent package does not depend on them.) If your dependency contains some peerDependencies that conflict with the root project's dependency, run npm install --legacy-peer-deps to skips strict peer dependency checks, allowing installation of packages with unmet peer dependencies to avoid errors. (--force flag will ignore and override any dependency conflicts, forcing the installation of packages.)

optionalDependencies are dependencies that are not essential for the primary functionality of a package but are beneficial for providing additional features. Let’s say you have a dependency that may be used, but you would like the package manager to proceed if it cannot be found or fails to install. In that case, you can add those dependencies in the optionalDependencies object. A good use case for optionalDependencies is if you have a dependency that won’t necessarily work on every machine. But you should have a fallback plan in case the installation fails.

@npmcli/arborist is the library that calculates dependency trees and manages the node_modules folder hierarchy for the npm command line interface. It's used in some tools like npm-why to help identify why a package has been installed.

Arborist - the npm tree doctor: npx @npmcli/arborist --help

URLs as dependencies

See details at https://docs.npmjs.com/cli/v8/configuring-npm/package-json#urls-as-dependencies

  1. Git URLs as dependencies
  2. GitHub URLs: refer to GitHub urls as "foo": "user/foo-project"
  3. Local Paths: You can provide a path to a local directory that contains a package "bar": "file:../foo/bar"

You can configure npm to resolve your dependencies across multiple registries.

# .npmrc

# Fetch `@lihbr` packages from GitHub registry
@lihbr:registry=https://npm.pkg.github.com

# Fetch `@my-company` packages from My Company registry
@my-company:registry=https://npm.pkg.my-company.com

fix broken node modules instantly

patch-package lets app authors instantly make and keep fixes to npm dependencies. Patches created are automatically and gracefully applied when you use npm or yarn.

# fix a bug in one of your dependencies
vim node_modules/some-package/brokenFile.js

# it will create a folder called `patches` in the root dir of your app. 
# Inside will be a `.patch` file, which is a diff between normal old package and your fixed version
npx patch-package some-package

# commit the patch file to share the fix with your team
git add patches/some-package+3.14.15.patch
git commit -m "fix brokenFile.js in some-package"

// package.json
"scripts": {
  "postinstall": "patch-package"
}

npm and npx

One might install a package locally on a certain project using npm install some-package, then we want to execute that package from the command line. Only globally installed packages can be executed by typing their name only. To fix this, you must type the local path ./node_modules/.bin/some-package.

npx comes bundled with npm version 5.2+. It will check whether the command exists in $PATH or in the local project binaries and then execute it. So if you wish to execute the locally installed package, all you need to do is type npx some-package.

Have you ever run into a situation where you want to try some CLI tool, but it’s annoying to have to install a global just to run it once? npx is great for that. It will automatically install a package with that name from the npm registry and invoke it. When it’s done, the installed package won’t be anywhere in the global, so you won’t have to worry about pollution in the long-term. For example, npx create-react-app my-app will generate a react app boilerplate within the path the command had run in, and ensures that you always use the latest version of the package without having to upgrade each time you’re about to use it. There’s an awesome-npx repo with examples of things that work great with npx.

npm will cache the packages in the directory ~/.npm/_npx. The whole point of npx is that you can run the packages without installing them somewhere permanent. So I wouldn't use that cache location for anything. I wouldn't be surprised if cache entries were cleared from time to time. I don't know what algorithm, if any, npx uses for time-based cache invalidation.

You can find the npm-debug.log file in your .npm directory. To find your .npm directory, use npm config get cache. (It is located in ~/.npm so shared accross nodejs versions that nvm installed.) The default location of the logs directory is a directory named _logs inside the npm cache.

npm init and exec

npm init <initializer> can be used to set up a npm package. initializer in this case is an npm package named create-<initializer>, which will be installed by npm exec. The init command is transformed to a corresponding npm exec operation like npm init foo -> npm exec create-foo. Another example is npm init react-app myapp, which is same as npx create-react-app myapp. If the initializer is omitted (by just calling npm init), init will fall back to legacy init behavior. It will ask you a bunch of questions, and then write a package.json for you. You can also use -y/--yes to skip the questionnaire altogether.

npm 7 introduced the new npm exec command which, like npx, provided an easy way to run npm scripts on the fly. If the package is not present in the local project dependencies, npm exec installs the required package and its dependencies to a folder in the npm cache. With the introduction of npm exec, npx had been rewritten to use npm exec under the hood in a backwards compatible way.

npm create is an alias for npm init. Check more about npm init --help.

npm link

  1. Run npm link from your MyModule directory: this will create a global package {prefix}/node/{version}/lib/node_modules/<package> symlinked to the MyModule directory.
  2. Run npm link MyModule from your MyApp directory: this will create a MyModule folder in node_modules symlinked to the globally-installed package and thus to the real location of MyModule. Note that <package-name> is taken from package.json, not from the directory name.
  3. Now any changes to MyModule will be reflected in MyApp/node_modules/MyModule/. Use npm ls -g --depth=0 --link to list all the globally linked modules.
  4. Run npm unlink --no-save <package> on your project’s directory to remove the local symlink.

publish npm packages

Learn how to create a new npm package and publish the code to npm by the demo Building a business card CLI tool. Once your package is published to npm, you can run npx {your-command} to execute your script whenever you like.

Most popular npm packages: https://socket.dev/npm/category/popular

npm and pnpm

The very first package manager ever released was npm, back in January 2010. In 2020, GitHub acquired npm, so in principle, npm is now under the stewardship of Microsoft. (npm should never be capitalized unless it is being displayed in a location that is customarily all-capitals.)

npm handles the dependencies by splitting the installation process into three phases: Resolving -> Fetching -> Linking. Each phase needs to end for the next one to begin.

pnpm was released in 2017. It is a drop-in replacement for npm, so if you have an npm project, you can use pnpm right away. The main problem the creators of pnpm had with npm was the redundant storage of dependencies that were used across projects. 1) The way npm manages the disc space is not efficient. 2) pnpm doesn’t have the blocking stages of installation - the processes run for each of the packages independently.

Traditionally, npm installed dependencies in a flat node_modules folder. On the other hand, pnpm manages node_modules by using hard linking and symbolic linking to a global on-disk content-addressable store. It results in a nested node_modules folder that stores packages in a global store on your home folder (~/.pnpm-store/). Every version of a dependency is physically stored in that folder only once, constituting a single source of truth. pnpm identifies the files by a hash id (also called "content integrity" or "checksum") and not by the filename, which means that two same files will have identical hash id and pnpm will determine that there’s no reason for duplication.

<img alt="pnpm" src="https://raw.gitmirror.com/kexiZeroing/blog-images/main/008vxvgGly1h7aw9ablr4j30vm0u0q5z.jpg" width="650" />

npm scripts

npm scripts are a set of built-in and custom scripts defined in the package.json file. Their goal is to provide a simple way to execute repetitive tasks.

Despite "npm scripts" high usage they are not particularly well optimized.

  1. By running cat $(which npm), you will find npm CLI is a standard JavaScript file. The only special thing is the first line #!/usr/bin/env node which tells your shell the current file can be executed with node.
  2. Because it's just a js file, we can rely on all the usual ways to generate a profile. My favorite one is node’s --cpu-prof argument. Combine that knowledge together and we can generate a profile from an npm script via node --cpu-prof $(which npm) run myscript. Loading that profile into speedscope reveals quite a bit about how npm is structured. The majority of time is spent on loading all the modules that compose the npm cli. The time of the script that we’re running pales in comparison.