Skip to content

Commit

Permalink
Update docs site for version 2.23.0.dev4 (#236)
Browse files Browse the repository at this point in the history
  • Loading branch information
WorkerPants authored Jul 25, 2024
1 parent efed535 commit 0a669d4
Show file tree
Hide file tree
Showing 64 changed files with 4,157 additions and 871 deletions.
109 changes: 109 additions & 0 deletions docs/docs/contributions/development/maintenance-tasks-and-scripts.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
title: Maintenance tasks and scripts
sidebar_position: 6
---

There's a variety of maintenance tasks that happen on different frequencies.

## Update the default/known versions of a built-in tool/subsystem

### External tools (downloaded executables)

Some tools use [the `ExternalTool` class](../../writing-plugins/the-rules-api/installing-tools#externaltool-install-pre-compiled-binaries) to download a binary from the internet, verify its size and hash, and then execute it.

To update these:

1. For each platform and version to add:
1. Download the archive/binary.
2. Verify it: check signatures and/or hashes if available.
3. Compute the sha256 hash and byte length. For example, if it's called `archive.zip`: `tee >(shasum -a 256) >(wc -c) > /dev/null < archive.zip`.
2. Apply the new values:
1. Adjust `default_version` to the new version.
2. Add or replace to `default_known_versions` using the hashes and lengths above (for some tools we don't preserve older versions, especially if they have strong backwards compatibility guarantees, while for others we do retain older versions).

Example: [#20469](https://github.com/pantsbuild/pants/pull/20469).

#### PEX

The PEX external tool is a bit special, because it also appears as a requirement of the Pants project itself in `3rdparty/python/requirements.txt`. To update pex, do both:

1. Update the `pex-cli` subsystem, as above (in `src/python/pants/backend/python/util_rules/pex_cli.py`).
2. Update the requirements file and run `pants generate-lockfiles --resolve=python-default` to update Pants' own lockfile.

Example: [#20782](https://github.com/pantsbuild/pants/pull/20782).

#### Terraform

The `build-support/bin/terraform_tool_versions.py` script can help update Terraform versions.

### Python tools

Some tools use `PythonToolBase` to install executable PyPI packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these:

1. Adjust `default_requirements` and/or `default_interpreter_constraints` as required.
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class.

Example: [#20924](https://github.com/pantsbuild/pants/pull/20924).

### JVM tools

Some tools use `JVMToolBase` to install executable JVM packages, using a lockfile. Pants packages a default lockfile with a specific version of the package. To update these:

1. Adjust `default_version` and/or `default_artifacts` as required.
2. Run `build-support/bin/generate_builtin_lockfiles.py $scope`, where `$scope` is the `options_scope` of the subsystem class.

Example: none yet.

### JS tools

Some tools use `NodeJSToolBase` to install executable npm packages. To update these:

1. Update `default_version`. That's all.

Example: [#21007](https://github.com/pantsbuild/pants/pull/21007).

## Update or create FaaS complete platforms files

The function-as-a-service (FaaS) subsystems provide some built-in PEX complete platforms JSON files, for specific runtimes. To update or create these:

### AWS Lambda

1. Adjust `PythonAwsLambdaRuntime.known_runtimes` as required
2. Run `build-support/bin/generate_faas_complete_platforms.py` to create any new files and update the existing ones, using AWS's published docker images

Example: [#21004](https://github.com/pantsbuild/pants/pull/21004).

### Google Cloud Functions

Complete platforms are not provided for GCF yet: [#18195](https://github.com/pantsbuild/pants/issues/18195).

## Cherry-pick a pull request to an older version

We maintain multiple versions, with `main` being our development branch, and various `2.*.x` branches for the stable versions (see [Release strategy](../releases/release-strategy.mdx) for more details).

### Cherry-picking a new pull request

When a change needs to land in `main` but also one or more older versions, the usual process is:

1. Create or review the pull request against `main` as usual
2. Label it as `needs-cherrypick` and set milestone to the oldest release to which it should be cherry-picked
3. Merge the pull request as normal
4. At this point, automation kicks in and attempts to cherrypick the merged commit to the release in the milestone and any newer ones.
5. The automation opens pull requests targeting each of the relevant `2.*.x` branches for which cherry-picking succeeds.
6. If the automation fails to do a cherry-pick, it will mark the PR as `auto-cherry-picking-failed`
7. In either case, the automation will add a comment describing what happened to the original pull request.

For example, suppose `main` is for `2.23.x` and we're still maintaining `2.20.x`, `2.21.x` and `2.22.x`. If a pull request is labelled `needs-cherrypick` and has milestone `2.21.x`, then merging it will attempt to cherry pick to `2.21.x` and `2.22.x`.

The process may fail in one of two ways:

- The cherry-picking process failed, and tagged the PR with `auto-cherry-picking-failed`: follow the instructions in the comment on the pull request. (This likely means there are merge conflicts that require manual resolution.)
- the cherry-pick hasn't (yet) run: trigger the automation manually by going to [the GitHub Action](https://github.com/pantsbuild/pants/actions/workflows/auto-cherry-picker.yaml), clicking on the "Run workflow" button, and providing the PR number.

### Cherry-picking a merged pull request

A pull request might merged without being configured for cherry-picking, and we decide later that it should be. To cherry-pick in this case:

1. Label the pull request as `needs-cherrypick` and set milestone to the oldest release to which it should be cherry-picked
2. Trigger the automation manually by going to [the GitHub Action](https://github.com/pantsbuild/pants/actions/workflows/auto-cherry-picker.yaml), clicking on the "Run workflow" button, and providing the PR number.
3. As above, the automation may (partially) succeed or fail, and will leave a comment describing what happened.
6 changes: 3 additions & 3 deletions docs/docs/contributions/development/setting-up-pants.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ Pants requires a more modern OpenSSL version than the one that comes with macOS.

```bash
$ brew install openssl
$ echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bashrc
$ echo 'export LDFLAGS="-L/usr/local/opt/openssl/lib"' >> ~/.bashrc
$ echo 'export CPPFLAGS="-I/usr/local/opt/openssl/include"' >> ~/.bashrc
$ echo 'export PATH="$(brew --prefix)/opt/openssl/bin:$PATH"' >> ~/.bashrc
$ echo 'export LDFLAGS="-L$(brew --prefix)/opt/openssl/lib"' >> ~/.bashrc
$ echo 'export CPPFLAGS="-I$(brew --prefix)/opt/openssl/include"' >> ~/.bashrc
```

(If you don't have `brew` installed, see [https://brew.sh](https://brew.sh))
Expand Down
26 changes: 25 additions & 1 deletion docs/docs/contributions/releases/release-process.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ The release commit is the commit that bumps the VERSION string. For `dev`/`a0` r

### Bump the VERSION

From the `main` branch, run `pants run src/python/pants_release/start_release.py -- --new 2.9.0.dev1 --release-manager your_github_username --publish` with the relevant version and your own GitHub username.
From the `main` branch, run `pants --no-watch-filesystem --no-pantsd run src/python/pants_release/start_release.py -- --new 2.99.0.dev1 --release-manager your_github_username --publish` with the relevant version and your own GitHub username.

This will create a pull request that:

Expand Down Expand Up @@ -158,6 +158,30 @@ Once the workflow finishes, look through any failures and determine if there's a

Alternatively, after starting the workflow, post the link to the in-progress run in `#development` in Slack, so that someone can come back to it when it does finish.

## Step 5: Publish a schema in JSON Schema Store

Some editors can use JSON Schema for better completions when editing TOML files like `pants.toml`.
Pants configuration file schema is published at https://www.schemastore.org/. For every stable `2.x.0` release,
a new schema needs to be generated and uploaded by submitting a PR against https://github.com/SchemaStore/schemastore.
This is an example pull request for reference: https://github.com/SchemaStore/schemastore/pull/3880.

To produce `pantsbuild-<version>.json` schema file, run:

```bash
pants help-all > all-help.json
pants run build-support/bin/generate_json_schema.py -- --all-help-file=all-help.json
```

It may be helpful to compare the last schema file with the newly produced one to make sure there are no discrepancies
(e.g. the config values have a sensible type and the help strings are rendered adequately). You can download the
schemas of previous releases from the store website; the JSON files are available at
`https://json.schemastore.org/pantsbuild-<version>.json`.

Watch out for any configuration parameters that may rely on your local environment as certain default config values
will be expanded using local runtime environment which is undesirable. The script handles those known config values
by keeping a list of them, however, it may need to be extended as more options with environment specific default
values are added.

## When Things Go Wrong

From time to time, a release will fail. It's a complex process. The first thing to do after you've
Expand Down
4 changes: 4 additions & 0 deletions docs/docs/javascript/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"label": "Javascript",
"position": 13
}
4 changes: 4 additions & 0 deletions docs/docs/javascript/overview/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"label": "Javascript overview",
"position": 1
}
71 changes: 71 additions & 0 deletions docs/docs/javascript/overview/enabling-javascript-support.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
title: Enabling Javascript support
sidebar_position: 0
---

How to enable Pants's bundled Javascript backend package.

---

:::note Example Javascript repository
See [here](https://github.com/pantsbuild/example-javascript) for examples of Pants's Javascript functionality.

:::

### Configuring the repository

Enable the experimental Javascript [backend](../../using-pants/key-concepts/backends.mdx) like this:

```toml title="pants.toml"
[GLOBAL]
...
backend_packages = [
"pants.backend.experimental.javascript"
]
```

Pants uses [`package_json`](../../../reference/targets/package_json.mdx) targets to model a NodeJS package.
Further, [`javascript_source`](../../../reference/targets/javascript_source.mdx) and
[`javascript_tests`](../../../reference/targets/javascript_test.mdx) targets are used to know which Javascript files to
run on and to set any metadata.

You can generate these targets by running [`pants tailor ::`](../../getting-started/initial-configuration.mdx#5-generate-build-files).

```
❯ pants tailor ::
Created project/BUILD:
- Add javascript_sources target project
- Add javascript_tests target tests
```


### Setting up node
Pants will by default download a distribution of `node` according to the
[`nodejs` subsystem](../../../reference/subsystems/nodejs) configuration. If you wish to instead use a locally installed
version of, for example, 18.0.0 using `nvm` and its `.nvmrc` file, the following will get you there:

```toml tab={"label": "pants.toml"}
[nodejs]
known_versions = [] # Assign this to the empty list to ensure Pants never downloads.
version = "v18.0.0"
search_path = ["<NVM_LOCAL>"]

```

```txt tab={"label": ".nvmrc"}
v18.0.0
```

### Setting up a package manager
To set a package manager project wide, do the following:

```toml title="pants.toml"
[nodejs]
package_manager = "pnpm" # or yarn, or npm.

```

you can instead opt to use the [`package.json#packageManager`](./package.mdx#package-manager) field for this setting.
Regardless of setting, pants uses the [`corepack`](https://github.com/nodejs/corepack) version distributed with the Node
version you have chosen to install and manage package managers.
29 changes: 29 additions & 0 deletions docs/docs/javascript/overview/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Javascript overview
sidebar_position: 0
---

Pants's support for Javascript.

:::caution Javascript support is beta stage
We are done implementing most functionality for Pants's Javascript support
([tracked here](https://github.com/pantsbuild/pants/labels/backend%3A%20javascript)).
However, there may be use cases that we aren't yet handling.
:::

The Javascript and NodeJS ecosystem has a seemingly endless amount of frameworks and tooling,
all orchestrated via package managers.

Pants employs a wrapping approach with a thin caching layer applied on top of current supported package managers:
`npm`, `pnpm` and `yarn`.

Features for Javascript:

- Caching the results of your test scripts and build scripts,
making the latter available in your Pants workflows as [`resources`](../../reference/targets/resource) and
package artifacts.
- A consistent interface for all languages/tools in your repository,
such as being able to run `pants fmt lint check test package`.
- [Remote execution and remote caching](../../using-pants/remote-caching-and-execution/index.mdx).
- [Advanced project introspection](../../using-pants/project-introspection.mdx),
such as finding all code that transitively depends on a certain package.
65 changes: 65 additions & 0 deletions docs/docs/javascript/overview/lockfiles.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
title: Lockfiles
sidebar_position: 2
---

Package manager lockfile integration

---

Third-party dependencies are specified in the package.json fields.
All package managers vendors a lockfile format specific for the package manager you are using. Pants knows of this
lockfile and models it as a "resolve".

Resolves is the only way to deal with dependencies within pants, and no extra configuration is required.

You can however name your resolves/lockfiles. The resolve name is otherwise auto-generated.

```toml title="pants.toml"
[GLOBAL]
backend_packages.add = [
"pants.backend.experimental.javascript"
]

[nodejs.resolves]
package-lock.json = "my-lock"

```

You generate the lockfile as follows:

```shell title="Bash"
$ pants generate-lockfiles
19:00:39.26 [INFO] Completed: Generate lockfile for my-lock
19:00:39.29 [INFO] Wrote lockfile for the resolve `my-lock` to package-lock.json
```


## Using lockfiles for tools

To ensure that the same version of tooling you have specified in `package.json` is used with a NodeJS powered tool,
specify the resolve name for the tool.
E.g., for the Prettier linter:

```toml tab={"label": "pants.toml"}
[GLOBAL]
backend_packages.add = [
"pants.backend.experimental.javascript",
"pants.backend.experimental.javascript.lint.prettier",
]

[prettier]
install_from_resolve = "nodejs-default"

```
```json tab={"label": "package.json"}
{
"name": "@my-company/pkg",
"devDependencies": {
"prettier": "^2.6.2"
}
}
```
```python tab={"label": "BUILD"}
package_json(name="pkg")
```
Loading

0 comments on commit 0a669d4

Please sign in to comment.