Initial sketch of what's new in 2.18 (#19999)

This sketches (in rather extensive detail...) a "what's new" file for
2.18, up to
https://github.com/pantsbuild/pants/commits/bd28f12623a57313486f9d2f834c0e78051aa8a7/
(bd28f12...2.18.x
is what's happened since then).

This basically lists everything from all the GitHub Releases, in human
form, because I don't think there's another way for a user to get a
sense of what's included in a single place.

I'm hoping we can do a quick pass here, and then iterate once it's in,
because there's so many sections that different people will know best
about, rather than having to gate it all through me (as the author of
this PR).

A downside of this current framing is that it doesn't emphasise the
breaking changes (user API changes) very well.

src/python/pants/notes/2.18.x.md

@@ -1,7 +1,205 @@
 # 2.18.x Release Series
 
+❗❗❗ This project is no longer being released via PyPI, and is instead released via GitHub Releases. See https://github.com/pantsbuild/pants/releases
+
+Individuals and companies can now [sponsor Pants financially](https://www.pantsbuild.org/docs/sponsorship).
+
+Pants is an open-source project that is not owned or controlled by any one company or organization, and does incur some expenses. These expenses are managed by Pants Build, a non-profit that was established for this purpose. This non-profit's only source of revenue is sponsorship by individuals and companies that use Pants.
+
+We offer [formal sponsorship tiers for companies](https://www.pantsbuild.org/docs/sponsorship), as well as individual sponsorships via [GitHub](https://github.com/sponsors/pantsbuild).
+
 ## What's New
 
+### Highlights
+
+- The improvements to AWS Lambda and Google Cloud Function support in 2.17 have continued in 2.18, including simpler configuration and support for AWS Lambda Layers.
+- Many more options can now be controlled when building Docker images.
+- Significant feature work and bug fixes to "deploy" backends like Helm and Terraform.
+
+Keep reading to see the details and what's also included.
+
+### Overall
+
+[The `pants` launcher binary](https://www.pantsbuild.org/v2.18/docs/installation) is now **required** to run Pants. It must be version 0.10.0 or greater: run `PANTS_BOOTSTRAP_VERSION=report pants` to check and `SCIE_BOOT=update pants` to upgrade to the latest. Pants 2.18 and later are not distributed via PyPI, and thus the launcher is required to coordinate downloading the appropriate versions from GitHub Releases.
+
+[The `[cli.alias]` section](https://www.pantsbuild.org/v2.18/docs/reference-cli) of `pants.toml` now supports defining flag-like aliases. For example: `--all-changed = "--changed-since=HEAD --changed-dependents=transitive"`.
+
+[The `paths` goal](https://www.pantsbuild.org/v2.18/docs/reference-paths) now supports passing groups of targets to `--from` and/or `--to`.
+
+The `help` goal now supports showing help for an environment variable. For example: `pants help PANTS_FIX_ONLY`.
+
+Additional bug fixes:
+
+- The client now detects changes in pants configuration that require restarting the pants daemon.
+- Sequences of goals that mutate files on disk and then immediately re-read them now no longer fail spuriously if the second goal happens faster than the OS file-change notification is received. For instance, `pants fix lint ::` would sometimes run the `lint` goal on the files before the `fix` changes were applied.
+- More reliable detection of when optimised hard-linking can be used for creating sandboxes. For instance, no longer hitting errors when a single device is mounted multiple times.
+
+The deprecation for referring to a "secondary" owner target via the file name has expired, and so this is no longer supported. For example, if `path/to/main.py` is the entry point of the `path/to:binary` `pex_binary` target, an invocation like `pants package path/to/main.py` must now use the target address instead: `pants package path/to:binary`.
+
+### BUILD files
+
+Several bug fixes:
+
+- Better error messages when a `BUILD` file refers to an unknown value.
+- `__defaults__` can now be used in the same `BUILD` file as [environment targets](https://www.pantsbuild.org/v2.18/docs/environments) like `local_environment`.
+
+### Performance
+
+Memory usage has been reduced, particularly for the Go backend.
+
+A best-effort limit has been placed on the number of open cache file handles (approximately 1000), reducing the need for `ulimit` adjustments.
+
+### Remote caching/execution
+
+Remote caching using the gRPC remote execution API now supports mTLS, via the new [`remote_client_certs_path`](https://www.pantsbuild.org/v2.18/docs/reference-global#remote_client_certs_path) and [`remote_client_key_path`](https://www.pantsbuild.org/v2.18/docs/reference-global#remote_client_key_path) options in the `[GLOBAL]` section.
+
+### Backends
+
+Several backends that were intended to be included in previous releases were accidentally missing from the released pants artifacts, these are now available in 2.18:
+
+- `pants.backend.experimental.openapi.codegen.java`: Enables generating Java from OpenAPI.
+- `pants.backend.experimental.python.framework.django`: Enables better support for projects using Django.
+- `pants.backend.experimental.swift`: Enables core Swift support.
+- `pants.backend.python.providers.experimental.pyenv.custom_install`: Enables customising how the pyenv provider builds a Python interpreter.
+
+In addition, the [documentation of backends](https://www.pantsbuild.org/v2.18/docs/enabling-backends) now lists all backends, including preview ones that were not previously advertised. Please go check it out, try backends relevant to you and let us know how they go for your use cases, [report bugs](https://www.pantsbuild.org/v2.18/docs/getting-help) and/or [submit fixes](https://www.pantsbuild.org/v2.18/docs/contributor-overview)!
+
+#### Docker
+
+More options can be passed when building images (see  and  docs for more details:
+
+- on [the `[docker]` subsystem](https://www.pantsbuild.org/v2.18/docs/reference-docker): `build_hosts`, `build_no_cache`
+- on [the `docker_image` target](https://www.pantsbuild.org/v2.18/docs/reference-docker_image): `build_network`, `build_platform`, `extra_build_hosts`
+
+Additional features and bug fixes:
+
+- Registries with the same address no longer silently overwrite each other, and instead now emit an error.
+- [The `secrets` field](https://www.pantsbuild.org/v2.18/docs/reference-docker_image#codesecretscode) now supports expanding `~` to refer to the current user's home directory.
+- [The `image_tags` field](https://www.pantsbuild.org/v2.18/docs/reference-docker_image#codeimage_tagscode) now has an explanatory error if it is incorrectly empty.
+
+#### Helm
+
+[The `helm_unittest_tests` target](https://www.pantsbuild.org/v2.18/docs/helm-overview#helm-unit-tests) has had several improvements:
+
+- support for being added by the `tailor` goal (this can be disabled with the new `[helm].tailor_unittests` option)
+- support for [snapshot testing](https://www.pantsbuild.org/v2.18/docs/helm-overview#snapshot-testing) via the new `generate-snapshots` goal
+
+[The new `pants.backend.experimental.helm.check.kubeconform` backend](https://www.pantsbuild.org/v2.18/docs/helm-kubeconform) allows running Kubeconform on `helm_chart` and `helm_deployment` targets as part of the `check` goal.
+
+**Deprecations**:
+
+- The `helm_deployment` target should now specify the [new `chart` field](https://www.pantsbuild.org/v2.18/docs/reference-helm_deployment#codechartcode) to explicitly indicate which chart is being deployed.
+- The `create_namespace` field for `helm_deployment` will not be supported in future, use the passthrough argument `--create-namespace` instead.
+- The `{env.VAR_NAME}` interpolation syntax some fields of `helm_deployment` has been replaced by the more [general `env` build file function](https://www.pantsbuild.org/v2.18/docs/targets#environment-variables); for instance `"text-{env.VAR_NAME}"` can become `f"text-{env('VAR_NAME')}"`.
+- The `[helm].tailor` option has been renamed to [`tailor_charts`](https://www.pantsbuild.org/v2.18/docs/reference-helm#tailor_charts).
+
+Additional features and bug fixes:
+
+- Relative target addresses for `docker_image`s can be passed like `helm_deployment(values={"some_image": ":target-name"})`, and are expanded in the same way as non-relative addreses `path/to:target-name`.
+- If the passthrough `--debug` argument is specified, like `pants experimental-deploy path/to:helm-deployment -- --debug`, that flag is now passed to all Helm invocations pants does.
+- The Helm binary has been upgraded to 3.12.3 by default.
+- [The new `version` field to `helm_chart`](https://www.pantsbuild.org/v2.18/docs/helm-overview#helm-chart-version) allows overriding the version in `Chart.yaml`.
+
+
+#### JavaScript
+
+The `node_build_script` function now supports `extra_env_vars` to specify additional environment variables when executing.
+
+Node-based tools can now install from a user-provided 'resolve'/`package.json`, similar to Python tools.
+
+The dependency inference for JavaScript has been rewritten into Rust, and thus runs much faster.
+
+#### Python
+
+The functions-as-a-service (FaaS) backends, [`pants.backend.awslambda.python`](https://www.pantsbuild.org/v2.18/docs/awslambda-python) and [`pants.backend.google_cloud_function.python`](https://www.pantsbuild.org/v2.18/docs/google-cloud-function-python) have seen more improvements in 2.18:
+
+- Deprecation: the `python_awslambda` target has been renamed to `python_aws_lambda_function`.
+- Deprecation: the `[lambdex].layout = "zip"` value in `pants.toml` is now the default, and the `[lambdex]` section should be removed.
+- New: [the `python_aws_lambda_layer` target](https://www.pantsbuild.org/v2.18/docs/awslambda-python#building-a-lambda-layer) can be used to build Layers for AWS Lambda.
+- New: the `runtime` field is no longer necessary when the interpreter constraints unambiguously refer to an entire single major version. For instance, with `[python].interpreter_constraints = ["==3.11.*"]`, one doesn't need to specify `runtime="python3.11"` to `python_aws_lambda_function` or `runtime="python311"` to `python_google_cloud_function`, as it will be inferred.
+- New: for AWS Lambda targets, the `complete_platforms` argument now has default values provided by Pants, which gives more reliable builds, especially when building on a different platform than the AWS Lambda itself. This change also means that only one of the `complete_platforms` or `platforms` fields should be provided.
+
+The new, faster parser for dependency inference is now the default, so the `[python-infer].use_rust_parser` option can be removed from `pants.toml`, if it is being explicitly set to `true`. The old parser is deprecated and will be removed: follow the warnings in 2.17 to move to the new parser.
+
+The [new `pants.backend.experimental.python.typecheck.pytype` backend](https://www.pantsbuild.org/v2.18/docs/reference-pytype) allows running the `pytype` type checker as part of the `check` goal.
+
+Additional features and bug fixes:
+
+- The `tailor` goal now supports automatically creating `python_requirements` targets for in `pyproject.toml` files that follow [PEP 621 to specify dependencies](https://peps.python.org/pep-0621/#dependencies-optional-dependencies).
+- Imports within a `with contextlib.suppress(ImportError): ...` construct are now considered weak imports.
+- The global coverage report now correctly handles namespaced packages.
+- `pex_binary` targets that consume `python_distribution` targets now work with just the distributions, not their transitive dependencies (the files that are used to build the distribution).
+- Pants now emits an error if `isort` cannot read its configuration file (NB. `isort` itself emits warnings for this, but they would not be visible when running `isort` under the default Pants configuration).
+- [The `pants.backend.python.typecheck.mypy` backend](https://www.pantsbuild.org/v2.18/docs/python-check-goal) is now more resilient to its cache being removed.
+
+Some deprecations have expired and the supporting code has been removed:
+
+- the "old style" tool lockfiles are no longer supported, and instead customising the versions/installation of tools like `black`, `mypy` and `pytest` should configure [a "normal" resolve](https://www.pantsbuild.org/v2.18/docs/python-lockfiles#lockfiles-for-tools)
+- using `pants export` without a `--resolve` flag is no longer supported, always pass that flag
+
+#### Go
+
+Memory usage has been reduced.
+
+Go processes are now cached using the full version of the Go compiler (for example, `go1.20.6`) so that artifacts are not incorrectly reused between different patch versions.
+
+Building with coverage no longer has an exponential blow-up when preparing the build requests.
+
+#### JVM
+
+The deprecation for the `excludes` field on `jvm_artifact` has expired, use `exclusions` instead.
+
+[The coursier tool](https://www.pantsbuild.org/v2.18/docs/reference-coursier) now uses v2.1.6 by default, and, for this version, supports a native binary on ARM macOS (Apple Silicon).
+
+The `deploy_jar` target now supports [an `exclude_files` field](https://www.pantsbuild.org/v2.18/docs/reference-deploy_jar#codeexclude_filescode).
+
+#### NEW: Rust
+
+The new `pants.backend.experimental.rust` backend has been added, with support for using `rustfmt` to format Rust source files.
+
+#### Scala
+
+The [new `scala_artifact` target](https://www.pantsbuild.org/v2.18/docs/reference-scala_artifact) can be used to track third-party artifacts from Scala, with Pants then understanding how Scala artifacts include the Scala version in their coordinates.
+
+#### Shell
+
+The `shellcheck` binary for [the `pants.backend.shell.lint.shellcheck` backend](https://www.pantsbuild.org/v2.18/docs/shell#shellcheck-linter) is now fetched from `vscode-shellcheck`, and thus now supports AArch64 macOS (Apple Silicon) natively.
+
+The deprecations for the pre-2.16 target names have expired, and thus these have been removed:
+
+- `experimental_shell_command`: use `shell_command`
+- `experimental_run_shell_command`: use `run_shell_command`
+
+#### Terraform
+
+The [new `terraform_deployment` target](https://www.pantsbuild.org/v2.18/docs/reference-terraform_deployment) allows configuring a `terraform_module` to be deployed with [the `experimental-deploy` goal](https://www.pantsbuild.org/v2.18/docs/reference-experimental-deploy).
+
+A new `pants.backend.experimental.terraform.lint.tfsec` backend has been added to run the `tfsec` linter as part of the `lint` goal.
+
+The `check` goal now runs `terraform init` to ensure modules are installed before validating.
+
+#### Visibility
+
+[The `pants.backend.experimental.visibility` backend](https://www.pantsbuild.org/v2.18/docs/validating-dependencies) is now also runs as a linter, such as when running `pants lint ::`, to report all visibility violations at one time.
+
+Visibility rules about dependencies in the root of the repository are now supported.
+
+### Plugin API Changes
+
+Pants no longer publishes wheels to PyPI. The only supported way to depend on pants itself is [the `pants_requirements` target](https://www.pantsbuild.org/v2.18/docs/reference-pants_requirements), which now supports a new `version_spec` field if one needs to depend on a different version of Pants than the currently executing one.
+
+The `include_special_cased_deps` flag (on `DependenciesRequest`, `TransitiveTargetsRequest`, and `CoarsenedTargetsRequest`)  is replaced with `should_traverse_deps_predicate`. If set to `True`, use `should_traverse_deps_predicate=pants.engine.target.AlwaysTraverseDeps()` instead. See [#19272](https://github.com/pantsbuild/pants/pull/19272) and [#19387](https://github.com/pantsbuild/pants/pull/19387) for details.
+
+The `TestResult` `stdout` and `stderr` fields have been renamed to `stdout_bytes` and `stderr_bytes`, and store `bytes` instead of `str`, to handle non-UTF-8 data without crashing.
+
+Other changes:
+
+- Plugins can now declare additional requirements in a `requirements.txt` file next to `register.py`.
+- Pants only supports running on Python 3.9 now.
+- Named caches can now used digits in addition to lower-case letters and underscores.
+- The `pants.core.goals.package.TraverseIfNotPackageTarget` type can be used with the new `should_traverse_deps_predicate` to treat packageable targets as "leaves" when finding dependencies; for instance, stopping at a `pex_binary` without also surfacing the `python_sources` that are used for it.
+
+The deprecations for many request types has expired (follow instructions in warnings in 2.17 to resolve), so they are now removed: `AllAssetTargetsRequest`, `AllTargetsRequest`, `BashBinaryRequest`, `CatBinaryRequest`, `CGoCompilerWrapperScriptRequest`, `ChmodBinaryRequest`, `DiffBinaryRequest`, `DjangoAppsRequest`, `DockerBinaryRequest`, `GitBinaryRequest`, `GunzipBinaryRequest`, `MkdirBinaryRequest`, `PythonBinary`, `ReadlinkBinaryRequest`, `TarBinaryRequest`, `UnzipBinaryRequest`, `ZipBinaryRequest`.
 
 ## Full Changelog