Add docs for jvm_artifacts target generator (Cherry-pick of #20890)
#20916
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eploy` goal (pantsbuild#20488) Changed the following things to achieve this - Add a `--dry-run` flag to the `experimental-deploy` goal to handle dry runs such as `terraform plan` - Run `plan` rather than `apply` when setting the `--dry-run` flag while deploying a `terraform_deployment` - Changed Helm deployments to use the new `--dry-run` flag instead of a passthrough arg for dry-running Tested - the terraform change via a unit test and also tried it out manually in my own project. - the helm change via a unit test. Closes pantsbuild#18490
Related issues and discussions: * pantsbuild#20548 * https://pantsbuild.slack.com/archives/C0D7TNJHL/p1708092220006959
…antsbuild#20554) When using go 1.22, trying to compile a go package using pants fails with: ``` 13:07:42.25 [INFO] Initialization options changed: reinitializing scheduler... 13:07:46.62 [INFO] Scheduler initialized. 13:07:51.89 [ERROR] Completed: Compile with Go - runtime/internal/syscall - runtime/internal/syscall failed (exit code 1). flag provided but not defined: -compiling-runtime usage: asm [options] file.s ... Flags: -D value predefined symbol with optional simple value -D=identifier=value; can be set multiple times -I value include directory; can be set multiple times -S print assembly and machine code -V print version and exit -d value enable debugging settings; try -d help -debug dump instructions as they are parsed -dynlink support references to Go symbols defined in other shared libraries -e no limit on number of errors reported -gensymabis write symbol ABI information to output file, don't assemble -linkshared generate code that will be linked against Go shared libraries -o string output file; default foo.o for /a/b/c/foo.s as first argument -p string set expected package import to path (default "<unlinkable>") -shared generate code that can be linked into a shared library -spectre list enable spectre mitigations in list (all, ret) -trimpath string remove prefix from recorded source file paths -v print debug output ``` This flag has been removed from version 1.22 - asm can now make the determination itself regarding whether or not we're compiling a `runtime` package, so there's no need to pass that flag anymore. See: https://cs.opensource.google/go/go/+/72946ae8674a295e7485982fe57c65c7142b2c14
Previously the top-level menu was called "Java and Scala" and "Kotlin" was a subpage. Now the top-level is called "JVM" and there are two subpages - "Java and Scala", and "Kotlin".
Previously the options sources list could have only one source per type (flag/env/config/default). Since we support multiple config files, we had the concept of a "merged config" that encompassed multiple "config sources". This led to duplication of the logic that traverses and merges sources. This change gets rid of the concept of merged config. Instead, the list of options sources we consult can now include multiple config files. This is achieved by enhancing the Source::Config enum to include an ordinal (to ensure that configs are applied in the right order) and the path to the config file (purely informational). The natural `#[derive(Ord)]` will order these augmented configs appropriately within all the sources. To make this easier to achieve we change the order of the enum to be from lowest to highest priority, so that we can add config files naturally in the order in which we encounter them. This also allows us to simplify some types. In particular, we no longer need to traffic in `Vec<DictEdit>` - a single source can only return a single `DictEdit`. More importantly, this supports derivation history that includes the specific config file each value comes from.
Upgrade to Rust v1.76.0. [Blog post](https://blog.rust-lang.org/2024/02/08/Rust-1.76.0.html)
Closes pantsbuild#20576. Although this is a relatively small change, I think it's good to understand the change in context so I've included the new and old logic below for comparison. New logic: ```py formatter_to_request_class: dict[Formatter, type[RewrittenBuildFileRequest]] = { Formatter.BLACK: FormatWithBlackRequest, Formatter.YAPF: FormatWithYapfRequest, Formatter.RUFF: FormatWithRuffRequest, } chosen_formatter_request_class = formatter_to_request_class.get( update_build_files_subsystem.formatter ) if not chosen_formatter_request_class: raise ValueError(f"Unrecognized formatter: {update_build_files_subsystem.formatter}") for request in union_membership[RewrittenBuildFileRequest]: if update_build_files_subsystem.fmt and request == chosen_formatter_request_class: rewrite_request_classes.append(request) if update_build_files_subsystem.fix_safe_deprecations and issubclass( request, DeprecationFixerRequest ): rewrite_request_classes.append(request) # If there are other types of requests that aren't the standard formatter # backends or deprecation fixers, add them here. if request not in formatter_to_request_class.values() and not issubclass( request, DeprecationFixerRequest ): rewrite_request_classes.append(request) ``` Old logic: ```py for request in union_membership[RewrittenBuildFileRequest]: if issubclass(request, (FormatWithBlackRequest, FormatWithYapfRequest)): is_chosen_formatter = issubclass(request, FormatWithBlackRequest) ^ ( update_build_files_subsystem.formatter == Formatter.YAPF ) if update_build_files_subsystem.fmt and is_chosen_formatter: rewrite_request_classes.append(request) else: continue if update_build_files_subsystem.fix_safe_deprecations or not issubclass( request, DeprecationFixerRequest ): rewrite_request_classes.append(request) ``` The `else: continue` in the old logic was load-bearing, because it would skip over the second top-level `if` statement and move to the next loop iteration in cases where the request class was one of the regular formatters (`FormatWithBlackRequest` or `FormatWithYapfRequest`). The `or not issubclass(request, DeprecationFixerRequest)` was intended to catch formatters that live outside of the list of formatter backends and deprecation fixers - the last `if` statement in the new logic is intended to cover this case as well.
Add docs to support the newly added `--format` option for the `dependencies` and `dependents` goals which are in 2.20.
This PR fixes pantsbuild#20427 by updating our use of `doc_url` to reflect the new website structure. It does this semi-automatically, spread across separate commits: 1. Update the `doc_url` function to reflect the new website structure, including support for unversioned URLs like https://www.pantsbuild.org/community/getting-help 2. Apply automated rewrites (see below for details) 3. Do one or two manual fix-ups The automatic rewrites were done by: 1. finding the args passed to `doc_url`: `rg --only-matching "doc_url\([^)]*\)" . | cut -f2 -d\( | cut -f1 -d\) | tr \' \" | sort -u` 2. put them into CSV file, then use https://github.com/pantsbuild/pantsbuild.org/blob/92d5ce8d3442890c3fbeb0fade620b762ae2aa7e/old_site_redirects.js to find the new URL for each one 3. transform the CSV into a long series of `s@...@...@` substitutions for `sed` 4. apply them with `git ls-files '*.py' '*.rs' | xargs sed -i '' "..."` 5. do a basic check that the appropriate files that call `doc_url` were updated: identify non-updated files via `comm -13 <(git diff --name-only be42bc0^...be42bc0) <( rg --files-with-matches doc_url | sort)`. The files without updates are expected, e.g. - `src/python/pants/util/docutil.py` that defines `doc_url` itself (and the test file etc.) - `src/python/pants/backend/codegen/avro/target_types.py` that didn't have an immediately obvious replacement, with more examples in pantsbuild#20584 This is marked for cherry-picking to 2.20.x. There _are_ redirects for the URLs for 2.20, but it seems a bit annoying to rely on them, if we don't have to. <details> ```csv old,new "getting-help","community/getting-help" "advanced-target-selection","docs/using-pants/advanced-target-selection" "anonymous-telemetry","docs/using-pants/anonymous-telemetry" "awslambda-python","docs/python/integrations/aws-lambda" "enabling-backends","docs/using-pants/key-concepts/backends" "google-cloud-function-python","docs/python/integrations/google-cloud-functions" "installation","docs/getting-started/installing-pants" "macros","docs/writing-plugins/macros" "options","docs/using-pants/key-concepts/options" "options#addremove-semantics","docs/using-pants/key-concepts/options#addremove-semantics" "options#config-file-entries","docs/using-pants/key-concepts/options#config-file-entries" "options#config-file-interpolation","docs/using-pants/key-concepts/options#config-file-interpolation" "options#pantsrc-file","docs/using-pants/key-concepts/options#pantsrc-file" "options#reading-individual-option-values-from-files","docs/using-pants/key-concepts/options#reading-individual-option-values-from-files" "pex","docs/python/overview/pex" "plugin-upgrade-guide","docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide" "plugins-overview","docs/writing-plugins/overview" "plugins-setup-py","docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs" "protobuf-go","docs/go/integrations/protobuf" "protobuf-python","docs/python/integrations/protobuf-and-grpc" "python-check-goal","docs/python/goals/check" "python-distributions","docs/python/overview/building-distributions" "python-interpreter-compatibility","docs/python/overview/interpreter-compatibility" "python-lockfiles#lockfiles-for-tools","docs/python/overview/lockfiles#lockfiles-for-tools" "python-test-goal","docs/python/goals/test" "python-test-goal#pytest-version-and-plugins","docs/python/goals/test#pytest-version-and-plugins" "python-third-party-dependencies","docs/python/overview/third-party-dependencies" "python-third-party-dependencies#local-requirements","docs/python/overview/third-party-dependencies#local-requirements" "python-third-party-dependencies#multiple-lockfiles","docs/python/overview/lockfiles#multiple-lockfiles" "python-third-party-dependencies#user-lockfiles","docs/python/overview/third-party-dependencies#user-lockfiles" "reference-deploy_jar#coderesolvecode","reference/targets/deploy_jar#resolve" "reference-global#remote_oauth_bearer_token","reference/global-options#remote_oauth_bearer_token" "remote-caching-execution","docs/using-pants/remote-caching-and-execution" "source-roots","docs/using-pants/key-concepts/source-roots" "tagging-docker-images","docs/docker/tagging-docker-images" "target-api-concepts","docs/writing-plugins/the-target-api/concepts" "targets","docs/using-pants/key-concepts/targets-and-build-files" "targets#field-default-values","docs/using-pants/key-concepts/targets-and-build-files#field-default-values" "targets#dependencies-and-dependency-inference","docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference" "thrift-python","docs/python/integrations/thrift" "troubleshooting","docs/using-pants/troubleshooting-common-issues" "troubleshooting#import-errors-and-missing-dependencies","docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies" "upgrade-tips","docs/releases/upgrade-tips" ``` ```python import pandas as pd print("\n".join( f"""s@\\(doc_url.*[\\"']\\){r.old}\\([\\"']\\)@\\1{r.new}\\2@g;""" for _, r in pd.read_csv("replacements.csv").iterrows() )) ``` ```shell git ls-files '*.py' '*.rs' | xargs sed -i '' " s@\(doc_url.*[\"']\)getting-help\([\"']\)@\1community/getting-help\2@g; s@\(doc_url.*[\"']\)advanced-target-selection\([\"']\)@\1docs/using-pants/advanced-target-selection\2@g; s@\(doc_url.*[\"']\)anonymous-telemetry\([\"']\)@\1docs/using-pants/anonymous-telemetry\2@g; s@\(doc_url.*[\"']\)awslambda-python\([\"']\)@\1docs/python/integrations/aws-lambda\2@g; s@\(doc_url.*[\"']\)enabling-backends\([\"']\)@\1docs/using-pants/key-concepts/backends\2@g; s@\(doc_url.*[\"']\)google-cloud-function-python\([\"']\)@\1docs/python/integrations/google-cloud-functions\2@g; s@\(doc_url.*[\"']\)installation\([\"']\)@\1docs/getting-started/installing-pants\2@g; s@\(doc_url.*[\"']\)macros\([\"']\)@\1docs/writing-plugins/macros\2@g; s@\(doc_url.*[\"']\)options\([\"']\)@\1docs/using-pants/key-concepts/options\2@g; s@\(doc_url.*[\"']\)options#addremove-semantics\([\"']\)@\1docs/using-pants/key-concepts/options#addremove-semantics\2@g; s@\(doc_url.*[\"']\)options#config-file-entries\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-entries\2@g; s@\(doc_url.*[\"']\)options#config-file-interpolation\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-interpolation\2@g; s@\(doc_url.*[\"']\)options#pantsrc-file\([\"']\)@\1docs/using-pants/key-concepts/options#pantsrc-file\2@g; s@\(doc_url.*[\"']\)options#reading-individual-option-values-from-files\([\"']\)@\1docs/using-pants/key-concepts/options#reading-individual-option-values-from-files\2@g; s@\(doc_url.*[\"']\)pex\([\"']\)@\1docs/python/overview/pex\2@g; s@\(doc_url.*[\"']\)plugin-upgrade-guide\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide\2@g; s@\(doc_url.*[\"']\)plugins-overview\([\"']\)@\1docs/writing-plugins/overview\2@g; s@\(doc_url.*[\"']\)plugins-setup-py\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs\2@g; s@\(doc_url.*[\"']\)protobuf-go\([\"']\)@\1docs/go/integrations/protobuf\2@g; s@\(doc_url.*[\"']\)protobuf-python\([\"']\)@\1docs/python/integrations/protobuf-and-grpc\2@g; s@\(doc_url.*[\"']\)python-check-goal\([\"']\)@\1docs/python/goals/check\2@g; s@\(doc_url.*[\"']\)python-distributions\([\"']\)@\1docs/python/overview/building-distributions\2@g; s@\(doc_url.*[\"']\)python-interpreter-compatibility\([\"']\)@\1docs/python/overview/interpreter-compatibility\2@g; s@\(doc_url.*[\"']\)python-lockfiles#lockfiles-for-tools\([\"']\)@\1docs/python/overview/lockfiles#lockfiles-for-tools\2@g; s@\(doc_url.*[\"']\)python-test-goal\([\"']\)@\1docs/python/goals/test\2@g; s@\(doc_url.*[\"']\)python-test-goal#pytest-version-and-plugins\([\"']\)@\1docs/python/goals/test#pytest-version-and-plugins\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies\([\"']\)@\1docs/python/overview/third-party-dependencies\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies#local-requirements\([\"']\)@\1docs/python/overview/third-party-dependencies#local-requirements\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies#multiple-lockfiles\([\"']\)@\1docs/python/overview/lockfiles#multiple-lockfiles\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies#user-lockfiles\([\"']\)@\1docs/python/overview/third-party-dependencies#user-lockfiles\2@g; s@\(doc_url.*[\"']\)reference-deploy_jar#coderesolvecode\([\"']\)@\1reference/targets/deploy_jar#resolve\2@g; s@\(doc_url.*[\"']\)reference-global#remote_oauth_bearer_token\([\"']\)@\1reference/global-options#remote_oauth_bearer_token\2@g; s@\(doc_url.*[\"']\)remote-caching-execution\([\"']\)@\1docs/using-pants/remote-caching-and-execution\2@g; s@\(doc_url.*[\"']\)source-roots\([\"']\)@\1docs/using-pants/key-concepts/source-roots\2@g; s@\(doc_url.*[\"']\)tagging-docker-images\([\"']\)@\1docs/docker/tagging-docker-images\2@g; s@\(doc_url.*[\"']\)target-api-concepts\([\"']\)@\1docs/writing-plugins/the-target-api/concepts\2@g; s@\(doc_url.*[\"']\)targets\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files\2@g; s@\(doc_url.*[\"']\)targets#field-default-values\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#field-default-values\2@g; s@\(doc_url.*[\"']\)targets#dependencies-and-dependency-inference\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference\2@g; s@\(doc_url.*[\"']\)thrift-python\([\"']\)@\1docs/python/integrations/thrift\2@g; s@\(doc_url.*[\"']\)troubleshooting\([\"']\)@\1docs/using-pants/troubleshooting-common-issues\2@g; s@\(doc_url.*[\"']\)troubleshooting#import-errors-and-missing-dependencies\([\"']\)@\1docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies\2@g; s@\(doc_url.*[\"']\)upgrade-tips\([\"']\)@\1docs/releases/upgrade-tips\2@g; " ``` </details>
Add support for `pants package` using yarn@v2. Currently when trying to package a yarn@v2 project, the following error occurs: ``` Engine traceback: in `package` goal ProcessExecutionFailure: Process 'Installing [email protected].' failed with exit code 1. stdout: ➤ YN0050: The --frozen-lockfile option is deprecated; use --immutable and/or --immutable-cache instead stderr: ``` Thats because the option `--frozen-lockfile` was replaced with `--immutable` on yarn >= 2. This behavior can be reproducible at https://github.com/kevinobruno/pants-yarn running `pants package //yarnv2`
Changelogs: * https://github.com/pex-tool/pex/releases/tag/v2.1.164 * https://github.com/pex-tool/pex/releases/tag/v2.2.0 * https://github.com/pex-tool/pex/releases/tag/v2.2.1 (New home, new docs, but no direct Pants related changes.) ``` Lockfile diff: 3rdparty/python/user_reqs.lock [python-default] == Upgraded dependencies == cryptography 42.0.2 --> 42.0.3 pex 2.1.163 --> 2.2.1 urllib3 2.2.0 --> 2.2.1 ```
…antsbuild#20513) This was motivated by trying to add some additional module and stub-related mappings. While doing that, I realized the patterns only applied to implementation modules only, and realized that a refactor would be the easiest way forward, while also making the logic easier to step through. I haven't added any of those module mappings in this PR since I wanted this to only be the logic change. ## Changes * Make the replacement functions more generic (allow injecting any separator) * Better type hinting - more complete types for generics, also made return values more specific (return values and variables should be hinted as specifically as possible; functions can choose to accept more generic types as arguments) * Made impl and type package mappings behave identically * The previous logic seemed to have potential for bugs - it does a fallback on each call, has some hackiness around checking for a stubs package, nested conditions, etc. Simplified the module lookup logic. Now it's just: * check if it's in the standard module mapping * else check if it's in the type stub module mapping * else check if the patterns for type stub modules match - this needs to be done first because a stub module might match an implementation module pattern (example, `python-pkg-types`) * else check if the patterns for impl modules match * else use the fallback * Update tests to ensure symmetry between standard modules and type stub modules * Added new test for `test_generate_type_stub_mappings_from_pattern_matches_para`, which is the main new logic here * All other changes created by `pants fmt` ## Testing * Added and modified doctest examples, and `python src/python/pants/backend/python/dependency_inference/default_module_mapping.py` passes * `pants test src/python/pants/backend/python/dependency_inference/::` passes * Ran this build on our monorepo, which uses quite a tangle of every feature here * CI
This is a short-term workaround for pantsbuild#20577 and pantsbuild#20586, where Pants' internal/default use of Pex triggers user-visible warnings, and those warnings are now visible due to pantsbuild#20480... so, instead of showing them by default, let's hide them for now. Users with desire for insight into the warnings can still enable this. Doing this means we're not having to rush in fixes for the root causes of these warnings for 2.20.0 stable release, and thus reduce feature-creep/risk. We can/should reenable warnings by default in a future release. Per @cburroughs's idea in pantsbuild#20586 (comment), this explicitly switches it on for the Pants repo itself, so we're eating our own dogfood and catching real and/or spurious errors earlier.
…ation. (pantsbuild#20574) As discussed in pantsbuild#20572, this change adds a built-in goal `migrate-call-by-name` which emits all `Get` calls which should be migrated to call-by-name syntax. A followup change should use the resulting information (`@rule` function pointers and `Get` signatures) to execute rewrites of the relevant files. Emits lines like: ``` <function infer_python_conftest_dependencies at 0x113633af0> Get(<class 'pants.engine.target.Targets'>, [<class 'pants.engine.addresses.Addresses'>]) -> <function resolve_targets at 0x112335d30> Get(<class 'pants.engine.internals.graph.Owners'>, [<class 'pants.engine.internals.graph.OwnersRequest'>]) -> <function find_owners at 0x112349160> Get(<class 'pants.backend.python.util_rules.ancestor_files.AncestorFiles'>, [<class 'pants.backend.python.util_rules.ancestor_files.AncestorFilesRequest'>]) -> <function find_ancestor_files at 0x1131e6790> ```
Implemented for `fix` and `lint` goals.
… to target location (pantsbuild#20581) This PR fixes pantsbuild#20575 by adjusting the shared infrastructure for `adhoc_tool` and `code_quality_tool` to resolve the `execution_dependencies` field relative to the `adhoc_tool`/`code_quality_tool` target location, rather than relative to the `runnable=...` arg. I think this was a minor typo in pantsbuild#20255, and we didn't have tests covering it. I imagine it's common that the runnable and `..._tool` targets are often in the same BUILD file (in which case the behaviour is the same either way), but it's not impossible to have them be different (e.g. my work repo has a a few shared runnable that are used by more than one `adhoc_tool`). I think being relative to the target is easier to reason about, and this was the old behaviour of `adhoc_tool`. This PR also adds tests to confirm the behaviour of `execution_dependencies` (including its relative-path resolution behaviour), as well as the behaviour of `runnable_dependencies` while I'm there.
) The `python_awslambda` target was replaced by `python_aws_lambda_function` in pantsbuild#19216, first released in 2.18.0.dev1 (the deprecation was extended in pantsbuild#20101). We're now up to 2.21 (pantsbuild#20609), and thus the deprecation has expired and this can be removed. This has been deprecated for 3 versions, and is easy for users to adapt to, so doesn't seem worth extending.
Title says it all. Just adding a new default module mapping. Signed-off-by: Jason Barnett <[email protected]>
…n=partial (pantsbuild#20616) There were three deprecations originally scheduled for 2.21 (pantsbuild#20609), two of will have only been deprecated for one release, and aren't a significant burden to support, so we can ease users life for a little longer: | description | deprecation started in | what could be removed | |------------------------------------------------------|------------------------|-------------------------------------------------------------------------------------------------| | the `[GLOBAL].remote_oauth_bearer_token_path` option | 2.20.0.dev1 (pantsbuild#20116) | configuration for the option, plus some a few small `if` statements, all in `global_options.py` | | passing `crossversion="partial"` for scala artifacts | 2.20.0 (pantsbuild#20264) ^ | an extra enum variant, and a test | ^ NB. due to an easy-to-make misuse of `start_version` I believe people will only be getting the deprecation warnings in 2.20.0 stable, not any of the release candidates. The third (`python_awslambda`) is removed in pantsbuild#20619.
Previously to pass along only/no-binary options Pants would create a
"requirements" file and use Pip's ability to put cli args in said file
[1]. This worked fine to generate the artifacts in a lockfile from
scratch, but meant that the lockfile itself did not contain sufficient
information to carry its own configuration forward. That is if one did
`pex3 lock update` on a Pants created lockfile that originally required
everything to use wheels, that specification could not be preserved in
the updated lockfile. Since v2.1.161 [2] Pex has provided
`--only-{wheel,build}` options to support this directly.
[1]
https://pip.pypa.io/en/stable/reference/requirements-file-format/#global-options
[2] https://github.com/pex-tool/pex/releases/tag/v2.1.161
ref pantsbuild#15704
`During a release, the contents of this directory is synced with our website repo (https://github.com/pantsbuild/pantsbuild.org) to be deployed.` Does this mean it's okay to edit the docs file in either the pants repo or the pantsbuild.rog repo?
Closes pantsbuild#20620. Added tests for both new fields; I couldn't figure out how to get the deprecated alias `skip_ruff` to still work in tests. I had another version of this change which kept the `skip_ruff` behavior as it is now to skip **both** linting and formatting; if that's a preferred solution, I can tack on those changes to this branch.
…ip_version = 24.0, for Py 3.12 support (pantsbuild#20365) This does three things to fix pantsbuild#20354 and give Pants Python 3.12 support by default: - update the default `[python].pip_version` value to `24.0` (from `23.1.2`), which is the current latest and is after 23.2, which is the first that supports Python 3.12 - run `pants run build-support/bin/generate_builtin_lockfiles.py -- --all-python` to update all tool lockfiles... this involves updating many tools, as a Big Bang, which is potentially unhelpful. - to make this work, `docformatter` has to be restricted to the current version 1.4: 1.5.1 (the newest version that satisfies the old constraints) crashes on Pants itself (PyCQA/docformatter#151), while 1.7.5 (the newest version) also doesn't work on Pants (pantsbuild#20498) Here's a list of (as best I can tell), the changes in versions for the "main" requirement for each subsystem: | subsystem options scope | main requirement | old version | new version | |-------------------------|------------------|-------------|-------------| | `helm-k8s-parser` | `hikaru` | 0.11.0b0 | 0.16.0b0 | | `helm-post-renderer` | `yamlpath` | 3.7.0 | 3.8.1 | | `autoflake` | `autoflake` | 2.0.1 | 2.1.1 | | `bandit` | `bandit` | 1.7.4 | 1.7.5 | | `black` | `black` | 23.1.0 | 23.3.0 | | `pyupgrade` | `pyupgrade` | 3.3.1 | 3.3.2 | | `ruff` | `ruff` | 0.2.1 | 0.2.2 | | `yapf` | `yapf` | 0.32.0 | 0.40.2 | | `coverage-py` | `coverage` | 7.2.1 | 7.2.7 | | `debugpy` | `debugpy` | 1.6.6 | 1.6.7.post1 | | `mypy` | `mypy` | 1.1.1 | 1.4.1 | | `terraform-hcl2-parser` | `python-hcl2` | 4.3.0 | 4.3.2 | | `yamllint` | `yamllint` | 1.29.0 | 1.32.0 | There's numerous other transitive dependencies updated too, including in subsystems for which the main requirement hasn't changed and aren't listed above. I haven't checked changelogs for any of these. I've confirmed that running the reproducer from pantsbuild#20354 with this code now works. The commits are 'sensible'.
…ntsbuild#20789) Because it's a (not) common use-case for people to deploy google cloud functions via something like ``` gcloud functions deploy abc --trigger-http --runtime=python311 --entry-point=handler ``` When omitted the `--source` flag will default to the current directory. This is often easier than uploading to an explicit bucket and specifying `--source=gs://bucket`
…es found.However, ..." (pantsbuild#20861) Currently, running `pants update-build-files ::` prints a message that's slightly misformatted, where there's no space after the first sentence. This adds a space: Before: ``` 15:27:34.28 [INFO] No required changes to BUILD files found.However, there may still be deprecations that `update-build-files` doesn't know how to fix. See https://www.pantsbuild.org/2.22/docs/releases/upgrade-tips for upgrade tips. ``` After: ``` 15:29:00.90 [INFO] No required changes to BUILD files found. However, there may still be deprecations that `update-build-files` doesn't know how to fix. See https://www.pantsbuild.org/2.22/docs/releases/upgrade-tips for upgrade tips. ```
This adds 3.8.0 as a known version of shfmt and bumps to that version by default, from 3.6.0 previously. It's easy enough to retain the old versions as known versions, so they're kept. Changelogs: - https://github.com/mvdan/sh/releases/tag/v3.7.0 - https://github.com/mvdan/sh/releases/tag/v3.8.0
If the scope name is foo.bar, the env var should start with PANTS_FOO_BAR_, as is done in the Python parser. In practice dots aren't allowed in scope names, and a dotted scope indicates "nested subsystems" which aren't a thing any more. But some aspects of this are still tested in the Python parser, and we want those tests to continue to pass under the Rust parser. This PR also changes the return value of get_passthrough_args to one that turns out to be more useful in practice, but which wasn't worth an independent PR.
Previously we only used DEFAULT for seeding ${placeholder}s,
but we are also supposed to fall back to reading from it, when
the value doesn't exist in the specific section, as the Python
parser does.
Issues were exposed by running the tests under the Rust options parser.
As discussed in the previous [Pants Team Meetup](https://docs.google.com/document/d/1LhGXW2kwaV4u1uqz_GvYe9fbeQilJLbCJ49kGxBOKe4/edit?pli=1), we think that the stable release process requires some dedicated attention, so we'd like to choose a dedicated stable release manager per stable release. For now, the selection process is completely manual and adhoc (because not everyone in the MOTW rotation will necessarily want to volunteer for a longer-term engagement). But the stable release manager will probably always be a member of that rotation. --------- Co-authored-by: Huon Wilson <[email protected]>
Previously it required actual files on disk. This change also includes a minor modification to the equivalent logic in the Python parser, to prepare for connecting the two: The _ConfigValues struct now remembers the ConfigSource it was parsed from, instead of just the path.
Various changes needed to support access to the Rust parser from Python code. These include: 1. Modeling option source "rank", using the same enum values as the Python model. 2. Making the OptionsParser concurrency-friendly (making it Send + Sync, and using Arc instead of Rc). 3. Plumbing passthrough args in. 4. Having get_list()'s `default` be Vec instead of array. 5. Ensuring that `--string-list-option=` is interpreted as adding an empty string to the list, rather than setting it to an empty list, and ensuring that for other list types, `--foo=` is an error. This is how the Python options parser treats these cases, so we must comply with that. The actual Python bindings will be provided in a future PR.
A follow up change will wire this up on the Python side.
<img width="920" alt="CleanShot 2024-05-07 at 22 58 11@2x" src="https://github.com/pantsbuild/pants/assets/31057849/993f681a-76fd-4dda-aa85-62f5eebb064b"> If you access the current link, you will be directed to a document that cannot be found as follows. I think it was probably intended to link to the latest document.
This starts a `2.22.x.md` for work-in-progress features on the `main` branch. This is preparation for pantsbuild#20850: once that's in, we'll be maintaining this file as we go, so it helps to seed it. Currently includes everything up to b75eb07. pantsbuild/pants@release_2.21.0a0...b75eb07 --------- Co-authored-by: Andreas Stenius <[email protected]>
This is CI in service of continually maintaining our detailed changelog, i.e. have long form descriptions of unreleased features listed in the "latest" `docs/notes/2.*.x.md` file, so we don't have to write this in a batch as part of prepping the release. The detailed changelog for a big release is a better for users than listing PR titles (as we do for each incremental dev release), because it gives space for linking to docs, giving code examples etc. This PR adds CI to enforce that every PR, either: - changes a `docs/notes/` file, usually in the form of adding a paragraph or two to `docs/notes/2.22.x.md` (or similar, whatever the latest `main` release is) - explicitly opts out via a label (`category:internal` or `release-notes:not-required`) This PR tries to be a minimal MVP of a process improvement. The key bit of this CI is enforcing an _explicit_ opt-out: we can say "let's maintain a changelog as we go" without this PR, but I think we'll forget to do it all the time, if we don't have a computer reminding us. ## Example of how this works in practice ### Feature or fix worth including in release notes 1. Someone opens a PR 2. The release notes CI step fails 3. The author or reviewer notices this and suggests that release notes be included in the appropriate `docs/notes/2.*.x.md` file 4. The PR is updated with release notes, CI reruns and passes now ### Minor change not worth including in release notes 1. Someone opens a PR 2. The release notes CI step fails 3. The author or reviewer notices this and a maintainer tags the PR with `release-notes:not-required` 4. The release notes CI step can be rerun and passes now ### Demo of error Demonstration of a failure when this PR was tagged with `category:new feature` (instead of `category:internal`): https://github.com/pantsbuild/pants/actions/runs/8905804902/job/24457010553?pr=20850 ## Problems There's a few ways this can go wrong: 1. someone adds notes to the wrong file by mistake: up to a reviewer to catch, but we could add active checks for it too (i.e. we could be explicitly enforcing that there should be changes to the _current_ release notes). 1. if a PR has been open for a long time, it might make changes to the `2.22.x.md` file, but only merge in time for `2.23.x`, and thus the release notes are targetting the wrong series. There's a few potential mitigations for this (I think 1 is likely good enough for now): 1. when branching for a new release, the person doing is now told to go and check open PRs for changes to the wrong release notes file and ask the author to update, with a script 2. if merged via a merge queue (e.g. pantsbuild#20193), a CI check for 1 would catch this too 7. this'll make cherry-picking more awkward and fail more often, e.g. if a change has release note in `2.22.x.md` and we cherry pick it to the `2.21.x` branch, that file won't exist and the cherry picker will fail. I think this is probably tolerable for now (at least, worth trying). There's a potential few mitigations (I think the first is the better use of energy): 1. switching to a different approach to notes where they end up in separate dedicated files (see "Future work") 2. making the cherry-picker script more intelligent, e.g. ignore notes or copy them across magically ## Initial roll-out We're early in the 2.22.x dev cycle on `main`. I've whipped up a `2.22.x.md` notes file with everything that's landed before this in pantsbuild#20878, and then future PRs will augment that. Once we branch for 2.22.x and start on 2.23.x we'll just need to make the `2.23.x.md` file then, and keep going as we were. The existing stable branches (2.19.x, 2.20.x, 2.21.x) will continue with the old process, release notes in a batch. ## Future work There's further iteration we can do on this: - We may want to eventually consider a tool like towncrier (as discussed pantsbuild#19247), especially if we're getting a lot of merge conflicts on the release notes files, since that'll aggregate the notes across separate files that don't interact between PRs. - Putting these human-curated notes into the GitHub release announcements, rather than the PR titles. However, for either of those, I think we'll still (conceptually) want enforcement along these lines: a PR needs to either have release notes included (in whatever format we land on), or a human needs to explicitly opt-out. Thus, this is an incremental step that won't be wasted work.
Upgrade Rust to v1.78.0. [Release notes](https://blog.rust-lang.org/2024/05/02/Rust-1.78.0.html).
An old branch I decided to dust off and finish. 🧹 The OpenAPI targets were a great addition to allow futher OpenAPI-related functionality in Pants (codegen, linters, etc.), but I failed to dogfood them for my own use case. My problem with the current implementation is that it's hard to use an `openapi_document` as a dependency of a `python_source` due to it behaving more or less like a `file()` would. At work we currently have all our API descriptions as `resource()` so they're bundled with the Python code that uses them for request validation and whatnot. With `openapi_document` this would require you to manually specify your `openapi_document` and all its `openapi_source` dependencies as inputs to `experimental_wrap_as_resources`, which is annoying when the OpenAPI targets all have dependency inference. This PR uses codegen and `redocly bundle` to bundle an `openapi_document` and its `openapi_source`s into a single file when the `openapi_document` is needed as a dependency by another target. This allow developers to keep their multi-file approach when writing the API description, existing OpenAPI backends will (should?) continue to work as-is, but when consuming it elsewhere as a dependency they now only have to care about one single OpenAPI file. I initially wrote this code back when `experimental_wrap_as_resources` didn't exist and `openapi_document` was (in our case) unusable, which is why I made it part of the main OpenAPI backend (and thus enforced). Nowadays you can, as I mentioned, likely get work around the issue with `experimental_wrap_as_resources` - so maybe it's better to move this functionality into its own, opt-in, backend? That being said, resolving `$ref`s in a multi-file approach is _tricky_ to get right, which is why a lot of OpenAPI tools don't even bother supporting it (or behave differently), so Pants "always" passing a single file around could be seen as a good thing. 🤷 Edit: `redocly bundle` could also be useful as a package goal to prepare the API definition for distribution by dereferencing includes, stripping `x-internal` paths etc., but that's a separate feature.
This change: A) Provides Python bindings to the Rust options parser. B) Integrates the Rust options parser into the options parsing flow. Currently, the existing legacy parser and the native parser are both consulted, and their results compared, with discrepancies logged. This will allow us to evaluate and fix such discrepancies during a transition period.
This makes a few minor adjustments to the docs for the `py_editable_in_resolve` and `py_hermetic_scripts` options to `pants export ...`: - add some indentation so that `softwrap` preserves the crucial newlines in a Markdown list - backticks around more things for code formatting - minor typo Fixes pantsbuild#20893
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add docs for the
jvm_artifactstarget implemented in #20336