Merge pull request #6758 from TheThingsNetwork/merge/v3.28-in-v3.29-os

Merge v3.28 in v3.29

.github/actions/install-go-and-deps/action.yml

@@ -4,7 +4,7 @@ inputs:
   go-version:
     description: Go Version
     required: true
-    default: "~1.20"
+    default: "~1.21"
 runs:
   using: composite
   steps:

.github/workflows/release-tag.yml

@@ -14,12 +14,12 @@ jobs:
       id-token: write
       packages: write
       pull-requests: write
-    timeout-minutes: 60
+    timeout-minutes: 90
     steps:
       - name: Maximize build space
         uses: easimon/maximize-build-space@v8
         with:
-          root-reserve-mb: 20480
+          root-reserve-mb: 30720
           swap-size-mb: 4096
           remove-dotnet: true
           remove-android: true

.goreleaser.release.yml

@@ -120,10 +120,14 @@ snapcrafts:
         plugs: [home, network, network-bind]
         command: ttn-lw-stack.wrapper
         completer: config/completion/bash/ttn-lw-stack
+        environment:
+          XDG_CACHE_HOME: $SNAP_USER_COMMON
       ttn-lw-cli:
         plugs: [home, network, network-bind]
         command: ttn-lw-cli.wrapper
         completer: config/completion/bash/ttn-lw-cli-snap
+        environment:
+          XDG_CACHE_HOME: $SNAP_USER_COMMON
 
 brews:
   - name: ttn-lw-stack

CHANGELOG.md

@@ -7,7 +7,50 @@ with the exception that this project **does not** follow Semantic Versioning.
 
 For details about compatibility between different releases, see the **Commitments and Releases** section of our README.
 
-## [3.28.0] - unreleased
+## [Unreleased]
+
+### Added
+
+- Rate limiting classes for individual HTTP paths.
+- Rate limiting keys for HTTP endpoints now contain the caller API key ID when available. The caller IP is still available as a fallback.
+- Allow users to set multiple frequency plans only in the same band in the Console.
+
+### Changed
+
+- Server side events replaced with single socket connection using the native WebSocket API.
+- Gateways now disconnect if the Gateway Server address has changed.
+  - This enables CUPS-enabled gateways to change their LNS before the periodic CUPS lookup occurs.
+- The LoRa Basics Station discovery endpoint now verifies the authorization credentials of the caller.
+  - This enables the gateways to migrate to another instance gracefully while using CUPS.
+
+### Deprecated
+
+### Removed
+
+### Fixed
+
+- Batch gateway rights assertions when multiple membership chains are available (for example, both via a user and an organization).
+
+### Security
+
+## [3.28.1] - 2023-11-27
+
+### Added
+
+- The `http.client.transport.compression` experimental flag. It controls whether the HTTP clients used by the stack support gzip and zstd decompression of server responses. It is enabled by default.
+- The `http.server.transport.compression` experimental flag. It controls whether the HTTP servers used by the stack support gzip compression of the server response. It is enabled by default.
+
+### Changed
+
+- The Things Stack is now built with Go 1.21.
+- Statistics for gateways are now fetched in a single request.
+
+### Fixed
+
+- Resolve scroll jumps when selecting different tabs of a table in the Console.
+- `BatchGetGatewayConnectionStats` RPC rights check in certain cases.
+
+## [3.28.0] - 2023-10-31
 
 ### Added
 
@@ -16,25 +59,27 @@ For details about compatibility between different releases, see the **Commitment
 - It is now possible to trigger a resending of the email validation email from within the Console. The new action is part of the error screen that users see when they log into the Console without having their contact info validated yet (and the network requires validation before usage).
 - Updated Japanese translations for the Console and backend.
 - `--grpc.correlation-ids-ignore-methods` configuration option, which allows certain gRPC methods to be skipped from the correlation ID middleware which adds a correlation ID with the name of the gRPC method. Methods bear the format used by `--grpc.log-ignore-methods`, such as `/ttn.lorawan.v3.GsNs/HandleUplink`.
+- Support for setting multiple frequency plans for gateways from the Console.
+- The `ns-db purge` command to purge unused data from the Network Server database.
 
 ### Changed
 
 - Users can now request a new email for the account validation from time to time instead of once per validation, the interval between email requests is determined by `is.user-registration.contact-info-validation.retry-interval` and by default it is an hour.
 - Traffic related correlation IDs have been simplified. Previously one correlation ID per component was added as traffic passed through different stack components. Now a singular correlation ID relating to the entry point of the message will be added (such as `gs:uplink:xxx` for uplinks, or `as:downlink:xxx` for downlinks), and subsequent components will no longer add any extra correlation IDs (such as `ns:uplink:xxx` or `as:up:xxx`). The uplink entry points are `pba` and `gs`, while the downlink entry points are `pba`, `ns` and `as`.
 - Packet Broker Agent uplink tokens are now serialized in a more efficient manner.
-
-### Deprecated
-
-### Removed
+- The Network Server now stores only the most recent uplinks tokens.
+- The Application Server webhook health system now records failures only every retry interval while in monitor mode, as opposed to recording every failure.
+  - Monitor mode in this context refers to situations in which either `--as.webhooks.unhealthy-attempts-threshold` or `--as.webhooks.unhealthy-retry-interval` are less or equal to zero. In such situations, the Application Server will record failures but will not stop the execution of the webhooks.
+  - Using a retry interval of zero and a non zero attempts threshold restores the previous behavior.
 
 ### Fixed
 
 - Providing fixed downlink paths to the `ttn-lw-cli devices downlink {push|replace}` commands using the `-class-b-c.gateways` parameter. The gateways IDs are comma separated, and the antenna index `i` can be provided by suffixing the ID with `:i` (i.e. `my-gateway:0` for antenna index 0). The group index `j` can be provided by suffixing the ID with `:j` (i.e. `my-gateway:0:1` for antenna index 0 and group index 1). The antenna index is mandatory if a group index is to be provided, but optional otherwise.
 - Gateway registration without gateway EUI not working.
 - Listing deleted entities is now fixed for both admin and standard users, which previously returned an `account_not_found` error.
 - Update to an user's `PrimaryEmailAddress` via a non admin now invalidates the `PrimaryEmailAddressValidatedAt` as it was intended.
-
-### Security
+- Negative number support in Cayenne LPP.
+- Fix panic in snapcraft CLI deployment, commands will no longer generate a panic error message when telemetry is enabled.
 
 ## [3.27.2] - 2023-09-14
 
@@ -2688,7 +2733,9 @@ For details about compatibility between different releases, see the **Commitment
 NOTE: These links should respect backports. See https://github.com/TheThingsNetwork/lorawan-stack/pull/1444/files#r333379706.
 -->
 
-[unreleased]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.27.2...v3.27
+[unreleased]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.28.1...v3.28
+[3.28.1]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.28.0...v3.28.1
+[3.28.0]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.27.2...v3.28.0
 [3.27.2]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.27.1...v3.27.2
 [3.27.1]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.27.0...v3.27.1
 [3.27.0]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.26.2...v3.27.0

CODEOWNERS

@@ -1,97 +1,97 @@
 # Root
-/** @johanstokking @adriansmares
-/CODEOWNERS @johanstokking
-/LICENSE @johanstokking
+/** @TheThingsNetwork/stack-reviewers-1
+/CODEOWNERS @TheThingsNetwork/stack-reviewers-2
+/LICENSE @TheThingsNetwork/stack-reviewers-2
 /CHANGELOG.md
 
 # Clear common folders
 /pkg
 
 # Dependencies
-/go.mod @johanstokking @adriansmares
-/go.sum @johanstokking @adriansmares
-/package.json @kschiffer @ryaplots
-/yarn.lock @kschiffer @ryaplots @mjamescompton
-/sdk/js/package.json @kschiffer @ryaplots @mjamescompton
-/sdk/js/yarn.lock @kschiffer @ryaplots @mjamescompton
+/go.mod @TheThingsNetwork/stack-reviewers-1
+/go.sum @TheThingsNetwork/stack-reviewers-1
+/package.json @TheThingsNetwork/stack-reviewers-3
+/yarn.lock @TheThingsNetwork/stack-reviewers-3
+/sdk/js/package.json @TheThingsNetwork/stack-reviewers-3
+/sdk/js/yarn.lock @TheThingsNetwork/stack-reviewers-3
 
 # Build pipeline and tooling
-/Makefile @adriansmares
-/tools @adriansmares
-/.github/workflows @adriansmares
-/config @kschiffer @ryaplots @mjamescompton
-/config/stack @benolayinka @KrishnaIyer
-/cypress @kschiffer @ryaplots @mjamescompton
+/Makefile @TheThingsNetwork/stack-reviewers-4
+/tools @TheThingsNetwork/stack-reviewers-4
+/.github/workflows @TheThingsNetwork/stack-reviewers-4
+/config @TheThingsNetwork/stack-reviewers-3
+/config/stack @TheThingsNetwork/stack-reviewers-5
+/cypress @TheThingsNetwork/stack-reviewers-3
 
 # API, configuration, CLI and storage compatibility
-/api/*.proto @johanstokking @adriansmares
-/cmd @adriansmares
-/pkg/**/config.go @adriansmares
-/pkg/**/registry.go @adriansmares
-/pkg/config/shared.go @adriansmares
-/pkg/goproto @johanstokking @adriansmares
-/pkg/identityserver/store @johanstokking @adriansmares
-/pkg/jsonpb @johanstokking @adriansmares
-/pkg/ttnpb @johanstokking @adriansmares
+/api/*.proto @TheThingsNetwork/stack-reviewers-1
+/cmd @TheThingsNetwork/stack-reviewers-4
+/pkg/**/config.go @TheThingsNetwork/stack-reviewers-4
+/pkg/**/registry.go @TheThingsNetwork/stack-reviewers-4
+/pkg/config/shared.go @TheThingsNetwork/stack-reviewers-4
+/pkg/goproto @TheThingsNetwork/stack-reviewers-1
+/pkg/identityserver/store @TheThingsNetwork/stack-reviewers-1
+/pkg/jsonpb @TheThingsNetwork/stack-reviewers-1
+/pkg/ttnpb @TheThingsNetwork/stack-reviewers-1
 
 # Security
-/pkg/auth @johanstokking @adriansmares
-/pkg/crypto @johanstokking @adriansmares
+/pkg/auth @TheThingsNetwork/stack-reviewers-1
+/pkg/crypto @TheThingsNetwork/stack-reviewers-1
 
 # Shared components
-/pkg/band @johanstokking @KrishnaIyer
-/pkg/cluster @johanstokking @adriansmares
-/pkg/component @johanstokking @adriansmares
-/pkg/errors @johanstokking @adriansmares
-/pkg/events @adriansmares
-/pkg/frequencyplans @johanstokking @KrishnaIyer
-/pkg/interop @johanstokking @adriansmares
-/pkg/rpcmiddleware @johanstokking @adriansmares
-/pkg/rpcserver @johanstokking @adriansmares
-/pkg/types @johanstokking
-/pkg/web @kschiffer @ryaplots @mjamescompton
-/pkg/webmiddleware @KrishnaIyer @kschiffer
-/pkg/webui @kschiffer @ryaplots @mjamescompton
-/pkg/ratelimit @johanstokking @KrishnaIyer
+/pkg/band @TheThingsNetwork/stack-reviewers-7
+/pkg/cluster @TheThingsNetwork/stack-reviewers-1
+/pkg/component @TheThingsNetwork/stack-reviewers-1
+/pkg/errors @TheThingsNetwork/stack-reviewers-1
+/pkg/events @TheThingsNetwork/stack-reviewers-4
+/pkg/frequencyplans @TheThingsNetwork/stack-reviewers-7
+/pkg/interop @TheThingsNetwork/stack-reviewers-1
+/pkg/rpcmiddleware @TheThingsNetwork/stack-reviewers-1
+/pkg/rpcserver @TheThingsNetwork/stack-reviewers-1
+/pkg/types @TheThingsNetwork/stack-reviewers-2
+/pkg/web @TheThingsNetwork/stack-reviewers-3
+/pkg/webmiddleware @TheThingsNetwork/stack-reviewers-8
+/pkg/webui @TheThingsNetwork/stack-reviewers-3
+/pkg/ratelimit @TheThingsNetwork/stack-reviewers-7
 
 # Subsystems
 
-/pkg/identityserver @adriansmares @nicholaspcr
-/pkg/email @kschiffer @nicholaspcr
-/pkg/oauth @kschiffer @nicholaspcr
-/pkg/account @kschiffer @nicholaspcr
-/pkg/web/oauthclient @kschiffer @nicholaspcr
+/pkg/identityserver @TheThingsNetwork/stack-reviewers-9
+/pkg/email @TheThingsNetwork/stack-reviewers-10
+/pkg/oauth @TheThingsNetwork/stack-reviewers-10
+/pkg/account @TheThingsNetwork/stack-reviewers-10
+/pkg/web/oauthclient @TheThingsNetwork/stack-reviewers-10
 
-/pkg/gatewayserver @johanstokking @KrishnaIyer @adriansmares
-/pkg/gatewayconfigurationserver @KrishnaIyer @adriansmares
-/pkg/pfconfig @KrishnaIyer @adriansmares
-/pkg/basicstation @KrishnaIyer @adriansmares
+/pkg/gatewayserver @TheThingsNetwork/stack-reviewers-11
+/pkg/gatewayconfigurationserver @TheThingsNetwork/stack-reviewers-12
+/pkg/pfconfig @TheThingsNetwork/stack-reviewers-12
+/pkg/basicstation @TheThingsNetwork/stack-reviewers-12
 
-/pkg/networkserver @johanstokking @adriansmares
-/pkg/specification/macspec @johanstokking @adriansmares
-/pkg/encoding/lorawan @johanstokking @adriansmares
+/pkg/networkserver @TheThingsNetwork/stack-reviewers-1
+/pkg/specification/macspec @TheThingsNetwork/stack-reviewers-1
+/pkg/encoding/lorawan @TheThingsNetwork/stack-reviewers-1
 
-/pkg/applicationserver @johanstokking @adriansmares @KrishnaIyer
-/pkg/messageprocessors @johanstokking @adriansmares @KrishnaIyer
-/pkg/scripting @johanstokking @adriansmares @KrishnaIyer
+/pkg/applicationserver @TheThingsNetwork/stack-reviewers-11
+/pkg/messageprocessors @TheThingsNetwork/stack-reviewers-11
+/pkg/scripting @TheThingsNetwork/stack-reviewers-11
 
-/pkg/joinserver @johanstokking @adriansmares
+/pkg/joinserver @TheThingsNetwork/stack-reviewers-1
 
-/pkg/console @kschiffer @ryaplots @mjamescompton
+/pkg/console @TheThingsNetwork/stack-reviewers-3
 
-/pkg/packetbrokeragent @johanstokking @adriansmares
+/pkg/packetbrokeragent @TheThingsNetwork/stack-reviewers-1
 
-/pkg/devicetemplateconverter @adriansmares @KrishnaIyer
-/pkg/devicetemplates @adriansmares @KrishnaIyer
-/pkg/qrcode @adriansmares @KrishnaIyer
-/pkg/qrcodegenerator @adriansmares @KrishnaIyer
+/pkg/devicetemplateconverter  @TheThingsNetwork/stack-reviewers-12
+/pkg/devicetemplates  @TheThingsNetwork/stack-reviewers-12
+/pkg/qrcode  @TheThingsNetwork/stack-reviewers-12
+/pkg/qrcodegenerator  @TheThingsNetwork/stack-reviewers-12
 
-/pkg/deviceclaimingserver @nicholaspcr @KrishnaIyer
+/pkg/deviceclaimingserver @TheThingsNetwork/stack-reviewers-13
 
-/pkg/devicerepository @adriansmares @KrishnaIyer
+/pkg/devicerepository  @TheThingsNetwork/stack-reviewers-12
 
 # SDKs
-/sdk/js @kschiffer @ryaplots @mjamescompton
+/sdk/js @TheThingsNetwork/stack-reviewers-3
 
 # Clear generated code
 /api/api.md

CONTRIBUTING.md

@@ -2,11 +2,11 @@
 
 Thank you for your interest in building this thing together with us. We're really happy with our active community and are glad that you're a part of it. There are many ways to contribute to our project, but given the fact that you're on Github looking at the code for The Things Stack for LoRaWAN, you're probably here for one of the following reasons:
 
-- **Asking a question**: If you have questions, please use the [forum](https://www.thethingsnetwork.org/forum/). We have a special [category for The Things Stack](https://www.thethingsnetwork.org/forum/c/network-and-routing/v3).
+- **Asking a question**: If you have questions, please use the [forum](https://www.thethingsnetwork.org/forum/). We have a special [category for The Things Stack](https://www.thethingsnetwork.org/forum/c/ttn-network/v3/90).
 - **Requesting a new feature**: If you have a great idea or think some functionality is missing, we want to know! The only thing you have to do for that is to [create an issue](https://github.com/TheThingsNetwork/lorawan-stack/issues) if it doesn't exist yet. Please use the issue template and fill out all sections.
 - **Reporting an issue**: If you notice that a component of The Things Stack is not behaving as it should, there may be a bug in our systems. In this case you should [create an issue](https://github.com/TheThingsNetwork/lorawan-stack/issues) if it doesn't exist yet. Please use the issue template and fill out all sections. For sensitive (security) issues, you can [contact us directly](#security-issues).
 - **Implementing a new feature or fixing a bug**: If you see an [open issue](https://github.com/TheThingsNetwork/lorawan-stack/issues) that you would like to work on, let the other contributors know by commenting in the issue.
-- **Writing documentation**: If you see that our documentation is lacking or incorrect, it would be great if you could help us improve it. This will help users and fellow contributors understand how to better work with our stack. Better documentation helps prevent making mistakes and introducing new bugs. Our documentation is spread across a number of places. Code documentation obviously lives together with the code, and is therefore probably in this repository. User documentation for The Things Stack that is published on [thethingsstack.io](https://thethingsstack.io), is built from the [`lorawan-stack-docs` repository]https://github.com/TheThingsIndustries/lorawan-stack-docs. More general documentation can be found on [The Things Network's official documentation pages](https://www.thethingsnetwork.org/docs). The source files for that documentation can be found in [the `docs` repository](https://github.com/TheThingsNetwork/docs).
+- **Writing documentation**: If you see that our documentation is lacking or incorrect, it would be great if you could help us improve it. This will help users and fellow contributors understand how to better work with our stack. Better documentation helps prevent making mistakes and introducing new bugs. Our documentation is spread across a number of places. Code documentation obviously lives together with the code, and is therefore probably in this repository. User documentation that is published on [the official The Things Stack documentation page](https://thethingsstack.io), is built from the [`lorawan-stack-docs` repository](https://github.com/TheThingsIndustries/lorawan-stack-docs). More general documentation can be found on [The Things Network's official documentation pages](https://www.thethingsnetwork.org/docs). The source files for that documentation can be found in [the `docs` repository](https://github.com/TheThingsNetwork/docs).
 
 If you'd like to contribute by writing code, you'll find [here](DEVELOPMENT.md) how to set up your development environment.
 

DEVELOPMENT.md

@@ -675,7 +675,7 @@ To decide whether a component is a container component, ask yourself:
 - Is this component more concerned with how things work, rather than how things look?
 - Does this component connect to the store?
 - Does this component fetch or send data?
-- Is the component generated by higher order components (e.g. `withFeatureRequirement`)?
+- Is the component generated by higher order components?
 - Does this component render simple nodes, like a single presentational component?
 
 If you can answer more than 2 questions with yes, then you likely have a container component.
@@ -692,7 +692,6 @@ View components always represent a single view of the application, represented b
 - Fetching necessary data (via `withRequest` HOC), if not done by a container
 - Unavailable "catch-all"-routes are caught by `<NotFoundRoute />` component, including subviews
 - Errors should be caught by the `<ErrorView />` error boundary component
-- `withFeatureRequirement` HOC is used to prevent access to routes that the user has no rights for
 - Ensured responsiveness and usage of the grid system
 
 #### Utility components