release-7145

What's Changed

• pageserver: refuse to load tenants with suspiciously old indices in old generations by @jcsp in #9719
• Improve error handling for NeonAPI fixture by @tristan957 in #9751
• safekeeper: fix atomicity of WAL truncation by @arssher in #9685
• cargo: use neon branch of rust-postgres by @VladLazar in #9757
• Correctly truncate VM by @knizhnik in #9342
• test: disable test_readonly_node_gc until proper fix by @yliang412 in #9755
• Remove the replication slot in test_snap_files at the end of the test by @tristan957 in #9752
• tests: add test_timeline_archival_chaos by @jcsp in #9609
• storcon: exclude non-Active tenants from shard autosplitting by @jcsp in #9743
• deny.toml: allow CDDL-1.0 license by @erikgrinaker in #9766
• tests: tolerate pageserver 500s in test_timeline_archival_chaos by @jcsp in #9769
• Correct mistakes in offloaded timeline retain_lsn management by @arpad-m in #9760
• Collect max_connections metric by @myrrc in #9770
• wal_decoder: make InterpretedWalRecord serde by @VladLazar in #9775
• Fix PG_MAJORVERSION_NUM typo by @tristan957 in #9774
• wal_decoder: rename end_lsn to next_record_lsn by @VladLazar in #9776
• safekeeper: send AppendResponse on segment flush by @erikgrinaker in #9692
• Cargo.toml: upgrade tikv-jemallocator to 0.6.0 by @erikgrinaker in #9779
• build(deps): bump smallvec to 1.13.2 to get UB fix by @problame in #9781
• Storage & Compute release 2024-11-18 by @vipvap in #9783

New Contributors

• @myrrc made their first contribution in #9770

Full Changelog: release-7125...release-7145

release-7125

What's Changed

• storcon: add peer token for peer to peer communication by @VladLazar in #9695
• CI: retry actions/github-script for 5XX errors by @bayandin in #9703
• fix(pageserver): increase frozen layer warning threshold; ignore in tests by @skyzh in #9705
• rfc: update aux file rfc to reflect latest optimizations by @skyzh in #9681
• safekeeper: add initialize_segment variant of safekeeper_wal_storage_operation_seconds by @erikgrinaker in #9691
• Remove checklist from the PR template by @bayandin in #9702
• new ingest benchmark by @Bodobolero in #9711
• test_runner: skip more tests using decorator instead of pytest.skip by @bayandin in #9704
• Clean up C.UTF-8 locale changes by @tristan957 in #9714
• feat(pageserver): support partial gc-compaction for delta layers by @skyzh in #9611
• use batch gh-workflow-stats-action with separate table by @fedordikarev in #9722
• Add a postgres_exporter config file by @tristan957 in #9720
• Add a retain_lsn test by @arpad-m in #9599
• Improve typing related to regress/test_logical_replication.py by @tristan957 in #9725
• Update pg_cron to 1.6.4 by @tristan957 in #9726
• safekeeper: don't flush control file on WAL ingest path by @erikgrinaker in #9698
• test(pageserver): add gc-compaction tests with delta will_init by @skyzh in #9724
• utils: remove unnecessary fsync in durable_rename() by @erikgrinaker in #9686
• Calculate compute_backpressure_throttling_seconds correctly by @tristan957 in #9732
• Add neon.logical_replication_max_logicalsnapdir_size by @tristan957 in #9467
• safekeeper: use WAL_SEGMENT_SIZE for empty timeline state by @erikgrinaker in #9734
• Fix autocommit footguns in performance tests by @tristan957 in #9735
• Do not wallog AUX files at replica by @knizhnik in #9457
• pageserver: add no_sync for use in regression tests (2/2) by @jcsp in #9678
• Add installed_extensions prometheus metric by @lubennikovaav in #9608
• tests: Increase timeout in test_create_churn_during_restart by @hlinnaka in #9736
• PostgreSQL minor version updates (17.1, 16.5, 15.9, 14.14) by @hlinnaka in #9727
• Compute release 2024-11-13 by @ololobus in #9745

Full Changelog: release-7097...release-7125

release-7097

What's Changed

• cherry-pick Clean up C.UTF-8 locale changes by @skyzh in #9721

Full Changelog: release-7095...release-7097

release-7095

What's Changed

• safekeeper: add walreceiver metrics by @erikgrinaker in #9450
• fix(test_regress): always use storcon virtual pageserver API to set tenant config by @problame in #9622
• quieten aws_config logs by @conradludgate in #9626
• proxy: Wrap JWT errors in separate AuthError variant by @cloneable in #9625
• proxy: Rewrite ControlPlaneEvent as enum by @cloneable in #9627
• Add tenant config option to allow timeline_offloading by @arpad-m in #9598
• Improve some typing related to performance testing for LR by @tristan957 in #9629
• Update psycopg2 to latest version for complete PG 17 support by @tristan957 in #9632
• pageserver: return 503 during tenant shutdown by @erikgrinaker in #9635
• Construct models::TenantConfig only once by @arpad-m in #9630
• scrubber: remove AWS region assumption, tolerate negative max_project_size by @jcsp in #9636
• proxy: Unify local and remote conn pool client structures by @awarus in #9604
• safekeeper: don't allocate send buffers on stack by @erikgrinaker in #9644
• [proxy] use the proxy protocol v2 command to silence some logs by @conradludgate in #9620
• proxy: Revert ControlPlaneEvent back to struct by @cloneable in #9649
• proxy: Refactor cplane types by @cloneable in #9643
• proxy: Read cplane JWT with clap arg by @cloneable in #9654
• proxy: Replace "web(auth)" with "console redirect" everywhere by @cloneable in #9655
• pageserver: ingest pre-serialized batches of values by @VladLazar in #9579
• storcon: respect tenant scheduling policy in drain/fill by @jcsp in #9657
• [proxy]: minor changes to endpoint-cache handling by @conradludgate in #9666
• feat(pageserver): add fast path for sparse keyspace read by @skyzh in #9631
• Rename compute_backpressure_throttling_ms to compute_backpressure_throttling_seconds by @tristan957 in #9668
• fix(test): use layer map dump in test_readonly_node_gc to validate layers protected by leases by @yliang412 in #9551
• refactor(pageserver): better pageservice command parsing by @skyzh in #9597
• Support copying layers in detach_ancestor from before shard splits by @arpad-m in #9669
• chore(compute): Bump pg_mooncake to the latest version by @dpxcc in #9672
• postgres_ffi: make WalGenerator generic over record generator by @erikgrinaker in #9614
• Fix direct reading from WAL buffers. by @arssher in #9639
• safekeeper: add basic WAL ingestion benchmarks by @erikgrinaker in #9531
• safekeeper: use set_len() to zero out segments by @erikgrinaker in #9665
• fix(pageserver): reduce aux compaction threshold by @skyzh in #9647
• Don't attach is_archived to debug output by @arpad-m in #9679
• [proxy/authorize]: improve JWKS reliability by @conradludgate in #9676
• add pgcopydb tool to build tools image by @Bodobolero in #9658
• Disable rust_2024_compatibility lint option by @hlinnaka in #9615
• ci: use set-docker-config-dir from dev-actions by @jcgruenhage in #9638
• pageserver: add no_sync for use in regression tests (1/2) by @jcsp in #9677
• Do not copy logical replicaiton slots to replica by @knizhnik in #9458
• Update TimescaleDB to 2.17.1 for PG 17 by @tristan957 in #9640
• fix(pageserver): drain upload queue before detaching ancestor by @skyzh in #9651
• tests: use tigther storcon scopes by @VladLazar in #9696
• CI: move check codestyle python to reusable workflow and run on a merge_group by @bayandin in #9683
• Switch compute-related locales to C.UTF-8 by default by @tristan957 in #6746
• feat(pageserver): more log lines around frozen layers by @skyzh in #9697
• safekeeper: remove unused WriteGuardSharedState::skip_update by @erikgrinaker in #9699
• fix(pageserver): drain upload queue before offloading timeline by @skyzh in #9682
• Improve type safety according to pyright by @tristan957 in #9633
• CI(pre-merge-checks): add required checks by @bayandin in #9700
• Storage & Compute release 2024-11-11 by @vipvap in #9710

New Contributors

• @dpxcc made their first contribution in #9672
• @jcgruenhage made their first contribution in #9638

Full Changelog: release-7044...release-7095

release-7044

What's Changed

• Add --replica parameter to basebackup by @knizhnik in #9553
• feat(pageserver): support partial gc-compaction for lowest retain lsn by @skyzh in #9134
• fix(pagectl): layer parsing and image layer dump by @skyzh in #9571
• Add tests for sql-exporter metrics by @tristan957 in #9184
• pageserver: generation-aware storage for TenantManifest by @jcsp in #9555
• Remove non-gzipped basebackup code path by @tristan957 in #9453
• chore(compute): Bump pg_mooncake to the latest version by @ololobus in #9576
• postgres_ffi: add WAL generator for tests/benchmarks by @erikgrinaker in #9503
• Collect compute backpressure throttling time by @tristan957 in #9176
• Disallow archived timelines to be detached or reparented by @arpad-m in #9578
• Don't keep around the timeline's remote_client by @arpad-m in #9583
• refactor(pageserver): use JSON field encoding for consumption metrics cache by @skyzh in #9470
• Disallow offloaded children during timeline deletion by @arpad-m in #9582
• pageserver: lift decoding and interpreting of wal into wal_decoder by @VladLazar in #9524
• pageserver: don't return option from DeletionQueue::new by @erikgrinaker in #9588
• Add support of extensions for v17 (part 4) by @lubennikovaav in #9568
• increase lifetime of AWS session token to 12 hours by @Bodobolero in #9590
• pageserver: fix spurious error logs in timeline lifecycle by @jcsp in #9589
• auth_broker: fix local_proxy conn count by @conradludgate in #9593
• update pg_session_jwt to use pgrx 0.12 for pg17 by @conradludgate in #9595
• storage_controller: allow deployment infra to use infra token by @VladLazar in #9596
• fix pre-commit hook with python stubs by @conradludgate in #9602
• test_runner: fix tenant_get_shards with one pageserver by @erikgrinaker in #9603
• revert higher token expiration by @Bodobolero in #9605
• safekeeper: log slow WalAcceptor sends by @erikgrinaker in #9564
• storcon: skip non-active projects in chaos injection by @jcsp in #9606
• Do not assign page LSN to new (uninitialized) page in ClearVisibilityMapFlags redo handler by @knizhnik in #9287
• test_runner/performance: add sharded ingest benchmark by @erikgrinaker in #9591
• pageserver: make local timeline deletion infallible by @jcsp in #9594
• Fix a time zone issue in PG17 test case by @MMeent in #9618
• remove aws-lc-rs dep and fix storage_broker tls by @conradludgate in #9613
• [proxy]: parse proxy protocol TLVs with aws/azure support by @conradludgate in #9610
• Storage & Compute release 2024-11-04 by @vipvap in #9617

Full Changelog: release-7011...release-7044

release-7011

What's Changed

• Compute hotfix release 2024-10-30 by @ololobus in #9577

Full Changelog: release-7009...release-7011

release-7009

What's Changed

• build(deps): bump werkzeug from 3.0.3 to 3.0.6 by @dependabot in #9527
• Don't consider archived timelines for synthetic size calculation by @arpad-m in #9497
• safekeeper: don't un-evict timelines during snapshot API handler by @jcsp in #9428
• storcon: add an API to cancel ongoing reconciler by @jcsp in #9520
• tests: add test_storage_controller_onboard_detached by @jcsp in #9431
• Route pgbouncer logs to virtio-serial by @petuhovskiy in #9488
• ci(proxy): Update GH action flag on proxy deployment by @rahulinux in #9535
• [proxy] demote warnings and remove dead-argument by @conradludgate in #9512
• Add pg_mooncake to compute-node.Dockerfile by @tristan957 in #9498
• We don't have legacy proxies anymore by @SergeyMelnikov in #9544
• safekeeper: refactor WalAcceptor to be event-driven by @erikgrinaker in #9462
• refactor(pageserver): remove aux v1 configs by @skyzh in #9494
• fix(pageserver): bump tokio-epoll-uring by @skyzh in #9546
• Build pgrag extensions x3 by @jawj in #8486
• Create the notion of unstable extensions by @knizhnik in #9540
• Get role name when not a C string by @tristan957 in #9526
• Assert the tenant to be active in unoffload_timeline by @arpad-m in #9539
• pageserver: move things around to prepare for decoding logic by @VladLazar in #9504
• Offloaded timeline deletion by @arpad-m in #9519
• temporarily increase timeout for clickbench benchmark until regression is resolved by @Bodobolero in #9554
• [proxy]: fix JWT handling for AWS cognito. by @conradludgate in #9536
• [auth_broker]: regress test by @conradludgate in #9541
• pageserver: refactor generation-aware loading code into generic by @jcsp in #9545
• pageserver: make concurrent offloaded timeline operations safe wrt manifest uploads by @jcsp in #9557
• fix allow_unstable_extensions GUC - make it USERSET, by @lubennikovaav in #9563
• proxy: more granular http status codes for sql-over-http errors by @mrl5 in #9549
• Use pg_mooncake from our fork. by @lubennikovaav in #9565
• Compute release 2024-10-29 by @lubennikovaav in #9573

Full Changelog: release-6981...release-7009

release-6981

What's Changed

• proxy: Make tests fail fast when test proxy exited early by @cloneable in #9432
• Rename dockerfiles from Dockerfile.<something> to <something>.Dockerfile by @bayandin in #9446
• CI: use build-tools on Debian 12 whenever we use Neon artifact by @bayandin in #9463
• Use '-f' for make clean in Makefile compute by @awarus in #9464
• proxy: Use common error interface for error handling with cplane by @awarus in #9454
• fix(pageserver): make image split layer writer finish atomic by @skyzh in #8841
• pageserver: use direct IO for delta and image layer reads by @yliang412 in #9326
• offloaded timeline list API by @arpad-m in #9461
• chore(compute): bumps pg_session_jwt to latest version by @davidgomes in #9474
• safekeeper: refactor timeline initialization by @arssher in #9362
• Use make's builtin RM variable by @tristan957 in #9468
• storage controller: add metrics for tenant shard, node count by @jcsp in #9475
• Fix local errors for the tests with the versions mix by @a-masterov in #9477
• Fix copy-paste error in checkpoints_timed metric by @tristan957 in #9483
• Timeline offloading persistence by @arpad-m in #9444
• fix(pageserver): make delta split layer writer finish atomic by @skyzh in #9048
• Rename IndexPart::{from_s3_bytes,to_s3_bytes} by @arpad-m in #9481
• proxy: clear lib.rs of code items by @cloneable in #9479
• Protect call to pg_current_wal_lsn() in retained_wal query by @tristan957 in #9425
• tests: avoid using background_process when invoking pg_ctl by @jcsp in #9469
• pageserver: add metric for number of zeroed pages on rel extend by @VladLazar in #9492
• Fix buggy sizeof by @tristan957 in #9495
• refactor(pageserver(load_remote_timeline)) remove dead code handling absence of IndexPart by @problame in #9408
• storcon: respect Reconciler::cancel during await_lsn by @jcsp in #9486
• CI: remove git config --add safe.directory hack by @bayandin in #9391
• refactor(pageserver): refactor split writers using batch layer writer by @skyzh in #9493
• pageserver: refactor ingest inplace to decouple decoding and handling by @VladLazar in #9472
• refactor(timeline creation): represent bootstrap vs branch using enum by @problame in #9366
• pageserver/walingest: log on gappy rel extend by @VladLazar in #9502
• Support offloaded timelines during shard split by @arpad-m in #9489
• fix(test): remove too strict layers_removed==0 check in test_readonly_node_gc by @yliang412 in #9506
• proxy: don't follow redirects for user provided JWKS urls + set custom user agent by @mrl5 in #9514
• pageserver: add testing API to scan layers for disposable keys by @problame in #9393
• remote_storage(local_fs): return correct file sizes by @problame in #9511
• Make test_pageserver_lsn_wait_error_safekeeper_stop less aggressive. by @arssher in #9517
• Move logical replication slot monitor by @tristan957 in #9496
• refactor(timeline creation): idempotency checking by @problame in #9501
• pageserver: fix N^2 I/O when processing relation drops in transaction abort by @jcsp in #9507
• proxy: propagate session IDs by @conradludgate in #9509
• safekeeper: retry eviction. by @arssher in #9485
• safekeeper: fix version in TimelinePersistentState::empty() by @erikgrinaker in #9521
• safekeeper: don't pass conf into storage constructors by @erikgrinaker in #9523
• Fix unoffload_timeline races with creation by @arpad-m in #9525
• pageserver: add tokio-epoll-uring slots waiters queue depth metrics by @yliang412 in #9482
• Storage & Compute release 2024-10-25 by @vipvap in #9528

Full Changelog: release-6935...release-6981

release-6935

What's Changed

• Compute release 2024-10-21 by @tristan957 in #9476

Full Changelog: release-6933...release-6935

release-6933

What's Changed

• feat(pageserver): do space check before gc-compaction by @skyzh in #9250
• storage_controller: delete stale shards when deleting tenant by @erikgrinaker in #9333
• safekeeper: flush WAL on compute disconnect by @erikgrinaker in #9436
• Add logging of installed_extensions by @lubennikovaav in #9438
• refactor(pageserver): remove aux v1 code path by @skyzh in #9424
• pageserver: detect & warn on loading an old index which is probably the result of a bad generation by @jcsp in #9383
• Update Rust to 1.82.0 and mold to 2.34.0 by @arpad-m in #9445
• update rustls by @conradludgate in #9396
• 2024 oct new clippy lints by @conradludgate in #9448
• compute_ctl: Add endpoint that allows setting role grants by @devjv in #9395
• storcon: fix case where we might fail to send compute notifications after two opposite migrations by @jcsp in #9435
• Increase shared_buffers in test_subscriber_synchronous_commit. by @arssher in #9427
• Update ruff to much newer version by @cloneable in #9433
• compute_ctl: Add endpoint that allows extensions to be installed by @devjv in #9344
• walproposer: immediately exit if sync-safekeepers collected 0/0. by @arssher in #9442
• pageserver: disconnect when asking client to reconnect by @VladLazar in #9390
• [local_proxy]: install pg_session_jwt extension on demand by @conradludgate in #9370
• storcon: handle ongoing deletions gracefully by @VladLazar in #9449
• Take the collector name as argument when generating sql_exporter configs by @tristan957 in #9447
• Accept basebackup --gzip requests by @arpad-m in #9456
• bump pg-session-jwt version by @conradludgate in #9455
• Compute release 2024-10-20 by @ololobus in #9459

Full Changelog: release-6911...release-6933