Skip to content

Commit

Permalink
Release 4.4.0 (2024-11-14)
Browse files Browse the repository at this point in the history 
### Features

* Add consented debug logs to seller KV V2
* Add Data Version for buyer to generateBid() browser signals
* add egress schema bucket fetcher
* add egress schema fetch manager
* add flag https_fetch_skips_tls_verification for local testing
* Add priority fields to B&A API
* Add PV support on BFE
* Add PV support on SFE
* Add Seller Data Version to ReportResult
* Add Seller DV to Bid Metadata for ScoreAd()
* Add support for DNS routing by regional LB latency in AWS
* Add support for DNS routing by regional LB latency in AWS
* add top level seller debugging data
* Data Version in BuyerReporting
* enable seller debugging for eligible request
* Implement delete model rpc for inference sidecar
* Implement KV V2 Adapter for Scoring Signals
* Implement KV V2 flag for seller
* Implement v2 bidding support - config flags propagation
* improve SFE response debug logging
* Launch the B&A inference for both PA and PAS
* Pass Data Version to AuctionService
* Pass DV Hdr to Auction Service
* Runtime flag to make all non-prod requests as always consented
* Set Data Version on AdWithBid
* Support Data Version for Seller BYOS KV
* Support inference model deletion in the model fetcher
* support per-request spec of egress schema
* support per-request specification of seller and buyer udf versions using bucket blobs
* update proto definition for multi blob support
* Use separate tf parameter/endpoint for buyer TKV
* Use separate tf parameter/endpoint for seller TKV
* Validate DV Hdr val size based on clientType
* Validate DV Header values for size based on client type

### Bug Fixes

* Check for key in perBuyerSignal before looking up buyer_signals for reporting
* Fix failing e2e test
* Fix Terraform apply when extra args are empty
* Load reportWin code when fetch_mode is local
* load the wrapper with code isolation for local fetch mode
* Log inference model registration failure reason
* Replace prefix-based matching with exact matching for model paths
* Revert saving and deleting Tensorflow models in place to fix memory leak
* Set the correct version_string for reportResult when udf loaded from bucket
* Support a wildcar SSL certificate in e2e benchmark
* update GCP cloudbuild with correct git tag name
* Use correct timeout for Kv V2 bidding signals call

Bug: N/A
GitOrigin-RevId: e0eadd262a6be13cf9925c50736f4f4b4f370fac
Change-Id: Ibae7f9519f9fc5429901a8286bb48203e2f02a92
  • Loading branch information
@lukedeng95
lukedeng95 committed Nov 20, 2024
1 parent 26d7a1d commit 8b1a980
Show file tree
Hide file tree
Showing 236 changed files with 9,675 additions and 2,931 deletions.
2 changes: 1 addition & 1 deletion .bazelrc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ build:cpp_no_warn --copt=-Werror
# Ignore warnings from Roma, zlib, differential privacy repo etc.
build:cpp_no_warn --per_file_copt=.*external/.*@-Wno-error
# Ignore deprecated declarations warnings from Roma, ProtectedAudience, Telemetry
build:cpp_no_warn --per_file_copt=(services|api/udf)/.*@-Wno-macro-redefined,-Wno-deprecated-declarations
build:cpp_no_warn --per_file_copt=(services|api/udf)/.*@-Wno-macro-redefined,-Wno-deprecated-declarations,-Wno-deprecated-builtins
build:cpp_no_warn --per_file_copt=tools/secure_invoke.*@-Wno-macro-redefined,-Wno-deprecated-declarations
# Telemetry has some unchecked results after registering observers.
build:cpp_no_warn --per_file_copt=services/.*_main.cc@-Wno-unused-result
Expand Down
56 changes: 56 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,62 @@

All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.

## 4.4.0 (2024-11-14)


### Features

* Add consented debug logs to seller KV V2
* Add Data Version for buyer to generateBid() browser signals
* add egress schema bucket fetcher
* add egress schema fetch manager
* add flag https_fetch_skips_tls_verification for local testing
* Add priority fields to B&A API
* Add PV support on BFE
* Add PV support on SFE
* Add Seller Data Version to ReportResult
* Add Seller DV to Bid Metadata for ScoreAd()
* Add support for DNS routing by regional LB latency in AWS
* Add support for DNS routing by regional LB latency in AWS
* add top level seller debugging data
* Data Version in BuyerReporting
* enable seller debugging for eligible request
* Implement delete model rpc for inference sidecar
* Implement KV V2 Adapter for Scoring Signals
* Implement KV V2 flag for seller
* Implement v2 bidding support - config flags propagation
* improve SFE response debug logging
* Launch the B&A inference for both PA and PAS
* Pass Data Version to AuctionService
* Pass DV Hdr to Auction Service
* Runtime flag to make all non-prod requests as always consented
* Set Data Version on AdWithBid
* Support Data Version for Seller BYOS KV
* Support inference model deletion in the model fetcher
* support per-request spec of egress schema
* support per-request specification of seller and buyer udf versions using bucket blobs
* update proto definition for multi blob support
* Use separate tf parameter/endpoint for buyer TKV
* Use separate tf parameter/endpoint for seller TKV
* Validate DV Hdr val size based on clientType
* Validate DV Header values for size based on client type


### Bug Fixes

* Check for key in perBuyerSignal before looking up buyer_signals for reporting
* Fix failing e2e test
* Fix Terraform apply when extra args are empty
* Load reportWin code when fetch_mode is local
* load the wrapper with code isolation for local fetch mode
* Log inference model registration failure reason
* Replace prefix-based matching with exact matching for model paths
* Revert saving and deleting Tensorflow models in place to fix memory leak
* Set the correct version_string for reportResult when udf loaded from bucket
* Support a wildcar SSL certificate in e2e benchmark
* update GCP cloudbuild with correct git tag name
* Use correct timeout for Kv V2 bidding signals call

## 4.3.0 (2024-10-25)


Expand Down
10 changes: 0 additions & 10 deletions WORKSPACE
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,6 @@ python_deps()

python_register_toolchains("//builders/bazel")

# TODO: Remove bazel_clang_tidy once we sync to the common repo commit 9edb0c3 (4/3/2024) or later
http_archive(
name = "bazel_clang_tidy",
sha256 = "352aeb57ad7ed53ff6e02344885de426421fb6fd7a3890b04d14768d759eb598",
strip_prefix = "bazel_clang_tidy-4884c32e09c1ea9ac96b3f08c3004f3ac4c3fe39",
urls = [
"https://github.com/erenon/bazel_clang_tidy/archive/4884c32e09c1ea9ac96b3f08c3004f3ac4c3fe39.zip",
],
)

http_archive(
name = "google_privacysandbox_servers_common",
# 2024-10-09
Expand Down
146 changes: 103 additions & 43 deletions api/bidding_auction_servers.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -419,12 +419,12 @@ message AuctionResult {
// BuyerAndSellerReportingId of the winning Ad.
// This will be verified with the buyerAndSellerReportingId in the Ad
// properties on the browser.
string buyer_and_seller_reporting_id = 20;
optional string buyer_and_seller_reporting_id = 20;

// SelectableBuyerAndSellerReportingId of the winning Ad.
// This will be verified with the buyerAndSellerReportingId in the Ad
// properties on the browser.
string selectable_buyer_and_seller_reporting_id = 26;
optional string selected_buyer_and_seller_reporting_id = 26;

// Join candidates for K-Anonymity for the winner. This should include
// key hashes corresponding to the winning ad only.
Expand All @@ -441,7 +441,7 @@ message AuctionResult {
// NOTE: The buyer's reportWin() UDF url endpoint must match with what
// Chrome has.
// Refer to the spec to create key hashes- https://wicg.github.io/turtledove/#query-ad-k-anonymity-count
string ad_render_url_hash = 1;
bytes ad_render_url_hash = 1;

// Protected Audience:
// - SHA-256 hash of an ad_component_render_url.
Expand All @@ -452,7 +452,7 @@ message AuctionResult {
// render url.
// Note: Currently component ads are not in scope of Protected App Signals
// for Android.
repeated string ad_component_render_urls_hash = 2;
repeated bytes ad_component_render_urls_hash = 2;

// Protected Audience:
// - SHA-256 hash should include IG owner, ad render url, reportWin() UDF
Expand All @@ -465,7 +465,7 @@ message AuctionResult {
// buyerAndSellerReportingId.
// Note: Currently reporting Ids are not in scope of Protected Audience
// for Android and Protected App Signal for Android.
string reporting_id_hash = 3;
bytes reporting_id_hash = 3;
}

KAnonJoinCandidate k_anon_winner_join_candidates = 21;
Expand Down Expand Up @@ -567,12 +567,12 @@ message AuctionResult {
// SelectableBuyerAndSellerReportingId of the ghost winning Ad.
// This will be verified with the buyerAndSellerReportingId in the Ad
// properties on the browser.
string selectable_buyer_and_seller_reporting_id = 7;
optional string selected_buyer_and_seller_reporting_id = 7;

// BuyerAndSellerReportingId of the ghost winning Ad.
// This will be verified with the buyerAndSellerReportingId in the Ad
// properties on the browser.
string buyer_and_seller_reporting_id = 8;
optional string buyer_and_seller_reporting_id = 8;
}
optional GhostWinnerForTopLevelAuction ghost_winner_for_top_level_auction = 7;
}
Expand Down Expand Up @@ -642,6 +642,45 @@ message ContextualProtectedAppSignalsData {
bool fetch_ads_from_retrieval_service = 2;
}

// Version information for various buyer blobs that may be used during the
// bidding process.
// A 'version' (each string field) is an absolute path to a blob in its
// corresponding bucket. Example: <bucket-name>/path/to/blob.
// Note: this message is only relevant if the buyer is using FETCH_MODE_BUCKET.
message BuyerBlobVersions {
// Optional.
// Version of buyer's GenerateBid() code for protected audience.
// A buyer can pass this information to the Seller in RTB response.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string protected_audience_generate_bid_udf = 1;

// Optional.
// Version of buyer's GenerateBid() code for protected app signals.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string protected_app_signals_generate_bid_udf = 2;

// Optional.
// Version of buyer's prepareDataForAdRetrieval() code for
// protected app signals.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string prepare_data_for_ad_retrieval_udf = 3;

// Optional.
// Schema for data egress from Protected App Signals bidding.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string egress_schema = 4;

// Optional.
// Schema for temporary unlimited data egress from Protected App Signals bidding.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string temporary_unlimited_egress_schema = 5;
}

// SelectAdRequest is sent by the untrusted Seller service to SellerFrontEnd
// (SFE) once it receives an encrypted ProtectedAuctionInput from a client.
// SelectAdRequest would also include contextual signals and other data
Expand Down Expand Up @@ -689,14 +728,7 @@ message SelectAdRequest {
// with the buyer's Key/Value services.
optional int32 buyer_kv_experiment_group_id = 2;

// Optional.
// Version of buyer's GenerateBid() code.
// The string must be an object name belonging to the
// Cloud Storage bucket specified at Bidding service startup.
// A buyer can pass this information to the Seller in RTB response.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string generate_bid_code_version = 3;
reserved 3;

// Optional.
// A debug id passed by the buyer that will be logged with VLOG, if
Expand All @@ -708,24 +740,9 @@ message SelectAdRequest {
// these.
string buyer_debug_id = 4;

// Optional.
// Version of buyer's GenerateBid() code for protected app signals.
// The string must be an object name belonging to the
// Cloud Storage bucket specified at Bidding service startup.
// A buyer can pass this information to the Seller in RTB response.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string protected_signals_generate_bid_code_version = 5;
reserved 5;

// Optional.
// Version of buyer's GenerateAdRetrievalUserMetadata() code for
// protected app signals.
// The string must be an object name belonging to the
// Cloud Storage bucket specified at Bidding service startup.
// A buyer can pass this information to the Seller in RTB response.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
string protected_signals_generate_embeddings_version = 6;
reserved 6;

// Specifies currency in which interest group bids will be placed (expressed
// as ISO 4217 alpha code).
Expand Down Expand Up @@ -758,6 +775,11 @@ message SelectAdRequest {
// provided in *this* field. Implying that priority_signals_overrides
// values take precedence over the values in the priority_signals below.
string priority_signals_overrides = 10;

// Optional.
// Represents all blob versions the buyer logic should use for this request.
// Buyer can pass this information to the Seller in RTB response.
BuyerBlobVersions blob_versions = 11;
}

// The key in the map corresponds to Interest Group Owner (IGOwner), a
Expand All @@ -782,6 +804,9 @@ message SelectAdRequest {
// Cloud Storage bucket specified at Auction service startup.
// If a version is not specified, the default version
// (specified in the service startup config) will be used.
// The absolute path of the blob must be used. Example:
// <bucket-name>/path/to/scoreAd.js
// Note: this field is only relevant if the seller is using FETCH_MODE_BUCKET.
string score_ad_version = 2;
}

Expand Down Expand Up @@ -1012,6 +1037,11 @@ message GetBidsRequest {
// Represents a serialized JSON map containing priority signal entries used to
// construct the priority signals vector.
string priority_signals = 18;

// Optional.
// Represents all blob versions the buyer logic should use for this request.
// Buyer can pass this information to the Seller in RTB response.
BuyerBlobVersions blob_versions = 19;
}

// Encrypted GetBidsRawRequest.
Expand Down Expand Up @@ -1108,15 +1138,16 @@ message AdWithBid {
// Private aggregation object.
repeated PrivateAggregateContribution private_aggregation_contributions = 12;

// If provided, and selectableBuyerAndSellerReportingId is not set then
// If provided, and selectedBuyerAndSellerReportingId is not set then
// this will be substituted for interestGroupName.
string buyer_and_seller_reporting_id = 13;
optional string buyer_and_seller_reporting_id = 13;

// Optional. selectableBuyerAndSellerReportingId in buyerReportingMetadata of
// Optional. selectedBuyerAndSellerReportingId in buyerReportingMetadata of
// reportWin(). If provided, this will be substituted for interestGroupName.
string selectable_buyer_and_seller_reporting_id = 14;
string selected_buyer_and_seller_reporting_id = 14;

// Optional. Value of the data version header from the buyer kv server. This is passed through to the buyer reporting function.
// Optional. Value of the data version header from the buyer kv server.
// This is passed through to the buyer reporting function.
int32 data_version = 15;
}
// (-- LINT.ThenChange(bid_generate_bid) --)
Expand Down Expand Up @@ -1272,6 +1303,11 @@ message GenerateBidsRequest {

// If BFE is sampling this request for debug.
bool is_sampled_for_debug = 14;

// Optional.
// Represents all blob versions the buyer logic should use for this request.
// Buyer can pass this information to the Seller in RTB response.
BuyerBlobVersions blob_versions = 15;
}

// Encrypted GenerateBidsRawRequest.
Expand Down Expand Up @@ -1376,6 +1412,11 @@ message GenerateProtectedAppSignalsBidsRequest {

// If BFE is sampling this request for debug.
bool is_sampled_for_debug = 13;

// Optional.
// Represents all blob versions the buyer logic should use for this request.
// Buyer can pass this information to the Seller in RTB response.
BuyerBlobVersions blob_versions = 14;
}

// Encrypted GenerateProtectedAppSignalsBidsRawRequest.
Expand Down Expand Up @@ -1518,14 +1559,14 @@ message ScoreAdsRequest {
string buyer_reporting_id = 14;

// Optional. buyerAndSellerReportingId in buyerReportingMetadata
// of reportWin(). If provided, and selectableBuyerAndSellerReportingId
// of reportWin(). If provided, and selectedBuyerAndSellerReportingId
// is not set then this will be substituted for interestGroupName.
string buyer_and_seller_reporting_id = 15;
optional string buyer_and_seller_reporting_id = 15;

// Optional. selectableBuyerAndSellerReportingId in buyerReportingMetadata
// Optional. selectedBuyerAndSellerReportingId in buyerReportingMetadata
// of reportWin(). If provided, this will be substituted for
// interestGroupName.
string selectable_buyer_and_seller_reporting_id = 16;
optional string selected_buyer_and_seller_reporting_id = 16;

// Indicates whether or not the ad meets the k-anonymity threshold.
bool k_anon_status = 17;
Expand Down Expand Up @@ -1665,6 +1706,16 @@ message ScoreAdsRequest {
// Clients can indicate whether K-Anonymity is enforced for the traffic.
// By default, this would be false, i.e. K-Anonymity is not enforced.
bool enforce_kanon = 16;

// Optional. Value of the data version header from the seller kv server.
// This is passed through to the seller reporting function.
uint32 seller_data_version = 17;

// If SFE is sampling this request for debug.
bool is_sampled_for_debug = 18;

// Optional. From AuctionConfig.code_experiment_spec.score_ad_version.
string score_ad_version = 19;
}

// Encrypted ScoreAdsRawRequest.
Expand Down Expand Up @@ -1814,7 +1865,7 @@ message ScoreAdsResponse {
// BuyerAnd SellerReportingId of the winning Ad.
// This will be verified with the buyerAndSellerReportingId in the Ad
// properties on the browser.
string buyer_and_seller_reporting_id = 19;
optional string buyer_and_seller_reporting_id = 19;

// Owner of the IG to which the winning ad belongs (applicable for
// Protected Audience only).
Expand Down Expand Up @@ -1872,7 +1923,9 @@ message WinReportingUrls {
ReportingUrls top_level_seller_reporting_urls = 3;
}

// message to store the server request/ response
// message to store the server request/ response.
// udf logs are always populated; raw request, KvSignal and encrypted request
// are populated according to server verbosity setting.
message EventMessage {
message KvSignal {
string request = 1;
Expand All @@ -1890,6 +1943,7 @@ message EventMessage {
ProtectedAudienceInput protected_audience = 4;
}
AuctionResult auction_result = 5;
repeated AuctionResult component_auction_result = 6;

// bfe
GetBidsRequest get_bid_request = 11;
Expand All @@ -1911,4 +1965,10 @@ message EventMessage {
ScoreAdsResponse.ScoreAdsRawResponse score_ad_raw_response = 33;

KvSignal kv_signal = 41;

message MetaData {
bool is_consented = 1;
bool is_prod_debug = 2;
}
MetaData meta_data = 51;
}
Loading

0 comments on commit 8b1a980

Please sign in to comment.