1. Introduction
Bidding & Auction Services (B&A) is comprised of 4 services for buyers and sellers to facilitate a Protected Audience auction:
Buyer stack:
- Buyer Front-End Service
- Bidding Service
Seller stack:
- Seller Front-End Service
- Auction Service
This codelab takes you through setting up and testing the end-to-end setup in your local environment. The walkthrough is expected to take approximately 1 hour, excluding the initial service build time.
Even though you may only work on the buy-side code or the sell-side code, it's useful to set up the end-to-end flow in your local environment to better understand how the buyer and seller stacks work with one another. By setting up both stacks, you can be more confident later when you integrate with another party's respective buyer or seller stack. Testing the services locally also saves your development cost.
In this codelab, we will use the B&A Local Testing App as a walkthrough companion.
The companion local testing app hosts various buyer and seller resources used in a B&A auction. The app also provides a UI at http://localhost:3000 where you are able to simulate various auction configurations.
2. Environment setup
2.1 Prepare a Linux machine
Use a local Linux machine, or provision a Linux VM of the cloud provider of your choice. To improve the build time, at least 16 cores are recommended as the minimum, and 32+ cores are preferred.
Note that we will be loading an HTML page from http://localhost:3000 of this machine. If the VM does not provide a GUI, then make sure port 3000 is accessible to your local machine so you can connect to it.
2.2 Install Docker
We use Docker to run the services and the local testing application.
We will use the convenience script to install Docker in a testing environment.
# Install Docker
> curl -fsSL https://get.docker.com -o get-docker.sh
> sudo sh get-docker.sh
# Set up sudo-less Docker
> sudo groupadd docker
> sudo usermod -aG docker $USER
> newgrp docker
# Test
> docker run hello-world
Warning: With the sudo-less setup, the Docker group grants root-level privileges to the user. Read the sudo-less Docker guide to learn more.
2.3 Start the companion Local Testing App
The companion app provides resources such as mock bidding/scoring scripts, and mock K/V BYOS endpoints. The app should be started before running the B&A services so the resources can be loaded into the services.
Pull down the Local Testing App repository:
git clone https://github.com/privacysandbox/bidding-auction-local-testing-app.git
From the root of the repository, run the setup script:
./setup
The setup script will create the ba-dev Docker network, generate SSL certificates, and build the test app image.
Once the build is successful, run the start script:
./start
3. Build and run B&A locally
3.1 Pull down B&A Services
We will run two sets of B&A Services on the same machine, so we will pull down the repository into two different folders.
Set A:
Pull down the B&A repository:
git clone https://github.com/privacysandbox/bidding-auction-servers.git bidding-auction-servers-set-a
Set B
Pull down the B&A repository:
git clone https://github.com/privacysandbox/bidding-auction-servers.git bidding-auction-servers-set-b
`
3.2 Build the services
From the root folder of each B&A folder, run the following command to build all 4 services:
production/packaging/build_and_test_all_in_docker \
--service-path bidding_service \
--service-path auction_service \
--service-path buyer_frontend_service \
--service-path seller_frontend_service \
--platform gcp \
--instance local \
--no-precommit \
--no-tests \
--build-flavor non_prod \
--gcp-skip-image-upload
The above command will build a local instance of the development build (non_prod) for Google Cloud Platform (gcp) of all 4 services. Update the --platform flag to the cloud provider of your choice if needed. To learn more about the build flags, check the build_and_test_all_docker file.
The initial build may take 2 hours if using a 64-core machine, and the build time scales linearly by core count. If you are on a 32-core machine, it may take 4 hours. If you are using a 16-core machine, it may take 8 hours.
Note that subsequent builds use the cache, and each build should take about 5-20 minutes.
(I met Randall Munroe of xkcd while writing this codelab, and told him that I added this comic here)
3.3 Run the services
For our testing, we will run 8 services (4 services per stack). Each command is set up to use the resources provided by the Local Testing App. It is highly recommended to use a window manager such as tmux.
Open at least 8 additional terminal windows, and run the following commands in each session.
Note that the Local Testing App server must already be running in a separate process, since the bidding and scoring scripts will be served by the app.
3.3.1 Set A commands
Service | Command |
|
|
|
|
|
|
|
|
3.3.2 Set B commands
Service | Command |
|
|
|
|
|
|
|
|
3.3.2 Environment variables for the local start scripts
The following environment variables can be used to control the start-up behavior.
Env var | Availability | Description |
| All services | Flags to set for the docker command |
| All services | When you are testing on your local machine, you may need to use a self-signed certificate which fails Defaults to |
| All services | The level of service logs. Higher level provides more logs. 0 is the lowest and 10 is the highest. Defaults to |
| Bidding Service | Bidding Service's port. Defaults to |
| Bidding Service | URL of the bidding logic |
| Bidding Service | Data egress for model training |
| Bidding Service | Arguments for bidding inference |
| BFE Service | BFE Service's port. Defaults to |
| BFE Service | Buyer's BYOS K/V address |
| BFE Service | Buyer's TEE K/V address |
| BFE Service | Buyer's Bidding Service address |
| Auction Service | Auction Service's port. Defaults to |
| Auction Service | URL of the scoring logic |
| Auction Service | Buyer's win reporting URL |
| Auction Service | Buyer's win reporting script |
| Auction Service | Buyer's win reporting script for PAS |
| SFE Service | SFE Service's port. Defaults to |
| SFE Service | Seller's Auction Service address |
| SFE Service | Seller's BYOS K/V address |
| SFE Service | Seller's TEE K/V address |
| SFE Service | Seller's origin |
| SFE Service | Buyer's BFE addresses |
4. Test with Secure Invoke from the command line
4.1 SFE testing
You can also use the Secure Invoke tool that is included in the B&A Services package to call the services directly from the command line.
4.1.2 SFE SelectAd payload
The following JSON request has been populated with SSP-BA's single-seller B&A auction setup:
{
"auction_config": {
"seller": "https://localhost:6002",
"auction_signals": "{\"testKey\":\"someValue\"}",
"seller_signals": "{\"testKey\":\"someValue\"}",
"buyer_list": [
"https://localhost:5003",
"https://localhost:5004"
],
"per_buyer_config": {
"https://localhost:5003": { "buyer_signals": "{\"testKey\":\"someValue\"}" },
"https://localhost:5004": { "buyer_signals": "{\"testKey\":\"someValue\"}" }
}
},
"raw_protected_audience_input": {
"publisher_name": "http://localhost:4001",
"generation_id": "1210718540",
"enable_debug_reporting": true,
"raw_buyer_input": {
"https://localhost:5003": {
"interest_groups": [
{
"name": "dsp-x-demo",
"ad_render_ids": ["1234"],
"bidding_signals_keys": ["demo-key"],
"browser_signals": { "bid_count": "1", "join_count": "1", "prev_wins": "[]" },
"user_bidding_signals": "[1, 2, 3]"
}
]
},
"https://localhost:5004": {
"interest_groups": [
{
"name": "dsp-x-demo",
"ad_render_ids": ["1234"],
"bidding_signals_keys": ["demo-key"],
"browser_signals": { "bid_count": "1", "join_count": "1", "prev_wins": "[]" },
"user_bidding_signals": "[1, 2, 3]"
}
]
}
}
}
}
Save the file as sfe-test.json in the root folder of the Set A's B&A Services repository(B&A build script will copy the files in the root folder of the repo into /src/workspace folder of the Docker image).
4.1.2 Call SFE with Secure Invoke
Run the following command from the root folder of the B&A Services repository:
DOCKER_NETWORK=ba-dev ./builders/tools/bazel-debian run //tools/secure_invoke:invoke \
-- \
-target_service=sfe \
-input_file=/src/workspace/sfe-test.json \
-host_addr=192.168.84.104:50053 \
-client_ip=192.168.84.100 \
-insecure=true
If the B&A stacks are set up correctly, you'll receive the following successful response:
{"adRenderUrl":"https://localhost:5004/ad.html","interestGroupName":"dsp-x-demo","interestGroupOwner":"https://localhost:5004","score":39,"bid":39,"biddingGroups":{"https://localhost:5003":{"index":[0]},"https://localhost:5004":{"index":[0]}}}
4.2 BFE testing
4.2.1 BFE GetBids request
{
"buyer_input": {
"interest_groups": {
"name": "dsp-x-demo",
"ad_render_ids": ["1234"],
"bidding_signals_keys": ["demo-key"],
"browser_signals": { "bid_count": "1", "join_count": "1", "prev_wins": "[]" }
}
},
"auction_signals": "{\"testKey\":\"someValue\"}",
"buyer_signals": "{\"testKey\":\"someValue\"}",
"seller": "https://localhost:6002",
"publisher_name": "http://localhost:4001",
"enable_debug_reporting": true,
"client_type": "CLIENT_TYPE_BROWSER"
}
Save the file as bfe-test.json in the root folder of the B&A Services repository.
4.2.2 Call BFE with Secure Invoke
Run the following command from the root folder of the B&A Services repository:
DOCKER_NETWORK=ba-dev ./builders/tools/bazel-debian run //tools/secure_invoke:invoke \
-- \
-target_service=bfe \
-input_file="/src/workspace/bfe-test.json" \
-host_addr="192.168.84.102:50051" \
-client_ip=192.168.84.100 \
-insecure=true
If the B&A stacks are set up correctly, BFE returns the following response:
{"bids":[{"bid":6,"render":"https://localhost:5003/ad.html","interestGroupName":"dsp-x-demo"}],"updateInterestGroupList":{}}
4.3 Check the SFE logs
Visit the terminal and check the SFE logs. We will walk through some of the notable logs in the output.
4.3.1 SelectAd request's encrypted payload
In this SFE log output, we find the encrypted payload of the navigator.getInterestGroupAdAuctionData() call. This payload is generated by the browser, sent from the tag to SAS, and then SAS forwards it to SFE:
I1210 21:20:47.266673 37 select_ad_reactor.cc:228] Protected auction ciphertext: AEAAIAABAAKg3OQSSLEBMR1MmJiwfOT8uef41iE+DQIvKUowsQvVcGbWZ+r17ff2r/iTbEnwqLPtpAxSTy77mi+cVV5cKjkKb4piaZnKiCDIMCyEvrQUD4+BG+HgrbUFhqiWPmQhNThvInMaHuYrXG7xushbRtvSPsVzEl6iBZcLZzD7W3CHMFK7bnJ+ufu52W6B7/8dOQ1gZW6SpwXMGNcrurTGoucbwbZRIPqCcbZk76J1gB69wrDYRo1wQ6jmBleLM36nulxbEj/sHhpQIgr895aKqEGQ3Fy+/HHPTO+zHphcX44YeF6mZ/I+WxZTDxCi1x27nI4yvzC8vI4a/giJN5XUlJZGb/fMrj3/Qly+gwJ8lbxdX+4GWTooDoS4MTtiBpJqh9wkIQ5XwfO54p8Kv6j/tk+51hOfVkOdtGwX0LvVrWT9PiJOJHa23nZ1fLg/22mTe05xdbR3WVeA+xiO9YwF4ozyCYDRUkDyWB/3k9RuaaT9H5S1szVpuOuD2nYSRpT9svmpfG4kipiS6BTPZPX2whDlukpuuoFgdiDtH6cFShNhZ5C47R0ayAvaZ1QSpOZMWfA6pdl4nhvNLpOKesKY/KFDEBzpUXRDsQWsfvmdqFl1mCeX9rOrEc9rodYVLXXhM8GPycehHmvqkccTBW6kipOQlmnSdIYaFmtWSFpUrzxajGkct3mZdTjVjOf...
4.3.2 Decrypted payload
In this output, we find the decrypted payload of the navigator.getInterestGroupAdAuctionData() call:
I1210 21:20:47.267590 37 select_ad_reactor.cc:347] (generationId: c2824bfc-d515-4a1a-9de9-7660ad963252) Decoded BuyerInput:
https://localhost:5003 : {interest_groups { name: "dsp-x-demo" bidding_signals_keys: "demo-key" browser_signals { join_count: 1 recency: 7 prev_wins: "[]" } }}
4.3.3 GetBids request to BFE
The GetBids request is sent from SFE to BFE:
I1210 21:20:47.267710 37 select_ad_reactor.cc:751] (generationId: c2824bfc-d515-4a1a-9de9-7660ad963252) Getting bid from a BFE
I1210 21:20:47.267729 37 buyer_frontend_async_client.cc:52] Raw request:
buyer_input {
interest_groups {
name: "dsp-x-demo"
bidding_signals_keys: "demo-key"
browser_signals {
join_count: 1
recency: 7
prev_wins: "[]"
}
}
}
auction_signals: "{\"testKey\":\"someValue\"}"
buyer_signals: "{\"testKey\": \"someValue\"}"
seller: "https://localhost:6002"
publisher_name: "localhost"
enable_debug_reporting: true
log_context {
generation_id: "c2824bfc-d515-4a1a-9de9-7660ad963252"
}
client_type: CLIENT_TYPE_BROWSER
4.3.4 GetBids response from BFE
BFE responds with the bid:
I1210 21:20:47.282549 41 select_ad_reactor.cc:856] (generationId: c2824bfc-d515-4a1a-9de9-7660ad963252) Received response from a BFE ...
I1210 21:20:47.282552 41 select_ad_reactor.cc:859] (generationId: c2824bfc-d515-4a1a-9de9-7660ad963252)
GetBidsResponse:
bids {
bid: 9
render: "https://localhost:5003/ad.html"
interest_group_name: "dsp-x-demo"
}
update_interest_group_list {
}
4.3.5 ScoreAds request to Auction Service
After the bids are received, the ScoreAds call is made by SFE to the Auction Service for each bid:
I1210 21:20:47.287898 18 select_ad_reactor.cc:1280] (generationId: c2824bfc-d515-4a1a-9de9-7660ad963252) ScoreAdsRawRequest:
ad_bids {
bid: 9
render: "https://localhost:5003/ad.html"
interest_group_name: "dsp-x-demo"
interest_group_owner: "https://localhost:5003"
join_count: 1
k_anon_status: true
}
seller_signals: "{\"testKey\":\"someValue\"}"
auction_signals: "{\"testKey\":\"someValue\"}"
scoring_signals: "{\"renderUrls\":{\"https://localhost:5003/ad.html\":[1,2,3],\"https://localhost:5004/ad.html\":[1,2,3]}}"
publisher_hostname: "localhost"
enable_debug_reporting: true
log_context {
generation_id: "c2824bfc-d515-4a1a-9de9-7660ad963252"
}
per_buyer_signals {
key: "https://localhost:5003"
value: "{\"testKey\": \"someValue\"}"
}
per_buyer_signals {
key: "https://localhost:5004"
value: "{\"testKey\": \"someValue\"}"
}
seller: "https://localhost:6002"
4.3.6 ScoreAds response from Auction Service
Seller's Auction Service responds with the desirability score:
I1210 21:20:47.298029 33 default_async_grpc_client.h:134] Decrypting the response ...
I1210 21:20:47.298100 33 default_async_grpc_client.h:152] Decryption/decoding of response succeeded: ad_score {
desirability: 9
render: "https://localhost:5003/ad.html"
interest_group_name: "dsp-x-demo"
buyer_bid: 9
interest_group_owner: "https://localhost:5003"
win_reporting_urls {
top_level_seller_reporting_urls {
}
}
ad_type: AD_TYPE_PROTECTED_AUDIENCE_AD
}
4.3.7 SelectAd response from SFE
The SFE's SelectAd response is encrypted and sent to SAS:
I1210 21:20:47.298428 33 select_ad_reactor.cc:1413] (generationId: c2824bfc-d515-4a1a-9de9-7660ad963252) Encrypted SelectAdResponse:
auction_result_ciphertext: "\023\233\252\321\000\331M\327\216\201S\332kl?\322\255\247\312\375h\007W\367\016\366FR\364\275\377\326\027h\204\203\265\t\271-\333\266\tH\342\033F\001k\211C\026V\013\271\255(\236\273\353\260\275\007\346\027\340?\005|\235S\244\265\2563U\246\322\177%\200/\3175\034 \020\024\362\345\3263\356F.s\261\005\231\262Z\000\330x\0230\335>\"\217\254\010\254\330>\206\007\274\235\037`\370W\032\207\356\367\206\026\341.i\"$\370\367\2554i\247\354\304e\306\2466S\337\321w\327+!\316\035:\002\231\246\362\366qm\211\000\013\343\345\224{\365py\361\374\316\202\217-\244\302\331\216D\025W#.\304nH\235\315\311<#\342\344\324\257\354\2441\251\312\320\226\342\021\377>3`\347/\350\254h\306\273\023\365\340@\321\2412\254\323\213\0137,\013n\220|\211\323/\030)\314\263\223\355>\254\312aa`!\375\335\023Z\234p\206\037\001\355\261{$\025+\341\275\327Ny\342\342\264=\376\2138\224\026\2058\251\020\202\245*46\023\307)K\342\364k<\2104r\247\034\216}\034\001\374\215\363\210\026\275\371S>\031;f.b\260\363\257.\255\023I\341A\237*_T\355\262\005\344L\336D%\327\267@\302$\300\340\203c\350|\337>C\275c\260\202o\315xp\260\257\241\305U\nK\033\274L6\025\350\373a:\253\212&>p\210\215\017&^\327\005_\037\020\212\362\351\341\231@g\372\0037\275b:)\207\303d\243?\261O\266\343\214\200\004\247\372\022S\326\200\037\330\252r\257+e\273[\231\202\3625{\330\213\240\370\245\\\214\350/\313\ty/\004\313\0304k\223\354\374\305\233\264\224K\344\241\251\322?\323q6\314D\027\200\203\035]\023O\306\230?\203\237:\254\305\265\332\330\2641l\322\020\022\330O-\242\207>Q\364_{F\235\353tk\207\035\205\005\221\373\207\0143\035\014\322\240H\220\347%\262f\347\017\352\273\265\231\014\204\r\312\254z\000\340&\354\222\323\245\356\217(i\246"
5. Test with Local Test App from the browser
5.1 Open Chrome
Read the Chromium article on starting Chrome from the command line, and start it with the following flags:
google-chrome --enable-privacy-sandbox-ads-apis --disable-features=EnforcePrivacySandboxAttestations,FledgeEnforceKAnonymity --enable-features=FledgeBiddingAndAuctionServerAPI,FledgeBiddingAndAuctionServer:FledgeBiddingAndAuctionKeyURL/https%3A%2F%2Fstorage.googleapis.com%2Fba-test-buyer%2Fcoordinator-test-key.json
The flag is set to load the mock coordinator key that we have hosted at https://storage.googleapis.com/ba-test-buyer/coordinator-test-key.json
Important: Make sure you have fully exited out of all Chrome instances before opening it from the command line with the B&A flags. To exit out of all Chrome processes, try running ps aux | grep -ie chrome/chrome | awk '{print $2}' | xargs kill -9 from the command line.
5.2 Visit the Local Testing App UI
The app UI can be found at http://localhost:3000 in the Chrome browser instance you opened with the flags. The exact address you visit may be different if you are using a VM with a different hostname.
In the top row of the app, the "Controls" panel contains the buttons to simulate different auction configurations. The other two panels contain iframes from the advertiser and publisher sites.
When you click the "Load DSP tags" button in the advertiser page iframe, a script from each DSP is added to the page. Those scripts add the user to interest groups. Open Chrome DevTools, and visit "Application / Storage / Interest Groups" to examine the interest groups joined by the DSP tags:
Then, click the buttons in the "Controls" panel to simulate various auction types.
If the B&A Services and the Local Testing App are set up correctly, the preceding ad is rendered in the "Publisher site iframe" panel.
5.3 Local Testing App auction configuration
In the companion local testing app, the following participants are hosted, differentiated by the port number (a different port is considered cross-origin):
Participant | Description | Port |
Advertiser | Loads DSP tags on the page |
|
Publisher | Loads SSP tags on the page |
|
| On-device buyer |
|
| On-device buyer |
|
| B&A buyer |
|
| B&A buyer |
|
| Top-level seller |
|
| On-device-only seller |
|
| B&A-only seller |
|
| Mixed-mode seller |
|
There are four DSPs:
DSP-AandDSP-Bparticipate in on-device auctionsDSP-XandDSP-Yparticipate in B&A auctions
There are four SSPs, and each seller runs a different auction configuration:
SSP-ODruns an on-device-only auctionSSP-BAruns a B&A-only auctionSSP-MIXruns a mixed-mode auctionSSP-TOPruns a multi-seller auctionSSP-OD/BA/MIXparticipate as component sellers ofSSP-TOP's multi-seller auction
For the full architecture, check the design diagram.
5.4 Bidding and Auction Services configuration
In this codelab, we run two sets of B&A alongside one another.
Set | Service | Participant |
|
Set A | Bidding Service |
|
|
BFE Service |
|
| |
Auction Service |
|
| |
SFE Service |
|
| |
Set B | Bidding Service |
|
|
BFE Service |
|
| |
Auction Service |
|
| |
SFE Service |
|
| |
App | Local Testing App | All |
|
- In Set A, the buyer services are used by
DSP-Xand the seller services are used bySSP-BA - In Set B, the buyer services are used by
DSP-Yand the seller services are used bySSP-MIX
For communicating between Local Testing App and B&A Services, a Docker bridge network is used. The ba-dev bridge network is created by the setup script of the Local Testing App. The B&A services are assigned an IP address in the subnet of 192.168.84.0.
For example, when you are calling Set A's SFE from the application, the network address of 192.168.84.104:50053 is used. When you are loading the mock bidding script from DSP-X into BFE, https://192.168.84.100:5003/generate-bid.js is used.
6. Wrap up
By now, we hope that you have familiarized yourself with running the B&A Services stack locally on your machine, and gained a better understanding in how the services communicate with one another.
6.1. Technical support
- If you have a question about setting up the Local Testing App, open an issue in the LTA repository.
- If you have a question about Bidding and Auction Services, open an issue in the B&A Services repository.
- If you have a question about Privacy Sandbox in general, open an issue in the privacy-sandbox-dev-support repository.
6.2 Learn more
- Learn more about the B&A architecture for web
- Learn more about the seller integration with B&A for web