Relevance API Lab

What you'll build

A synthetic world: 10,000 products (ASINs) in 50 categories and 1,000 creatives that advertise them, some brand new with no product yet.
The daily batch: ASIN-to-ASIN and creative-to-creative similarities, joined into about 277,000 relevant (ASIN, creative) pairs, validated, and published as one version with its own Bloom filter.
The API: an ASIN plus candidate creatives in, a score per creative out, through the Bloom filter, a hot-key cache and one batched database read, swapping versions while it serves.
Five experiments on your laptop, a garbage-collector experiment in Java, and the same service on AWS: DynamoDB, S3, a load balancer, an Auto Scaling group and its own load-generator machine.

Tick each step as you finish it; your progress is saved in this browser.

Part	You need	Cost
1 · Laptop	Python 3.10 or newer, about 1 GB free disk	Free
2 · Garbage collector	Java 21 (any OpenJDK build)	Free
3 · AWS	An AWS account you can create resources in, AWS CLI v2	Billable while it runs: EC2, a load balancer, DynamoDB, an Elastic IP. Delete it when done (step 17).

Part 1 · On your laptop
Five experiments
Part 2 · The garbage collector
Part 3 · On AWS
Troubleshooting
Clean up

PART 1

On your laptop

1. Get the code

get the lab

curl -LO https://tascano.github.io/notes/relevance-lab.zip
unzip relevance-lab.zip && cd relevance-lab

No curl? Download relevance-lab.zip and unzip it. Every file is also readable on GitHub in notes/relevance-lab.

File	What it is
`data.py`	The synthetic world of ASINs and creatives (same seed, same world, for everyone)
`precompute.py`	The daily batch: similarities → scores → validate → publish one version
`bloom.py`	The Bloom filter from the post, plus save and load
`store.py`	Score storage: SQLite on a laptop, DynamoDB on AWS, versions side by side
`service.py`	The API: Bloom filter → cache → one batched read, swapping versions live
`loadtest.py`	A load generator with skewed, realistic traffic
`GcLab.java`	The garbage-collector experiment
`aws/stack.yaml`	Everything AWS needs, as one CloudFormation template
`test_lab.py`	Checks for the laptop path and (with moto) the AWS path

2. Create a virtual environment and install

macOS / Linux

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Windows (PowerShell): py -m venv .venv, then .venv\Scripts\Activate.ps1, then the same pip install. Everything below works the same, except you set environment variables with $env:BLOOM="off" instead of BLOOM=off.

3. Run the daily batch: version 1

post §03, §08

build and publish 2026-10-30

python precompute.py --day 2026-10-30

output

[2026-10-30] scored pairs: 276,536 relevant (ASIN, creative) pairs in 0.6s
[2026-10-30] validation passed (first version)
[2026-10-30] Bloom filter: 2,650,614 bits (324 KiB), k=7
[2026-10-30] wrote 276,536 rows to sqlite in 0.8s
[2026-10-30] published: current.json now points at version 2026-10-30

That's the whole daily backfill in one command: similarities, the threshold, the validation gate, a Bloom filter for exactly these keys, a SQLite file for this version, and finally data/current.json pointing at it. That last write is atomic (write a temp file, then rename), which is the "publish as one version" rule from the post.

4. Start the API

post §05

terminal 1

uvicorn service:app --port 8000

output

INFO:     Application startup complete.
serving version 2026-10-30 (276,536 rows)
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

5. Ask it a question

In a second terminal (activate the same virtual environment), send one product and three candidate creatives, twice:

terminal 2

curl -s localhost:8000/relevance -H 'content-type: application/json' \
  -d '{"asin": "B000000042", "creatives": ["cr-179", "cr-396", "cr-7"]}'

output

{"version":"2026-10-30","scores":{"cr-179":0.935,"cr-396":0.923,"cr-7":0.0}}

what each layer did

curl -s localhost:8000/metrics

output, after asking twice

{"version":"2026-10-30", ..., "requests":2, "lookups":6, "bloom_no":2,
 "cache_hits":2, "db_reads":1, "db_keys":2, ...}

Six lookups, and only one trip to the database. cr-7 isn't relevant to this product, and the Bloom filter said so from memory, both times. The first request fetched the other two scores in a single batched read; the second found them in the cache.

6. Load-test it

post §02

terminal 2

python loadtest.py --seconds 12 --processes 2 --concurrency 32 --reset

output

requests: 8,102 ok, 0 errors in 12.2s -> 665 req/s
client latency ms: p50 59.1  p99 457.5  max 864.8
Little's law: 665 req/s x 0.0946 s = 63 in flight (you ran 64)
server [2026-10-30] 40,510 lookups: Bloom said no 50%, cache hits 25%, read from db 25% in 4,591 batched reads
server latency ms: p50 0.083  p99 1.994  max 16.939

Four things to read in that output:

Little's law holds: requests per second × average latency ≈ requests in flight, and in flight is exactly the concurrency you asked for (64).
The layers: about half the lookups never got past the Bloom filter, a quarter came from the cache, a quarter went to the database.
Client vs server time: the server needs well under a millisecond per request, but clients wait about 60 ms. That gap is queueing in the HTTP layer, not the lookups. Always measure both sides.
Who's the bottleneck? Rerun with --processes 3. On my 4-core machine throughput rose from about 690 to 970 requests a second while the server sat around 50% CPU. The load generator was the limit, which is why real load tests run from a fleet of their own.

EXPERIMENTS

Five experiments

Each one changes one thing. Stop the API (Ctrl+C in terminal 1), restart it with the setting shown, and rerun the load test from step 6. Make a prediction first; it's more fun when you're wrong.

7. Experiment 1 · Turn the Bloom filter off

post §06

terminal 1

BLOOM=off uvicorn service:app --port 8000

output

requests: 8,406 ok, 0 errors in 12.1s -> 692 req/s
client latency ms: p50 57.2  p99 456.2  max 1079.4
Little's law: 692 req/s x 0.0910 s = 63 in flight (you ran 64)
server [2026-10-30] 42,030 lookups: Bloom said no 0%, cache hits 39%, read from db 61% in 7,202 batched reads
server latency ms: p50 0.082  p99 1.702  max 12.737

Predict: How much more of the work reaches the database?
Observe: Database reads go from 25% of lookups to 61%, and batched reads from about 4,600 to 7,200.
Why: Without the filter, every "not relevant" pair costs a read that comes back empty. The cache absorbs some of them (negative caching), but the filter answers them all from memory, with certainty.

8. Experiment 2 · Turn the cache off

post §07

terminal 1

CACHE_SIZE=0 uvicorn service:app --port 8000

output

requests: 8,342 ok, 0 errors in 12.1s -> 688 req/s
client latency ms: p50 58.6  p99 468.1  max 904.1
Little's law: 688 req/s x 0.0917 s = 63 in flight (you ran 64)
server [2026-10-30] 41,710 lookups: Bloom said no 50%, cache hits 0%, read from db 50% in 8,055 batched reads
server latency ms: p50 0.132  p99 2.734  max 30.163

Observe: The Bloom filter still stops half the lookups, but everything else now goes to the database: batched reads nearly double, and the server's own p99 rises.
Why: Popularity is skewed (the load test uses a Zipf curve), so a small cache catches a big share. Layers multiply.

9. Experiment 3 · Make the database slow

terminal 1, then again with CACHE_SIZE=0

DB_DELAY_MS=2 uvicorn service:app --port 8000
DB_DELAY_MS=2 CACHE_SIZE=0 uvicorn service:app --port 8000

output, with the cache

requests: 8,487 ok, 0 errors in 12.1s -> 700 req/s
client latency ms: p50 56.1  p99 456.2  max 831.6
Little's law: 700 req/s x 0.0901 s = 63 in flight (you ran 64)
server [2026-10-30] 42,435 lookups: Bloom said no 50%, cache hits 25%, read from db 25% in 4,768 batched reads
server latency ms: p50 2.169  p99 3.834  max 28.117

output, without the cache

requests: 8,540 ok, 0 errors in 12.1s -> 704 req/s
client latency ms: p50 54.6  p99 450.9  max 1354.5
Little's law: 704 req/s x 0.0896 s = 63 in flight (you ran 64)
server [2026-10-30] 42,700 lookups: Bloom said no 50%, cache hits 0%, read from db 50% in 8,244 batched reads
server latency ms: p50 2.243  p99 4.108  max 12.432

Observe: Divide batched reads by requests: with the cache, 56% of requests waited the extra 2 ms; without it, 97% did.
Why: Every layer in front of a slow dependency decides how many requests feel it. This is also why the post's ad server fails open: a slow answer is worth the same as no answer.

10. Experiment 4 · Publish a new version under load

post §08

Restart the API normally. Start a 15-second load test, and while it runs, publish the next day from a third terminal:

terminal 2

python loadtest.py --seconds 15 --processes 2 --concurrency 32 --reset

terminal 3, a few seconds later

python precompute.py --day 2026-10-31

output

[2026-10-31] scored pairs: 280,948 relevant (ASIN, creative) pairs in 1.2s
[2026-10-31] validation passed (live version 2026-10-30 has 276,536 rows)
[2026-10-31] Bloom filter: 2,692,903 bits (329 KiB), k=7
[2026-10-31] wrote 280,948 rows to sqlite in 0.8s
[2026-10-31] published: current.json now points at version 2026-10-31

output

requests: 8,899 ok, 0 errors in 15.2s -> 586 req/s
client latency ms: p50 69.8  p99 531.7  max 1189.1
Little's law: 586 req/s x 0.1077 s = 63 in flight (you ran 64)
server [2026-10-31] 44,495 lookups: Bloom said no 70%, cache hits 13%, read from db 17% in 3,564 batched reads
server latency ms: p50 0.053  p99 1.828  max 12.682

terminal 1 (the API)

serving version 2026-10-30 (276,536 rows)
serving version 2026-10-31 (280,948 rows)

Observe: Zero errors. The service built the whole new snapshot (Bloom filter, database file, empty cache) next to the old one, then swapped a single reference. Requests in flight finished on the version they started with.
Why: The Bloom filter's "no false negatives" only holds for the keys it was built from, so filter, data and cache must change together, or not at all.

11. Experiment 5 · Try to publish broken data

post §11

a dataset that shrank overnight

python precompute.py --day 2026-11-01 --threshold 0.95

output

[2026-11-01] scored pairs: 5,910 relevant (ASIN, creative) pairs in 0.4s
refusing to publish: row count moved 98% (280,948 -> 5,910); rerun with --force if that is expected

Observe: The batch refuses, and version 2026-10-31 stays live. curl -s localhost:8000/healthz agrees.
Why: Data publishes are deploys too. A validation gate that keeps yesterday's version is the cheapest outage you'll never have.

12. Run the checks

the laptop path, and the AWS path against moto's fake DynamoDB and S3

pip install pytest moto
python -m pytest -q test_lab.py

output

3 passed

PART 2

The garbage collector

GcLab.java plays a relevance service: 20,000 requests a second on a fixed schedule, garbage on every request (key strings, lists, a response buffer), a 3-million-entry precomputed table, and a hot-key cache that churns. Latency is measured from when each request was supposed to start, so a pause shows up in every request it delays. Java 21 runs a single source file directly; no build step.

13. Three collectors, same workload

post §10

about 40 seconds each

java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseParallelGC GcLab.java objects
java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseG1GC GcLab.java objects
java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseZGC -XX:+ZGenerational GcLab.java objects

output, Parallel

objects layout, cache 1,000,000 entries, 20,000 req/s for 30s
latency ms: p50 0.003  p99 0.279  p99.9 99.954  max 127.9
collections: 1, stop-the-world pause time 127 ms

Collector	p99.9	Worst request	Stop-the-world time
Parallel	100 ms	128 ms	127 ms, one full collection
G1	55 ms	69 ms	264 ms across 5 pauses
Generational ZGC	1.3 ms	5.8 ms	under 1 ms across 16 cycles

Observe: The median request takes microseconds under every collector. The tail is where they differ, by almost a hundred times.
Why: Parallel stops everything and cleans the whole heap at once; G1 splits the work into shorter pauses; ZGC does almost all of it while your code keeps running. Your numbers will differ from my 4-core machine; the ranking shouldn't.

14. Now cap the cache instead

Parallel again, with a 100,000-entry cache

java -Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseParallelGC GcLab.java objects 100000

output

objects layout, cache 100,000 entries, 20,000 req/s for 30s
latency ms: p50 0.003  p99 0.012  p99.9 1.252  max 10.6
collections: 1, stop-the-world pause time 10 ms

Observe: Same collector, same table: the worst pause drops from about 127 ms to about 10 ms.
Why: The churning cache was most of the old generation's work. Holding the table as one flat array instead (GcLab.java array) barely changed anything here, because the cache, not the table, was the weight. Measure before you optimize: the playbook in the post is a list of suspects, not a list of fixes.

PART 3

On AWS

This part costs money while it runs (by default: two API hosts, a load-generator machine, an Application Load Balancer, an Elastic IP, and DynamoDB on demand). Do it in one sitting and finish with step 17. The template passes cfn-lint, and the DynamoDB and S3 code paths are covered by test_lab.py.

The commands below are for macOS or Linux shells (on Windows, use WSL or AWS CloudShell). They use the stack from aws/stack.yaml:

Data: a DynamoDB table (versions side by side, with a time-to-live so old ones expire) and a private S3 bucket for the Bloom filters and current.json.
The API: an Auto Scaling group of hosts behind an Application Load Balancer, with slow start and target tracking on requests per host.
A load generator: its own machine, which you drive through AWS Systems Manager, so nothing needs SSH.

15. Deploy the stack

post §09

check who you are, pick a region

aws sts get-caller-identity
export AWS_REGION=us-east-1 AWS_DEFAULT_REGION=us-east-1

find the default VPC, its subnets, and your IP

VPC=$(aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text)
SUBNETS=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC Name=default-for-az,Values=true \
  --query 'Subnets[].SubnetId' --output text | tr '\t' ',')
MYIP=$(curl -s https://checkip.amazonaws.com)/32
echo $VPC $SUBNETS $MYIP

deploy (about 5 minutes)

aws cloudformation deploy --stack-name relevance-lab --template-file aws/stack.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides VpcId=$VPC SubnetIds=$SUBNETS AllowedCidr=$MYIP

read the outputs into variables

out() { aws cloudformation describe-stacks --stack-name relevance-lab \
  --query "Stacks[0].Outputs[?OutputKey=='$1'].OutputValue" --output text; }
API=$(out ApiUrl); TABLE=$(out TableName); BUCKET=$(out BucketName)
ASG=$(out HostGroupName); LB=$(out LoadBalancerFullName); LOADGEN=$(out LoadGenInstanceId)
echo $API

The hosts start, install the lab, and answer /healthz with 503 until a data version exists. That's deliberate: the load balancer only sends traffic to hosts that have data.

16. Publish, test, scale, publish again

publish version 1 to DynamoDB and S3 (a few minutes for ~277,000 writes)

python precompute.py --day 2026-10-30 --table $TABLE --bucket $BUCKET

wait until the fleet has it (hosts check every 30 seconds)

until curl -sf $API/healthz; do sleep 10; done; echo
curl -s $API/relevance -H 'content-type: application/json' \
  -d '{"asin": "B000000042", "creatives": ["cr-179", "cr-396", "cr-7"]}'

You should get the same scores as on your laptop: same seed, same world, now served from DynamoDB. Next, load from the load generator (8 generator processes):

start a 3-minute load test on the generator, then read its output

CMD=$(aws ssm send-command --instance-ids $LOADGEN --document-name AWS-RunShellScript \
  --parameters "{\"commands\":[\"cd /opt/lab/relevance-lab && /opt/lab/venv/bin/python loadtest.py --url $API --seconds 180 --processes 8 --concurrency 64\"]}" \
  --query Command.CommandId --output text)
sleep 200
aws ssm get-command-invocation --command-id $CMD --instance-id $LOADGEN --query StandardOutputContent --output text

while it runs: watch the fleet react, and the load balancer's p99

aws autoscaling describe-scaling-activities --auto-scaling-group-name $ASG --max-items 5 \
  --query 'Activities[].[StartTime,Description]' --output table
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name TargetResponseTime \
  --dimensions Name=LoadBalancer,Value=$LB --extended-statistics p99 --period 60 \
  --start-time $(date -u -d '-20 min' +%FT%TZ 2>/dev/null || date -u -v-20M +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,ExtendedStatistics.p99]' --output table

Predict: Target tracking aims for 500 requests a second per host. When does a new host start helping?
Observe: The scaling activity appears within a few minutes, but the new host only takes its full share after it boots, installs, loads the data and finishes the load balancer's 60-second slow start. That gap is section 09 of the post, live.

pre-scale instead (the Prime Day move), then rerun the load test and compare p99

aws autoscaling set-desired-capacity --auto-scaling-group-name $ASG --desired-capacity 4
# or schedule it, the way you would before a known peak:
aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name $ASG \
  --scheduled-action-name before-the-peak --desired-capacity 4 \
  --start-time $(date -u -d '+10 min' +%FT%TZ 2>/dev/null || date -u -v+10M +%FT%TZ)

publish day 2 while the load test runs, and watch the fleet switch

python precompute.py --day 2026-10-31 --table $TABLE --bucket $BUCKET
for i in 1 2 3 4 5 6; do curl -s $API/healthz; echo; sleep 10; done

Within about 30 seconds every host reports 2026-10-31, and the load test's error count stays at zero: each host swaps its own snapshot, and DynamoDB keeps both versions' rows side by side (keys start with the version) until the old ones expire.

17. Clean up (don't skip this)

empty the bucket, delete the stack, wait

aws s3 rm s3://$BUCKET --recursive
aws cloudformation delete-stack --stack-name relevance-lab
aws cloudformation wait stack-delete-complete --stack-name relevance-lab && echo deleted

Deleting the stack removes the hosts, load balancer, load generator, Elastic IP, table, bucket and roles together. On your laptop, stop the API with Ctrl+C and delete the relevance-lab folder when you're done.

HELP

Troubleshooting

pip install fails on numpy: you're on Python older than 3.10. Install a newer Python and recreate the virtual environment.
The API says no data version loaded yet: run step 3 first, from the same folder you started the API in.
Port 8000 is busy: use --port 8001 for the API and --url http://localhost:8001 for the load test.
-XX:+ZGenerational is rejected: your Java is older than 21. Drop the flag (plain ZGC still works on 17), or install Java 21.
The stack fails with an instance type error: some regions or zones don't offer c7i. Add InstanceType=c6i.large LoadGenInstanceType=c6i.2xlarge to the parameter overrides.
curl $API hangs: your public IP changed since step 15. Redeploy with the new AllowedCidr.
Hosts never turn healthy: check that step 16's publish finished, then open a shell on a host without SSH: aws ssm start-session --target <instance-id> and run journalctl -u relevance.
The load generator returns nothing: it may still be installing; wait three minutes after the stack finishes and send the command again.

Back to the story

Every experiment here maps to a decision in A Million Questions a Second. If you ran it and found something surprising, I'd love to hear about it: tascano.github.io.