The $760.86 Google API Bill That Taught Me Cloud Cost Security From Scratch
Language Mismatch Disclaimer: Please be aware that the language of this article may not match the language settings of your browser or device.
Do you want to read articles in English instead ?
TL;DR
A Google API key bundled in my Expo mobile app got scraped and burned $644.80 of Places Autocomplete and Place Details requests over two days while I was not looking. After 18% Senegal VAT and currency-conversion fees, that worked out to roughly $760.86 out of my bank account for two days of someone else's traffic.
I had API restrictions on the key. I did not have application restrictions, a billing budget, or a kill switch.
What follows is the full course: the attack, the audit, the four-layer defense, and a working provisioning script that recreates the whole thing on a new project in under five minutes.
The Notification That Started It
It did not start with a billing email from Google. It started with an AMEX push notification on my phone: a charge from a vendor I had not authorized that month. My first thought was fraud on the card itself.
The vendor was Google.
I opened my Cloud billing dashboard and the shape of the spend was the giveaway: $0 every day for three weeks, then a sharp spike on April 7 ($150) and April 8 ($495). Two days, then back to zero.
That pattern is not human usage. Human usage has texture: small variations day to day, weekend dips, spikes that taper. Two flat days at the top, then nothing, is automation.
I knew exactly which API too. The "Places API (legacy)" line item dwarfed everything else. Places Autocomplete + Place Details, called server-to-server with no caching, no rate limit, no anything.
Someone had my key.
The Smoking Gun
I had three Google API keys at the time:
- A backend key — used by my Laravel server for Places, Geocoding, Directions
- A second backend key — used by a CRM-related scraping job
- An internal-tool key — used by a side service for Gemini
I ran gcloud services api-keys describe on each. The first one's restrictions read like this:
displayName: Android key (auto created by Firebase)
restrictions:
apiTargets:
- service: directions-backend.googleapis.com
- service: firebaseinstallations.googleapis.com
- service: geocoding-backend.googleapis.com
- service: maps-android-backend.googleapis.com
- service: places-backend.googleapis.com
# ...
apiTargets is the API allowlist. That part was fine: this key could only call Maps and Places APIs, not Cloud Storage or BigQuery.
What is missing from the YAML above is the smoking gun. There is no androidKeyRestrictions block. No iosKeyRestrictions. No serverKeyRestrictions. No browserKeyRestrictions. The key's display name says "Android key" because Firebase named it that, but Firebase did not actually restrict it to Android apps. It was open to anyone who held the value.
And the value was bundled inside my Expo mobile app, in app.config.js:
expo: {
extra: {
googleMapsApiKey: process.env.GOOGLE_MAPS_API_KEY,
},
ios: {
config: {
googleMaps: {
apiKey: process.env.GOOGLE_MAPS_API_KEY,
},
},
},
}
Anyone who downloads my IPA or APK can extract that string in seconds with strings or any reverse-engineering tool. Then they can call Places Autocomplete from any IP, with no Android header, no certificate, nothing. Google's only check is "is this key valid and is the API in the allowlist?". Both yes. They serve the request and bill it.
That is the full attack. There is no zero-day, no exotic vulnerability. It is the default behavior when you create a Firebase Android app and forget to add the package name + SHA-1 fingerprint to the auto-generated key.
What "Restricted" Actually Means
Google has two kinds of restriction on an API key, and you need both:
| Restriction type | Answers the question | Examples |
|---|---|---|
| API restrictions | What APIs can this key call? | Places, Geocoding, Maps SDK Android |
| Application restrictions | Who is allowed to use this key? | IP 1.2.3.4, Android package com.foo + SHA-1 cert, iOS bundle com.foo, HTTP referrer *.foo.com |
A key with only API restrictions is a public key. Anyone who reads it can use it to call those APIs from anywhere. Limiting which APIs they can call only limits the kind of bill they can run up, not whether they can run one up.
A key with only application restrictions but no API restrictions is also wrong: if someone discovers a way to forge the application context (cert spoofing, header injection, network-level abuse), they get access to every API in your project including the expensive ones.
You need both layers. Always.
The Four-Layer Defense
Here is the model I now run on every project. Each layer is independent. They escalate in blast radius — Layer 1 only kills the leaked key, Layer 4 disables every Google API for the whole project until manually re-enabled.
| Layer | What it stops | Mechanism |
|---|---|---|
| 1 — Key restrictions | A leaked key being usable from anywhere else | gcloud services api-keys with IP, bundle ID, package + SHA-1, or referrer |
| 2 — Per-API quotas | Slow burn on one specific API even with valid credentials | Cloud Console → APIs & Services → <API> → Quotas |
| 3 — Budget alerts | Slow burn that does not hit a quota cap | Cloud Billing budget → email + Pub/Sub |
| 4 — Hard kill switch | Anything Layers 1–3 missed | Pub/Sub → Cloud Function → cloudbilling.projects.updateBillingInfo with empty body |
I will walk through each.
Layer 1: Lock Every Key Down
This is the cheapest, highest-leverage step. Every key in your project should have both API restrictions and application restrictions, with as narrow a scope as you can tolerate.
Server keys (Laravel, Django, Node services calling Google APIs from a known IP):
gcloud services api-keys create \
--project=YOUR_PROJECT \
--display-name="myproject-server" \
--allowed-ips=YOUR_PROD_SERVER_IP \
--api-target=service=places-backend.googleapis.com \
--api-target=service=places.googleapis.com \
--api-target=service=geocoding-backend.googleapis.com \
--api-target=service=directions-backend.googleapis.com
If you have multiple servers, pass --allowed-ips multiple times. If you run on a NAT or load balancer, allowlist its egress IP, not the internal one.
Mobile keys (iOS Maps SDK):
gcloud services api-keys create \
--project=YOUR_PROJECT \
--display-name="myproject-mobile-ios" \
--allowed-bundle-ids=com.yourapp \
--api-target=service=maps-ios-backend.googleapis.com
Mobile keys (Android Maps SDK + Firebase):
gcloud services api-keys update YOUR_KEY_UID \
--project=YOUR_PROJECT \
--allowed-application=sha1_fingerprint=AA:BB:CC:...,package_name=com.yourapp \
--allowed-application=sha1_fingerprint=DD:EE:FF:...,package_name=com.yourapp \
--api-target=service=maps-android-backend.googleapis.com \
--api-target=service=maps-backend.googleapis.com \
--api-target=service=firebaseinstallations.googleapis.com
Critical for Android: you need both SHA-1 fingerprints — your upload key (used by EAS/Gradle to sign the APK before upload) and Google Play's app-signing key (which Google re-signs with before delivering to users). The upload key SHA is in your build credentials. The Play app-signing SHA is in Google Play Console → Test and Release → App integrity → App signing key certificate.
If you only add the upload key SHA, Maps will work in your internal builds and break for real users on the Play Store. Ask me how I know.
Audit existing keys:
gcloud services api-keys list --project=YOUR_PROJECT --format=json
Any key without an iosKeyRestrictions, androidKeyRestrictions, serverKeyRestrictions, or browserKeyRestrictions block in the output is unrestricted. Lock it down or delete it.
Delete keys you do not use. Firebase auto-creates a "Browser key" with permissions to call BigQuery, Cloud SQL admin, Cloud Storage, and 50+ other APIs, with browserKeyRestrictions: {} (empty allowlist). If you are not using Firebase web SDK, delete that key. If you are, lock it to your own domain.
Layer 2: Per-API Quota Caps
Even if a key is properly restricted, you can add a second guarantee: hard daily caps on the APIs that drive cost.
This is a one-time Cloud Console step (no clean gcloud equivalent for some metrics):
- Console → APIs & Services → <API> → Quotas & System Limits
- Find the relevant per-day metric (e.g. "Place Autocomplete requests per day")
- Edit → set to a number well above your normal usage but well below pain (e.g. 10,000/day)
Once a quota is exhausted, Google starts returning 429 for that specific API for the rest of the day. Other APIs keep working. It is the most surgical defense layer.
For my own project I cap (or plan to cap, this is the layer I am still tightening) Places Autocomplete at 10k/day, Place Details at 5k/day, and Geocoding at 5k/day. Actual daily usage is in the dozens, so the cap leaves three orders of magnitude of headroom for legitimate growth and zero room for a leaked-key flood.
Layer 3: Budget With Email Alerts
This is the slow-burn detector. A leaked key getting hit at low frequency might not trip a per-API quota for weeks. A monthly budget will tell you when total spend crosses meaningful thresholds.
gcloud billing budgets create \
--billing-account=YOUR_BILLING_ACCOUNT \
--display-name="myproject monthly hard cap" \
--budget-amount=60USD \
--threshold-rule=percent=0.5 \
--threshold-rule=percent=0.75 \
--threshold-rule=percent=0.9 \
--threshold-rule=percent=1.0 \
--notifications-rule-pubsub-topic=projects/YOUR_PROJECT/topics/your-budget-topic \
--filter-projects=projects/YOUR_PROJECT_NUMBER
A few things worth knowing:
- Budgets are billing-account-scoped, not project-scoped. Without
--filter-projects, a budget on a billing account that funds five projects will trigger when any of them collectively exceed the threshold. Almost always not what you want. --filter-projectstakes the project number, not the project ID. (Find it withgcloud projects describe YOUR_PROJECT --format='value(projectNumber)'.)- The default email recipients are anyone with
roles/billing.adminorroles/billing.useron the billing account. If you are the project owner, that is already you. If you want a different email, you have to wire up a Cloud Monitoring notification channel — usually overkill for one address.
The budget alone does not stop spend. It just publishes a Pub/Sub message and sends an email. If you sleep through the email, the spend keeps going. Which brings us to the last layer.
Layer 4: The Hard Kill Switch
This is Google's documented "cap costs" pattern: when a Pub/Sub notification crosses your hard threshold, a Cloud Function detaches billing from the project, which causes every Google API to start returning BILLING_DISABLED until you manually re-attach.
It sounds extreme. It is. That is the point — it is the only mechanism that actually stops spend. You set the trigger to 100% (or 200% if you want a buffer above the alert thresholds), and you let it fire if everything else fails.
The Function
# devops/gcp/killswitch/main.py
import base64
import json
import os
from googleapiclient import discovery
PROJECT_ID = os.environ["PROJECT_ID"]
DISABLE_AT_PERCENT = float(os.environ.get("DISABLE_AT_PERCENT", "1.0"))
def stop_billing(event, context=None):
payload = base64.b64decode(event["data"]).decode("utf-8")
data = json.loads(payload)
cost = float(data.get("costAmount", 0) or 0)
budget = float(data.get("budgetAmount", 0) or 0)
if budget <= 0:
print(f"Skip: budget is zero/missing in payload: {data}")
return
ratio = cost / budget
if ratio < DISABLE_AT_PERCENT:
print(f"Under threshold: cost={cost} budget={budget} ratio={ratio:.2f}")
return
billing = discovery.build("cloudbilling", "v1", cache_discovery=False)
project_name = f"projects/{PROJECT_ID}"
info = billing.projects().getBillingInfo(name=project_name).execute()
if not info.get("billingEnabled", False):
print(f"Already disabled: {PROJECT_ID}")
return
response = billing.projects().updateBillingInfo(
name=project_name,
body={"billingAccountName": ""},
).execute()
print(f"DISABLED billing for {PROJECT_ID}: {response}")
# devops/gcp/killswitch/requirements.txt
google-api-python-client==2.155.0
google-auth==2.36.0
The function is idempotent: if billing is already detached when the next message fires, it logs and returns. No infinite loop, no retries doing damage.
The IAM dance
The function needs to call cloudbilling.projects.updateBillingInfo. That requires:
- On the project:
roles/billing.projectManager(containscloudbilling.projects.updateBillingInfo) - On the billing account:
roles/billing.user(lets it know it can detach itself)
Plus, since Cloud Functions Gen 2 actually run on Cloud Run under the hood, the Eventarc trigger needs roles/run.invoker on the underlying Run service. Without this last one, every Pub/Sub message returns 401 and your kill switch silently does nothing. Ask me how I know that one too.
The provisioning script
I packaged everything into a single idempotent script. Save as devops/scripts/gcp-bootstrap.sh, edit the config section at the top, and run.
#!/usr/bin/env bash
set -euo pipefail
# ─── Config ─────────────────────────────────────────────────────────────────
PROJECT_ID="myproject"
PROJECT_NUMBER="123456789"
BILLING_ACCOUNT="ABCDEF-123456-7890AB"
REGION="europe-west1"
BUDGET_AMOUNT_USD="60"
BUDGET_TOPIC="myproject-budget-alerts"
THRESHOLD_PERCENTAGES=(0.5 0.75 0.9 1.0)
KILLSWITCH_FN_NAME="myproject-billing-killswitch"
KILLSWITCH_SA_NAME="myproject-killswitch"
KILLSWITCH_SA_EMAIL="${KILLSWITCH_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
DISABLE_AT_PERCENT="1.0"
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
KILLSWITCH_SOURCE_DIR="${REPO_ROOT}/devops/gcp/killswitch"
log() { printf "▶ %s\n" "$*"; }
# ─── 1. Required APIs ───────────────────────────────────────────────────────
log "Enabling APIs"
gcloud services enable \
billingbudgets.googleapis.com \
cloudbilling.googleapis.com \
pubsub.googleapis.com \
cloudfunctions.googleapis.com \
cloudbuild.googleapis.com \
run.googleapis.com \
eventarc.googleapis.com \
artifactregistry.googleapis.com \
--project="${PROJECT_ID}"
# ─── 2. Pub/Sub topic ───────────────────────────────────────────────────────
log "Ensuring Pub/Sub topic"
gcloud pubsub topics describe "${BUDGET_TOPIC}" --project="${PROJECT_ID}" >/dev/null 2>&1 \
|| gcloud pubsub topics create "${BUDGET_TOPIC}" --project="${PROJECT_ID}"
# ─── 3. Service account + IAM ───────────────────────────────────────────────
log "Ensuring killswitch service account"
gcloud iam service-accounts describe "${KILLSWITCH_SA_EMAIL}" --project="${PROJECT_ID}" >/dev/null 2>&1 \
|| gcloud iam service-accounts create "${KILLSWITCH_SA_NAME}" \
--display-name="Billing Killswitch" --project="${PROJECT_ID}"
log "Granting IAM"
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
--member="serviceAccount:${KILLSWITCH_SA_EMAIL}" \
--role="roles/billing.projectManager" >/dev/null
gcloud billing accounts add-iam-policy-binding "${BILLING_ACCOUNT}" \
--member="serviceAccount:${KILLSWITCH_SA_EMAIL}" \
--role="roles/billing.user" >/dev/null
# ─── 4. Cloud Function ──────────────────────────────────────────────────────
log "Deploying Cloud Function"
gcloud functions deploy "${KILLSWITCH_FN_NAME}" \
--gen2 --runtime=python312 --region="${REGION}" \
--source="${KILLSWITCH_SOURCE_DIR}" \
--entry-point=stop_billing \
--trigger-topic="${BUDGET_TOPIC}" \
--service-account="${KILLSWITCH_SA_EMAIL}" \
--set-env-vars="PROJECT_ID=${PROJECT_ID},DISABLE_AT_PERCENT=${DISABLE_AT_PERCENT}" \
--max-instances=1 --memory=256Mi --timeout=60s \
--project="${PROJECT_ID}"
log "Granting run.invoker on the Cloud Run service to the killswitch SA"
gcloud run services add-iam-policy-binding "${KILLSWITCH_FN_NAME}" \
--region="${REGION}" --project="${PROJECT_ID}" \
--member="serviceAccount:${KILLSWITCH_SA_EMAIL}" \
--role="roles/run.invoker" >/dev/null
# ─── 5. Budget ──────────────────────────────────────────────────────────────
log "Ensuring budget"
THRESHOLD_FLAGS=()
for pct in "${THRESHOLD_PERCENTAGES[@]}"; do
THRESHOLD_FLAGS+=(--threshold-rule="percent=${pct}")
done
EXISTING_BUDGET="$(gcloud billing budgets list \
--billing-account="${BILLING_ACCOUNT}" \
--filter="displayName='myproject monthly hard cap'" \
--format='value(name)' 2>/dev/null | head -n1)"
if [[ -z "${EXISTING_BUDGET}" ]]; then
gcloud billing budgets create \
--billing-account="${BILLING_ACCOUNT}" \
--display-name="myproject monthly hard cap" \
--budget-amount="${BUDGET_AMOUNT_USD}USD" \
"${THRESHOLD_FLAGS[@]}" \
--notifications-rule-pubsub-topic="projects/${PROJECT_ID}/topics/${BUDGET_TOPIC}" \
--filter-projects="projects/${PROJECT_NUMBER}"
else
gcloud billing budgets update "${EXISTING_BUDGET##*/}" \
--billing-account="${BILLING_ACCOUNT}" \
--display-name="myproject monthly hard cap" \
--budget-amount="${BUDGET_AMOUNT_USD}USD" \
"${THRESHOLD_FLAGS[@]}" \
--notifications-rule-pubsub-topic="projects/${PROJECT_ID}/topics/${BUDGET_TOPIC}" \
--filter-projects="projects/${PROJECT_NUMBER}"
fi
echo "Done."
The script is idempotent. Re-running it after editing the budget amount, threshold percentages, or topic name will update the live resources in place.
Testing the kill switch without nuking your billing
Publish a synthetic Pub/Sub message that intentionally stays below the threshold and check the function logs:
gcloud pubsub topics publish myproject-budget-alerts \
--project=myproject \
--message='{"costAmount":15,"budgetAmount":60}'
gcloud functions logs read myproject-billing-killswitch \
--region=europe-west1 --project=myproject --limit=5
You should see Under threshold: cost=15.0 budget=60.0 ratio=0.25 within 30 seconds. If you instead see The request was not authenticated warnings, the Eventarc trigger does not have run.invoker on the Cloud Run service. Add it.
When the kill switch fires for real
- The function logs
DISABLED billing for myprojectand returns. - Within seconds, every Google API call from your services starts returning
BILLING_DISABLEDerrors. Maps stops working. Storage buckets become read-only. Cloud SQL connections fail. - You investigate which API or which key drove the spend, and fix it.
- You re-attach billing:
gcloud billing projects link myproject --billing-account=ABCDEF-123456-7890AB
- The kill switch automatically re-arms for the next month.
Same Defense on AWS, Different Mechanics
The four-layer model is cloud-agnostic. The mechanisms are not. The same week I rebuilt the GCP guardrails I shipped the AWS equivalent, and the kill switch in particular looks completely different even though the contract is the same: "at 100% of monthly budget, take an action that stops spend."
How each layer maps
| Layer | GCP mechanism | AWS mechanism |
|---|---|---|
| 1 — Credential restrictions | gcloud services api-keys with IP / bundle / package + SHA / API allowlists |
IAM policies scoped to specific bucket ARNs, SES regions, SQS queue ARNs. No wildcards in Resource. |
| 2 — Per-API caps | Cloud Console → APIs & Services → Quotas | Mostly N/A — AWS doesn't have client-facing per-API quotas the same way; service-specific quotas exist (SES daily send) but are managed differently. |
| 3 — Budget alerts | Cloud Billing budget → email + Pub/Sub | AWS Budgets → email + SNS |
| 4 — Hard kill switch | Pub/Sub → Cloud Function (Python) → cloudbilling.projects.updateBillingInfo with empty body, detaches billing |
AWS Budget Action (built-in feature) → attaches a DenyAll IAM policy to runtime IAM users |
Layer 4 is where it gets interesting
GCP forced me to write code: a Pub/Sub topic, a service account with the right IAM, a Cloud Function in Python, an Eventarc trigger, and the cloudbilling.projects.updateBillingInfo call that actually severs billing. About 60 lines of Python plus an idempotent provisioning script.
AWS gave me the kill switch as a built-in feature. AWS Budgets has a concept called Budget Actions that fire automatically when spend crosses a threshold. The available actions include "apply this IAM policy to these users" out of the box. No code, no Lambda, no glue.
The blast radius is also different by design:
-
GCP at 100% detaches billing from the entire project. Every Google API call from that project starts returning
BILLING_DISABLED. Maps stops working. Storage buckets become read-only. Cloud SQL refuses connections. The whole project is effectively offline until you re-attach billing. -
AWS at 100% attaches a
DenyAllpolicy to a curated list of IAM users. Their API calls returnAccessDenied. Workloads stop, but admin keys and other-project keys keep working. Reversal is a per-useraws iam detach-user-policy, no billing dance.
I prefer AWS's blast radius shape. Killing only the runtime users means the admin user is still alive and can detach the policy without a chicken-and-egg problem. On GCP, when billing is detached you can re-attach it from any account that has Billing Admin on the billing account, but you can't run any API call against the project until that's done.
The AWS provisioning script (idempotent, run once)
Same shape as the GCP version. Edit the config block at the top, run the script, re-run safely.
#!/usr/bin/env bash
set -euo pipefail
ACCOUNT_ID="123456789012"
BUDGET_REGION="us-east-1" # AWS Budgets only runs here
BUDGET_NAME="myproject monthly hard cap"
BUDGET_AMOUNT_USD="200"
SNS_TOPIC_NAME="myproject-budget-alerts"
DENY_POLICY_NAME="MyprojectBudgetExceededDenyAll"
ALERT_EMAIL="[email protected]"
# Runtime IAM users that get DenyAll attached at 100%.
# Spare your admin user so the kill is reversible.
KILL_TARGET_USERS=("myproject-app" "myproject-worker" "myproject-ses")
# 1. DenyAll IAM policy
aws iam create-policy --policy-name "$DENY_POLICY_NAME" \
--policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*"}]}'
# 2. Budget Action execution role (assumed by budgets.amazonaws.com)
aws iam create-role --role-name "MyprojectBudgetActionRole" \
--assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"budgets.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name "MyprojectBudgetActionRole" \
--policy-arn arn:aws:iam::aws:policy/AWSBudgetsActions_RolePolicyForResourceAdministrationWithSSM
# 3. SNS topic + email subscription
aws sns create-topic --name "$SNS_TOPIC_NAME" --region "$BUDGET_REGION"
aws sns subscribe --topic-arn "arn:aws:sns:$BUDGET_REGION:$ACCOUNT_ID:$SNS_TOPIC_NAME" \
--protocol email --notification-endpoint "$ALERT_EMAIL" --region "$BUDGET_REGION"
# 4. Budget + Budget Action
# (See full version: thresholds at 50/75/90/100, IAM policy attachment at 100.
# Skipped here for brevity; full script in the linked repo.)
The full version (with budget thresholds, the SNS topic publish-policy, and the Budget Action wired up) is one shell script you run once and forget. It is structurally simpler than the GCP version because there's no Cloud Function to deploy.
When you should do both
Most projects don't run on just one cloud. Even if AWS is the main workload, you probably have GCP for one or two services that grew sideways: Maps, Firebase, BigQuery, Workspace integrations, ML APIs.
That's exactly how I got bitten. My main spend has always been AWS — I watch it. The GCP project was a side dependency for Maps that I rarely opened. The leaked key was on the cloud I wasn't looking at.
The lesson is to set up guardrails on every cloud you use, even the small ones. The cloud you're not paying attention to is where the next surprise lives.
Application-Level Belt-and-Suspenders
Layers 1–4 are infrastructure-side. None of them help if the abuser is going through your own backend (e.g., your /api/place proxy with valid auth tokens stolen from a real user).
For the cases where the abuse comes through your own app, you want application-level rate limits with sane per-IP and global ceilings. In Laravel:
RateLimiter::for('place', function (Request $request) {
return [
Limit::perMinute(30)->by($request->user()?->getKey() ?: $request->ip()),
Limit::perHour(200)->by($request->user()?->getKey() ?: $request->ip()),
Limit::perDay(500)->by($request->user()?->getKey() ?: $request->ip()),
Limit::perDay(5000), // global ceiling, no `by` clause
];
});
The global ceiling is the important one. Per-user limits do not help if the attacker rotates user accounts or scrapes your own auth flow.
You can also keep a Redis daily counter that increments on every Place lookup and hard-blocks the route when the counter exceeds a number you choose. Belt-and-suspenders against keys you have not yet identified as leaked.
$key = "places:daily:" . now()->format("Y-m-d");
$count = Redis::incr($key);
Redis::expire($key, 60 * 60 * 26);
if ($count > config('services.places.daily_hard_cap', 5000)) {
abort(503, 'Daily Places cap reached');
}
Asking Google for a Refund
After the rotation was complete and the abuse had visibly stopped, I filed for a refund of the $644.80. This part of the story is still unfolding at the time of writing — Google has not yet decided either way — but the process is worth documenting because it is more friction than it should be and a few small choices made it noticeably less painful.
The path is no longer a static form
I expected to land on a "Request a Refund" form with an upload field. That form does not exist anymore. Cloud Billing support is now fronted by an AI assistant. You open the assistant, describe the problem, and the assistant is supposed to triage you to either self-help docs or a human specialist. The assistant cannot issue refunds itself — it routes the case once it concludes (or you insist) that human review is needed.
Open the case in writing, not in chat
The assistant offers two escalation paths: live chat and email. Email is the right choice for a refund dispute, for three reasons:
- Async means thorough. A refund needs argued — citations, timestamps, key UIDs, the exact API that was abused. Chat invites pressure to wrap things up and skip evidence.
- Paper trail. Email is permanent, attachable, citable in follow-ups. Chat transcripts get fragmented and harder to forward to whoever ultimately approves the refund.
- The decision is asynchronous anyway. Refund cases get handed between an initial reviewer and a specialist who actually has authority. Email fits the workflow; chat creates a fake sense of immediacy.
What to put in the email
The strongest version of the refund request reads like a postmortem, not a complaint. Sections I included:
- Executive summary. What happened, in one paragraph, with the dollar amount and the ask.
- Timeline. Daily values around the abuse window, plus the detection date. The shape of the spend ($0/day for 21 days, then a 2-day spike, then $0 again) is the strongest single piece of evidence — that pattern is impossible from human usage.
- Root cause. Where the key leaked from (the Expo bundle), why it was vulnerable (no application restrictions despite the misleading "Android key" name), and the leaked key's UID so a specialist can audit it.
- Remediation. Five timestamped actions with the gcloud audit log as the source of truth: when the key was restricted, what was added (package + SHA-1 fingerprints), what was narrowed (API targets), what was created (the new IP-locked server key), what was deleted (unused auto-created keys).
- Defense layers added. The four-layer model from the rest of this post. The point: this is not a "fix it for me" request, it is a "we already fixed it, please refund the abuse charges from before the fix" request.
- The ask. Refund to original payment method, not in-account credit. Acknowledge VAT separately so the specialist does not have to do the math.
Tone matters
A refund request that opens with frustration gets routed to the slow queue. The same request that opens with "we are requesting a refund of $X for fraudulent usage caused by a leaked credential we have since rotated" reads as professional cleanup, not a customer who needs hand-holding. The technical depth is what gets it taken seriously — the case is no longer about whether the user is competent, it is about whether the abuse meets the policy bar for a refund.
Driving the AI toward escalation
If the assistant tries to help with self-service ("here is how to set up a budget"), reject it politely and ask explicitly for a billing specialist. A clear escalation request usually gets you a case number within a few exchanges. Once you have a case number, the case is in the human queue and the AI hands off.
Outcome (pending)
I am still waiting for Google's reply at the time of writing this post. I will update this section when there is news. The realistic outcomes are:
- Full refund. Best case. Google has historically credited fraudulent abuse charges when the leaked-key vector is clearly documented, the user has already remediated, and the traffic shape is unambiguous.
- Partial refund. They might issue, say, half the amount as a goodwill gesture without admitting the abuse pattern.
- Denial. If denied, the next move is an AMEX chargeback. That works but it damages the relationship with Google for any future ad / cloud spend, so it is a last resort, not a first option.
Whatever happens, the refund is the smaller win. The bigger win is the four-layer defense that means there will not be a next $645 incident to ask about.
Follow-up: what the Escalation Manager asks for
The case did get routed. Two hops in (initial reviewer → Maps Billing team → Specialized Team → Escalation Manager), a human named Shane replied with the actual refund-request rubric. It is worth reproducing in detail because every founder who ends up in this queue will get the same six asks, and the request goes faster if the answers are pre-built.
The six asks, in order:
- Apply a hard quota first. Before they will even submit the credit request, they want application-level quotas in place. Default per-API quotas on Google Cloud are effectively unlimited (
9223372036854775807requests/day on the legacy Places API). They want to see numbers that match your real traffic — typical caps are 2,000–5,000 requests/day per API. This is a precondition, not a follow-up. - Usage Context. What does the project do, and which APIs are you legitimately calling? Two-paragraph product description plus the API list.
- The Incident. Root cause, detection method, mitigation. The strongest version is one paragraph for the leak vector (where the key was embedded, why it was vulnerable), one paragraph for the abuse pattern (the spend shape), one bullet list for remediation steps with timestamps.
- Data Caching confirmation. They explicitly ask whether you retained any lasting benefit from the abusive traffic. The right answer for a stolen-key abuse case is "no, the traffic was attacker-driven, never reached our backend, nothing to delete." If you did cache Places responses anywhere in your normal flow, say so and confirm those caches are within the Places ToS retention window (30 days for most fields).
- Terms of Service alignment. Confirm your legitimate use case fits the ToS. The abusive traffic is not your usage, so it does not bear on this answer.
- Future Liability acknowledgment. A one-line confirmation that you take responsibility for valid charges going forward. They ask explicitly because the credit is one-time courtesy, not a precedent.
How to write the reply
Match the structure of the email to the structure of the asks. Six headings, in the order they were asked, each one or two paragraphs. No preamble, no summary. The Escalation Manager forwards the email up the chain to a Specialized Team that has approval authority — make their copy-paste job easy.
The phrase that does heavy lifting in the reply: "the April 7-8 traffic was unauthorized third-party abuse of a leaked key, not
What I would not include:
- A link to a public write-up of the incident. Even if your post is honest and helpful, public disclosure of API key abuse adds reviewer-side friction (some readers see it as ToS-adjacent risk, even though it is not). Keep the case private to the case.
- Frustration. Even mild frustration ("this should not have been possible at the default quota") slows the queue.
- An estimate of how long the work took or what it cost in stress. Not a refund factor.
The reply, as a template
Below is the reply I sent, with project-specific bits replaced by {{placeholders}}. Copy it, fill in your own incident details, and send. Headings match Shane's six asks one-for-one so the Specialized Team can scan it without parsing prose.
Subject: Re: Refund request — {{API_NAME}} charges (case [#])
Hi {{ESCALATION_MANAGER_NAME}},
Investigation is complete and the hard quotas are now in place. Answers
to your five questions below.
Hard quota — Applied. {{API_NAME}} capped at {{DAILY_CAP}}/day, with
matching per-minute-per-user limits. {{OTHER_APIS}} all tightened to
~3-5× normal daily volume. Worst-case daily ceiling across the project
is now ≈ ${{WORST_CASE_DAILY}}. Screenshots attached.
Usage Context
{{PRODUCT_NAME}} ({{URL}}) is {{ONE_LINE_DESCRIPTION}}. We use
{{GOOGLE_PRODUCT}} for:
- {{LEGITIMATE_USE_1}}
- {{LEGITIMATE_USE_2}}
- {{LEGITIMATE_USE_3}}
The Incident
The compromised key was {{KEY_PROVENANCE — e.g. "a Firebase-provisioned
Android key embedded in our Expo mobile app bundle"}}. {{WHY_VULNERABLE
— specific reason restrictions were missing}}. The key carried API-target
restrictions ({{ALLOWED_APIS}}) but was usable from anywhere by anyone who
extracted the string from {{ATTACK_SURFACE — e.g. "the APK or IPA"}}.
Between {{INCIDENT_DATES}}, the key was extracted and replayed
{{TRAFFIC_SHAPE — e.g. "server-to-server against Places Autocomplete and
Place Details"}}. The billing pattern was unmistakably automated:
{{SPEND_SHAPE — e.g. "zero spend for the prior three weeks, then $150 on
day one, $495 on day two, then back to zero"}} — a signature inconsistent
with any human usage of our app.
Discovery: detected via {{DETECTION_METHOD}}, not via Google's billing
alerts (we did not have alerts configured at the time — that gap is now
closed).
Mitigation deployed (four layers):
1. Key restrictions — every key now carries both API restrictions and
application restrictions ({{SPECIFIC_RESTRICTIONS}}). The compromised
key has been rotated and the old value revoked.
2. Hard per-API quotas — applied today as described above.
3. Budget alerts — Cloud Billing budget configured with email
notifications at 50% / 75% / 90% / 100% of expected monthly spend.
4. Kill switch — a Cloud Function listens on the billing Pub/Sub topic
and automatically detaches the billing account from the project once
spend crosses 100% of budget, so any further requests fail closed with
BILLING_DISABLED rather than continuing to accrue.
Data Caching
No lasting benefit was retained from this usage. The traffic was
attacker-driven, not initiated by our application or users, so no
responses ever reached our backend. Our application stores only
{{WHAT_YOU_LEGITIMATELY_STORE}} — never {{WHAT_YOU_DO_NOT_STORE}}.
Nothing to delete on our side.
Terms of Service
Confirmed — our legitimate use case ({{ONE_LINE_TOS_FIT}}) aligns with the
{{GOOGLE_PRODUCT}} Terms of Service. The {{INCIDENT_DATES}} traffic was
unauthorized third-party abuse of a leaked key, not {{PRODUCT_NAME}}
usage.
Future Liability
Confirmed and understood. I take full responsibility for all valid charges
going forward. This request is for a one-time courtesy credit covering
the unauthorized {{INCIDENT_DATES}} anomaly only.
Let me know if you need anything else for the submission.
Thanks,
{{YOUR_NAME}}
{{YOUR_TITLE}}, {{COMPANY}}
Billing account: {{BILLING_ACCOUNT_ID}}
Project: {{PROJECT_ID}}
The phrasing that does the most work in this reply: "unauthorized third-party abuse of a leaked key, not {{PRODUCT_NAME}} usage." That single sentence reframes the case from "customer asking for a discount" to "fraud cleanup." It is what the Specialized Team scans for before deciding whether to route the request to the courtesy-credit queue or the policy queue.
What changed about my baseline after the exchange
Before this exchange I thought "set a budget alert" was sufficient. Shane's first ask reframed that — budget alerts tell you after the spend has happened. Hard quotas prevent the spend from happening at all, and the support team will not even open a courtesy credit case until quotas are in place. That ordering tells you something about how Google internally weighs prevention vs detection: prevention is the precondition for any goodwill credit. If the next incident happens after the courtesy credit, with quotas still missing, the answer will be no.
The lesson generalizes: any time you are asking a vendor for goodwill on a self-inflicted bill, the vendor's first question will be "what changed so this does not happen again?" Have the answer ready, with screenshots, before you open the case.
Update: Google issued a $683 refund
About two weeks after I sent the reply, Shane wrote back with the decision. A refund adjustment of $683.00 is being processed against a total cost of $760.86, returned to the original payment method and landing in the bank in 7 to 10 business days, with a 30-day window to reopen the case. Roughly 89.7% of the damage, recovered without an AMEX chargeback.
Worth flagging the wording. Shane's email reads "refund adjustment", not "account credit". The money flows back to the card I paid with, not into a Cloud Billing balance that I would have to burn down by spending more on Google. Some vendors default to credit-only on goodwill adjustments, which is a strictly worse outcome for cash. If your own case lands as "credit", push back and ask for the refund path explicitly.
The phrasing in the resolution email is worth quoting, because it tells you what Google is not saying:
The final review confirmed the validity of the charges but I took the liberty of advocating for your situation [...] While they weren't able to approve the full amount, they understand your situation and the billing adjustment request has been approved.
Three things to read out of that paragraph.
"The validity of the charges" was confirmed. Google is explicitly not classifying the April 7-8 spend as abuse. From their side, the charges were valid: the API was called, the key was authentic, the billable units were served. The refund is a goodwill adjustment, not a fraud reversal. That language preserves their right to bill any future incident on the same vector, and it is why the case file matters more than the outcome amount: the next incident on the same project will be read against this one.
The Escalation Manager personally advocated. "I took the liberty of advocating for your situation" is policy-aware phrasing. Shane forwarded the case to the internal review team, the team confirmed the charges were valid (i.e., the rubric answers did not strictly satisfy refund policy), and Shane then went back and argued for an adjustment on top of that. The rubric is not the gate. It is the brief that the Escalation Manager uses to argue the case once the formal policy review comes back unfavorable. Make the brief easy to argue from.
Partial, not full. The unrefunded gap is $77.86, against an original VAT line of $116.06 (18% of the $644.80 base). The refund therefore covers the full pre-tax API spend plus roughly a third of the VAT. My read is that Google refunds the underlying API charges cleanly through Cloud Billing but stops short of returning the full tax line, since that would touch tax-authority reconciliation rather than their internal billing-adjustment system. Worth knowing if your own ask includes VAT, import duty, or currency-conversion fees: assume the cloud platform will refund the pre-tax line cleanly and the tax line only partially or not at all.
Net: the four-layer defense plus a well-structured case, written as a postmortem rather than a complaint, earned back ~90% of the loss in about three weeks of correspondence. The remaining $77.86 is the cost of the lesson, and noticeably less than I would have paid for any course that taught the same material.
What I Should Have Done From Day One
Three things, in order of priority:
1. Application restrictions on every key from creation. If a key gets created without a package + SHA, IP, or bundle, treat that as a P0. The default Firebase auto-created "Android key" is a trap. Rename it, lock it down, narrow its API targets, before you ship anything that uses it.
2. A monthly budget on every project before the first paying customer. A budget with email alerts is free. The Pub/Sub kill switch is free up to 2 million invocations a month. There is no reason not to have these from day one. Set the budget low — $5 or $10 if you have no idea yet what your usage will look like. You will get an email at 50% within hours of any abnormal usage and you will know exactly when to investigate.
3. Treat every embedded credential as public. Anything in a mobile bundle, a JS web bundle, a public GitHub repo, or a Docker image layer is public. Restrict it as if you have already published it on a billboard. If a credential cannot tolerate being public, it cannot be embedded — proxy it through your backend instead.
The Takeaway
The Apr 7-8 incident was a rite of passage. Every founder running Google Cloud, AWS, Azure or any pay-as-you-go cloud account will eventually have a moment like this — a number in the dashboard that should not be there, with no obvious cause, and no signal to act sooner.
The cheapest version of this lesson is the version where you read someone else's postmortem and ship the four-layer defense before you ever get the bill. That is what this post is for.
The provisioning script and Cloud Function code are above. Edit the config block, run the script, verify the wiring with a test Pub/Sub message, and move on. Total time investment: 30 minutes the first time, 2 minutes for every project after.
Whatever you spend setting this up will be the cheapest insurance you ever buy.
