Find out whether the cause is you or the upstream
Check the provider's status page, the deployment or commit status, and the provider's changelog for the day it broke. Reproduce the failing call with curl from outside your infrastructure and compare the status code with what your app logs. On Vercel, a deploy hook that fails to create a deployment shows its error in the status check on the associated commit.
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 https://api.example.com/health
Put a timeout and a degraded mode on every upstream call
AbortSignal.timeout() aborts the request after a fixed time and rejects with a TimeoutError. Catch it, log it with the dependency's name, and serve a stale or reduced result instead of hanging. Mark the value as stale so the UI does not present it as fresh.
async function getPrice(id: string) {
try {
const res = await fetch(`https://api.example.com/prices/${id}`, {
signal: AbortSignal.timeout(3000),
});
if (!res.ok) throw new Error(`upstream ${res.status}`);
return { ...(await res.json()), stale: false };
} catch (err) {
console.error("upstream_failed", { dep: "prices", id, err: String(err) });
return { ...lastKnownPrice(id), stale: true };
}
}
Pin the API version and test upgrades before they reach production
Set the version in code instead of relying on a dashboard default. For Stripe, pass apiVersion to the client and try a newer version in a sandbox first. An existing webhook endpoint keeps the version it was created with, so to move webhooks, create a new endpoint with the target api_version. While both endpoints exist, process events from one of them, then disable the old one.
import Stripe from "stripe";
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!, {
apiVersion: "2026-10-28.endive", // the version you tested
});
Run a contract test against the real service on a schedule
Call the live provider with a test credential from CI on a schedule, and assert the shape you depend on: status code, field names and types. A change in format fails this job before it fails a customer. Send the failure to a channel someone reads.
status=$(curl -s -o /tmp/body.json -w "%{http_code}" --max-time 5 \
https://api.example.com/v1/prices/test)
[ "$status" = "200" ] && jq -e '.amount | type == "number"' /tmp/body.json
Make the health check and the alert exercise the real path
A health endpoint should make one cheap call to each critical dependency and report the result. Alert on the upstream error rate and on stale data, not only on your own 5xx. Count a refresh that returns an empty or unchanged payload as a failure.
Clear account-level and edge-level blockers
Check billing, trial end dates, credits and quotas for every provider on the critical path, and renew or raise them before launch. On Google Cloud, quotas are managed on the Quotas & System Limits page under IAM & Admin. Raising one needs the Quota Administrator role and a request that is reviewed. On Vercel, add a bypass rule for trusted automation instead of leaving it behind a challenge rule. Pro and Enterprise plans also have System Bypass Rules. 'Pause System Mitigations' lasts 24 hours and leaves you responsible for usage from traffic that would have been blocked.
Recover missed events and keep a fallback for the critical path
Stripe retries undelivered live-mode events for up to three days with exponential backoff, and can resend one on demand. Make handlers idempotent, because events can arrive more than once and out of order. For hosting, keep a second route ready: during one provider incident, builders pointed DNS at a direct A record. That removes the proxy's protection, so use it as a temporary measure.
stripe events resend <event_id> --webhook-endpoint=<endpoint_id>