Why Your Payment Endpoint Charged Someone Twice

Why Your Payment Endpoint Charged Someone Twice

A customer clicked to pay once but was charged twice. The frontend showed a spinning loading indicator until it timed out. The client sent only one POST /payments request. Support tickets arrived for double charges. Logs showed two charges 31 seconds apart, even though the client didn't resend explicitly.

Here's what happened: your load balancer timed out after 30 seconds without a response. The client's HTTP library retried automatically, unaware the first charge had succeeded. The server processed the payment and sent the response, but it never reached the client. That retry caused a second charge, with only one user click.

The client cannot distinguish "the request failed" from "the response failed." Every retry is a bet on which one happened.

Why retries cannot be disabled in distributed systems

Networks drop packets, servers return temporary errors (5xx), and connections break. Retries at multiple layers (client, load balancer, server) keep systems responsive and available. You cannot just turn retries off; they keep your application working under real-world conditions. Multiple retries at different layers multiply attempts.

Marc Brookers article in Amazon Builders Library states that retrying APIs with side effects is unsafe without idempotency guarantees. Idempotency ensures that retrying the same request has the same effect once, preventing unintended duplication. Amazon limits retries to a single stack layer to avoid retry storms.

You cannot remove retries or make the network perfectly reliable. Instead, design operations to be replay-safe. Idempotency keys make repeated requests reproducible with single side effect execution.

Clarifying 'idempotency' beyond HTTP semantics

The term idempotency has multiple meanings. By HTTP spec, methods like GET, PUT, and DELETE are idempotent — repeating them yields the same server state. Business idempotency means repeating a request produces one charge or order. Idempotency keys provide POST requests this property by uniquely identifying user intent.

For example, UPDATE balance SET amount = amount - 50 called twice subtracts twice — not idempotent. But UPDATE balance SET amount = 950 repeated sets the same value — idempotent. Deduplicating on request body alone is unsafe: two legitimate $5 coffee purchases a minute apart look identical but are separate.

Common naive implementation and race conditions

existing = db.query("SELECT * FROM idempotency_keys WHERE key = %s", key)
if existing:
    return existing.response
result = charge_card(...)
db.execute("INSERT INTO idempotency_keys ...", key, result)
return result

This pattern queries for the key before processing. Concurrent requests both find no existing key and both charge. The read-then-write race causes double charges. Also, if the server crashes after charging but before recording the key, retries charge again since no record exists. This approach is broken under concurrency and crashes.

Atomic claiming of idempotency keys with example schema

CREATE TABLE idempotency_keys (
  key             TEXT PRIMARY KEY,
  user_id         BIGINT NOT NULL,
  request_fingerprint TEXT NOT NULL,
  state           TEXT NOT NULL,   -- 'in_progress' | 'completed'
  response_code   INT,
  response_body   JSONB,
  locked_at       TIMESTAMPTZ,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

Instead of checking first, atomically claim the idempotency key using a single INSERT ... ON CONFLICT DO NOTHING. If you insert the row, you own processing exclusively. If it conflicts, another request owns or completed the operation. This removes race conditions and acts as a distributed lock at the database level.

Scope the idempotency key to the caller by making (user_id, key) the composite primary key, preventing collisions and privacy leaks between different users or tenants. This isolates each users requests and stored responses.

Fingerprint the request by hashing its body and parameters. When a request with the same key has a different fingerprint, reject it with a 422 error. This detects client bugs where the idempotency key is reused incorrectly with differing details.

Store the full response status code and body for every request, including errors. Subsequent retries receive the stored response to maintain idempotency semantics. Returning stored 500 errors is correct and expected, avoiding hidden failures.

Dealing with foreign state and multi-phase requests

When your endpoint calls external services such as Stripe, you cannot wrap the external call and your database update in a single transaction. This leads to potential inconsistencies requiring your design to support partial failures and recovery.

Brandur Leachs approach splits the request into atomic phases with recovery points persisted between them. Insert the idempotency key marked in_progress before the external call. After charging, update the record to completed. If interrupted, requests resume from the most recent recovery point instead of restarting from scratch.

Never commit order state locally before external charge succeeds. Pass your idempotency key to the payment providers own idempotency mechanism to prevent duplicates downstream. Without passing your key along, the double charge problem merely shifts to the external service.

For charges that exceed HTTP timeouts, return HTTP 202 Accepted immediately and perform the charge asynchronously in a background worker. The worker applies the same idempotency logic to retry safely without risking duplicates.

Six common pitfalls in implementing idempotency keys correctly

Generating idempotency keys inside retry loops defeats deduplication. The key must be generated once when the users intent forms—ideally client-side before the initial request. This ensures all retries share the same key and deduplication works properly.

Placing rate limiting or validation before idempotency storage means rejected requests leave no record. Clients retry rejected requests because no stored response exists, making them retryable and risking duplicated processing or double charges.

Idempotency keys and stored responses expire after a retention window. Stripes first API retains keys for 24 hours; the second API extends to 30 days. Retries after expiry are treated as new requests, risking duplicates. Your retention period must exceed the longest realistic retry scenario, including offline retries from mobile clients reconnecting later.

If the process crashes while the key is in_progress, the lock remains indefinitely. Use a locked_at timestamp with a timeout and a cleanup job to detect and recover stuck entries. Recovery should reconcile external state before deleting to avoid inconsistent records.

Reusing the same idempotency key across different endpoints, like /refunds and /payments, causes conflicts. Scope keys to endpoints or include the request route in the fingerprint to prevent collisions and incorrect result retrievals.

Deleting idempotency keys one by one can fall behind under load. Batch deletions with a LIMIT clause maintain cleanup performance and prevent buildup of expired keys that degrade storage and query efficiency.

Testing concurrency and failure scenarios to prove correctness

with ThreadPoolExecutor(max_workers=50) as ex:
    responses = list(ex.map(lambda _: post(payload, key=K), range(50)))
assert charge_count(K) == 1

Inject faults during testing by crashing the process between phases. After crash, retry the request and assert exactly one charge. This active chaos testing verifies your idempotency logic works correctly under concurrency and partial failures.

Close

Idempotency keys are widely used for safe retries but remain an active Internet Draft, not an RFC. Implementations vary on expiry, error handling, and concurrency semantics. Read provider documentation carefully to avoid double charges or lost payments.

from fastapi import FastAPI, Request, HTTPException
from sqlalchemy import text
from sqlalchemy.orm import Session
import hashlib
import json

app = FastAPI()

@app.post("/payments")
async def payment_endpoint(request: Request, db: Session):
    data = await request.json()
    user_id = data["user_id"]
    key = request.headers.get("Idempotency-Key")
    fingerprint = hashlib.sha256(json.dumps(data, sort_keys=True).encode()).hexdigest()

    # Atomically claim the key
    insert_stmt = text("""
    INSERT INTO idempotency_keys (key, user_id, request_fingerprint, state, created_at)
    VALUES (:key, :user_id, :fingerprint, 'in_progress', now())
    ON CONFLICT (key, user_id) DO NOTHING
    """)
    result = db.execute(insert_stmt, {"key": key, "user_id": user_id, "fingerprint": fingerprint})
    db.commit()

    if result.rowcount == 0:  # Conflict, key already exists
        record = db.execute(
            text("SELECT state, response_code, response_body, request_fingerprint FROM idempotency_keys WHERE key=:key AND user_id=:user_id"),
            {"key": key, "user_id": user_id}
        ).fetchone()
        if not record:
            raise HTTPException(status_code=500, detail="Idempotency record not found after conflict")
        if record.request_fingerprint != fingerprint:
            raise HTTPException(status_code=422, detail="Idempotency key reused with different request body")
        if record.state == "completed":
            return json.loads(record.response_body)
        elif record.state == "in_progress":
            raise HTTPException(status_code=409, detail="Request currently in progress")

    # Process the payment
    response = charge_card(data)  # Implement actual charge logic

    # Update record to completed and store response
    update_stmt = text("""
    UPDATE idempotency_keys
    SET state = 'completed', response_code = :code, response_body = :body
    WHERE key = :key AND user_id = :user_id
    """)
    db.execute(update_stmt, {"code": response.status_code, "body": json.dumps(response.body), "key": key, "user_id": user_id})
    db.commit()

    return response.body

# Placeholder for payment charging logic
class Response:
    def __init__(self, status_code, body):
        self.status_code = status_code
        self.body = body

def charge_card(data):
    # Simulated payment processor call
    return Response(200, {"status": "charged", "amount": data.get("amount")})