Idempotency Keys

Idempotency is a crucial design pattern for building robust systems, particularly in areas such as payments, message queues, order processing, error recovery, and fault tolerance. It ensures system correctness in the event of failures.

PUT: An Idempotent Operation

The HTTP specification defines PUT as an idempotent operation, meaning multiple retries of the same request will not cause additional side effects or changes. This idempotency provides the benefit of safe retries, allowing clients to repeat the same operation multiple times with consistent outcomes.

For example, when updating a user's name:

PUT /users {"name": "John Doe"}

Executing this operation multiple times will always result in the database row containing only "John Doe" as the name. PUT can also be used for upsert operations, creating a resource if it doesn't exist or updating it if it does.

Note: The actual implementation of idempotency should be defined in server-side code.

Idempotency in Payment Systems

Payment systems are a prime example where idempotency is crucial. Several issues can arise during payment processing:

• Mid-transaction failures
• Request or response timeouts
• Partial execution
• Network failures

To ensure reliability, we have two main options:

1. Allow the request to fail and have the user retry the payment
2. Implement server-side retries with exponential backoff

The first option provides a poor user experience, while the second can lead to duplicate payments. To avoid duplicates and ensure "exactly once semantics," we use idempotent keys.

Idempotent Keys

Idempotent keys are unique identifiers associated with each operation, preventing side effects when retrying the same operation. They are particularly useful in distributed systems where HTTP specifications alone may not suffice.

Key Generation and Usage

Client applications generate unique identifiers for each request and cache them for safe retries. The process typically involves:

1. Storing the generated idempotency key for the original request
2. Detecting network failures or uncertain responses
3. Automatically retrying the request using the same idempotency key

Passing Idempotency Keys

API vendors often require idempotency keys to be passed in their APIs. These can be included as headers or as part of the payload.

Header Example (using Stripe SDK):

StripeClient stripe = new StripeClient("{{YOUR_API_KEY}}");
String idempotencyKey = "unique-idempotency-key";
Example result = stripe.v2().examples().create(
    ExampleCreateParams.builder()
        .setName("My example")
        .build(),
    RequestOptions.builder()
        .setIdempotencyKey(idempotencyKey)
        .build()
);

Payload Example:

{
    "customer_reference": "123456",
    "timestamp": "2024-11-12T13:14:00+00:00",
    "x-idempotency-key": "0018d00000h53dlaab_e67e9841_4437_44fc_a633_123456"
}

Key Retention

Idempotent key retention varies across systems and APIs:

• Chargebee: 30 minutes
• Adyen: Minimum 7 days after first submission
• Worldpay: 1 to 365 days (customer's choice)
• Amazon Pay: Indefinite storage
• Xero and Worldline: At least 24 hours

The optimal retention duration balances safe retry windows with efficient server-side storage.

Idempotency in Batch APIs

For batch processing or bulk payments, two approaches can ensure idempotency:

1. Use a single idempotency key for the entire batch, retrying the whole batch if issues occur.
2. Include an idempotency key for each record in the batch, allowing for partial retries and processing in any order.

The second approach offers more flexibility and efficiency in handling partial failures.

By implementing idempotency correctly, systems can significantly improve reliability and user experience, particularly in critical operations like payment processing.