Distributed systems cannot always tell whether a timed-out request failed before or after the destination committed it. Retrying improves resilience only when repeated delivery has a defined, safe outcome.
Start with business identity
Choose a stable idempotency key for each operation. It may combine source account, event ID, order ID and operation type. The same business intent must reuse the same key across attempts; a genuinely new intent needs a new key.
Do not deduplicate only on payload equality. A valid price or inventory value may occur twice at different times. Do not deduplicate only on order number without tenant or channel context.
Microsoft’s idempotent-consumer pattern describes retaining processed message identifiers and atomically recording the identifier with the business effect. AWS’s guidance on safe retries likewise emphasizes client request identifiers and semantic equivalence. Apply the mechanism supported by the systems in your flow.
Make the write and deduplication decision atomic
A consumer that writes an order and then separately records “processed” can crash between those steps. Redelivery may write the order again. Use a transaction, unique business constraint or destination idempotency feature that closes that gap.
When the key already exists, return or reconcile the recorded outcome. Do not merely drop the message if the caller still needs the created order or shipment identifier.
Use the order integration UAT guide to test duplicates and failures. The order lifecycle model helps define which transition each key protects.
Retry only the right failures
Classify errors as transient, permanent or uncertain. Timeouts, some connection faults and explicit throttling may be transient. Invalid credentials, schema violations and rejected product identities require correction. An uncertain timeout needs a status check before a create operation is repeated.
Use bounded attempts, increasing delay and jitter for transient failures. Honor destination retry guidance. Microsoft Azure’s Retry pattern recommends treating failure as an exception after a predefined number of attempts and choosing delay based on the fault. Endless immediate retries can overload the dependency and bury new work.
Work through a hypothetical duplicate
An OMS posts order CH-1042 to a warehouse. The warehouse creates fulfillment F-88, but the response is lost. The OMS retries with the same idempotency key. The warehouse returns F-88 instead of creating F-89.
If the destination lacks idempotent creation, the OMS first queries by its stable external order key. If exactly one fulfillment exists, it records the missing acknowledgment. If none exists, it retries through the controlled create path. If several exist, it quarantines the conflict rather than guessing.
Reconcile beyond transport success
An HTTP success does not prove the entire business process completed. Compare source and destination populations, quantities, terminal states and acknowledgments. Reconciliation should identify missing, duplicate and conflicting records.
Store attempt number, key, request fingerprint, response, destination identifier and final disposition. Redact secrets and sensitive customer data from operational logs.
Recover from a retry storm
Stop or reduce the producer of useless retries when authorized and safe. Separate permanent errors from transient work. Check whether multiple layers are retrying the same call; an SDK, queue and job scheduler can multiply attempts.
After correcting the cause, release a small cohort. Verify idempotent outcomes and downstream side effects before draining a backlog. A queue reaching zero is not proof if duplicate orders were created along the way.
A useful next step
Select five mutating operations: order create, reservation, cancellation, shipment and refund. For each, write the idempotency key, retention window, atomicity mechanism, retryable errors and reconciliation query.
Test key reuse deliberately. Send the same intent twice with the same key and verify one effect. Then send a materially different intent with that old key and verify it is rejected or surfaced, not silently treated as equivalent. Finally, repeat after the documented retention window to understand the destination’s behavior. Keep the test evidence with the integration contract.
1Commerce can support integration and exception workflows based on provider capabilities and agreed configuration. Explore the platform or contact 1Commerce to review retry safety.