APIs & Integrations

Building Reliable API Integrations with Webhooks, Retries & Idempotency

A production-focused guide to API integration reliability covering webhooks, idempotency, retries, queues, reconciliation, authentication, logging, rate limits, and failure recovery.

SlantFlow Back to insights ↗

APIs Are Easy. Reliable Integrations Are Not.

Sending a successful API request is the easy part of an integration.
The difficult part is making sure data still arrives correctly when networks fail, providers time out, webhook deliveries are duplicated, credentials expire, or the same event is processed more than once.

A production integration must assume failure will happen.
The architecture should be designed to recover automatically instead of depending on manual intervention every time something goes wrong.

Design Around Events, Not Screens

Good integrations are usually driven by business events:

  • An order was created
  • A payment was confirmed
  • A shipment status changed
  • Inventory was updated
  • A customer record changed
  • A call or CRM event was received

The integration layer should process those events independently from the user interface.
That keeps business synchronization predictable and easier to monitor.

Webhooks Are the First Layer

Webhooks are useful because they allow one system to notify another immediately when something changes.
Instead of polling an API every minute, the provider sends the event directly.

Provider
   |
   | webhook event
   v
Webhook Endpoint
   |
   v
Validation
   |
   v
Queue / Processor
   |
   v
Business Action

But receiving a webhook should not automatically mean executing the full business workflow inside the HTTP request.
For larger workflows, we prefer acknowledging the webhook quickly and processing the work separately.

Validate Every Webhook

Webhook endpoints are public HTTP endpoints, so incoming requests should never be trusted by default.
Depending on the provider, validation can include:

  • Signature verification
  • Shared secrets
  • Request timestamps
  • Allowed event types
  • Payload validation
  • Replay protection

A request should only reach the business layer after the integration knows it came from the expected source and has a valid structure.

Idempotency Is Essential

Most webhook providers retry delivery if they do not receive a successful response.
That means the same event may arrive more than once.

Without idempotency, duplicated delivery can create duplicated orders, shipments, payments, invoices, or CRM activity.

event_id = payload["event_id"]

if already_processed(event_id):
    return success

process_event(payload)
mark_as_processed(event_id)

The exact implementation varies, but the principle is always the same:
processing the same event multiple times should produce the same final state.

Use Stable External References

When two systems communicate, each synchronized record should have a stable reference that both sides understand.

Examples include:

  • External order ID
  • Shipment reference
  • Provider transaction ID
  • Webhook event ID
  • Customer external ID

These references make duplicate detection, reconciliation, and debugging much easier.

Retries Need Rules

Retries should not mean repeatedly sending every failed request forever.
Different errors need different behavior.

Retryable errors often include:

  • Connection timeouts
  • Temporary DNS failures
  • HTTP 429 rate limits
  • HTTP 502 / 503 provider errors
  • Temporary service unavailability

Non-retryable failures may include:

  • Invalid credentials
  • Invalid payloads
  • Missing required mappings
  • Business validation errors
  • Unsupported operations

Exponential Backoff

Repeated retries can make an outage worse.
Instead, we increase the delay between attempts.

Attempt 1  → immediately
Attempt 2  → 30 seconds
Attempt 3  → 2 minutes
Attempt 4  → 10 minutes
Attempt 5  → 30 minutes
Final      → manual review / dead-letter queue

The exact schedule depends on how time-sensitive the integration is.

Queues Decouple Systems

If System A depends directly on System B responding immediately, both systems become tightly coupled.
A queue introduces a safer boundary.

Order Created
     |
     v
Integration Queue
     |
     +--> Attempt ERP Sync
     |
     +--> Retry if temporary failure
     |
     +--> Log permanent failure

Queues are especially valuable for stock synchronization, bulk product updates, notifications, and provider APIs with strict rate limits.

Timeouts Must Be Explicit

An API request should never wait indefinitely.
Every integration should define sensible connection and request timeouts.

A timeout does not necessarily mean the remote operation failed.
The provider may have completed the operation but the response never reached us.
This is another reason idempotency and external references are important.

Reconciliation Catches What Events Miss

Even a good webhook implementation can miss events.
Providers may have temporary problems, DNS may fail, or application deployments may briefly interrupt processing.

We therefore combine event-driven integration with periodic reconciliation.

Webhooks:
Real-time synchronization

Reconciliation:
Periodic consistency check

For example, a scheduled task can check recent shipments and compare their current provider status against the local database.

Logging Should Be Structured

Logs are only useful when they answer operational questions quickly.
A production integration should record meaningful context rather than dumping random payloads.

Useful fields include:

  • Integration name
  • Operation
  • External reference
  • Local record ID
  • Provider request ID
  • HTTP status
  • Attempt number
  • Duration
  • Error category
  • Timestamp

Protect Sensitive Data in Logs

Logging complete API payloads can accidentally expose credentials, tokens, personal data, payment information, or customer details.

Sensitive values should be redacted before they are written to application logs.

{
    "authorization": "[REDACTED]",
    "customer_email": "o***@example.com",
    "transaction_id": "TX-92812"
}

Monitor Business Outcomes

Technical uptime does not guarantee that an integration is actually working.
The server may return HTTP 200 while orders have stopped synchronizing.

We prefer monitoring signals such as:

  • Last successful synchronization
  • Failed jobs in the last hour
  • Queue backlog
  • Orders waiting for ERP creation
  • Webhook failure rate
  • Provider response time
  • Authentication failures

Rate Limits Need First-Class Handling

Third-party APIs often limit how many requests can be sent within a time period.
Ignoring those limits leads to unnecessary failures.

The integration should detect rate-limit responses, respect retry headers when available, and control concurrency when processing large queues.

Authentication Will Expire

OAuth access tokens and provider sessions eventually expire.
Token refresh should be part of the integration architecture, not something discovered after production stops synchronizing.

We normally keep authentication logic isolated from business logic so every API operation can reuse the same token lifecycle safely.

Failure States Should Be Visible

A failed integration should not disappear silently into a log file.
Operations teams need to know which business records require attention.

Depending on the system, we may expose:

  • Pending synchronization
  • Last synchronization time
  • Retry count
  • Integration error
  • Manual retry action

The Integration Layer Is a System of Its Own

The biggest architecture mistake is treating integration as a few HTTP calls added inside unrelated application code.

A reliable integration has its own concerns:
authentication, mapping, validation, retries, idempotency, queues, reconciliation, monitoring, and error recovery.

Once these concerns are treated as part of the system design, integrations become much easier to operate and scale.

Our Approach

We design integrations with failure in mind from the beginning.
A production integration should be able to recover from temporary outages, reject invalid data, prevent duplicates, and show operators exactly what happened when something fails.

If you need to connect an ERP, CRM, eCommerce platform, logistics provider, payment service, or another business system,
view our API & systems integration services
or review selected integration case studies.

HAVE A PROJECT?

Need something like this built?

Tell us about your workflow, system, integration, or production challenge.