Failure Handling in Event-Driven Systems

Event-driven systems make failure handling explicit. That is a strength, but only if the team designs for it.

Producer
-> Kafka
-> Consumer
-> Database / External API

The central question is simple: what happens when the consumer cannot finish processing the event?

Transient Failures

Transient failures may succeed later. Examples include network timeouts, temporarily unavailable providers, connection resets, and rate limits.

Retries can be appropriate, but they need limits and delays. Immediate repeated retries can amplify the downstream outage.

Permanent Failures

Permanent failures will not be fixed by waiting. Invalid payloads, missing required data, unsupported event versions, and business rule violations usually need inspection or correction.

Retrying these forever creates noise and can block useful work.

Consumer Crash

A consumer can crash after receiving an event. Depending on when progress is committed, the event may be processed again.

That is why idempotency matters. Event processing should assume duplicates are possible.

Partial Processing

The most dangerous failure is partial success:

External call succeeds
-> consumer crashes
-> message is processed again

The system now has an ambiguous local view. Did the provider receive the request? Did the database record the result? Can retrying duplicate the operation?

The answer should come from operation identifiers, provider reconciliation, local transaction state, and idempotency keys, not hope.

Retries and Backoff

Retry policies should distinguish retryable from non-retryable failures. Exponential backoff with jitter is often useful conceptually because it avoids synchronized retry bursts.

Retries should be observable. Operators need to know whether failures are isolated, increasing, or stuck.

Dead-Letter Queues

A DLQ is not a trash bin. It is an operational holding area for messages that need inspection, correction, and a controlled replay strategy.

DLQ handling should include monitoring, payload visibility with sensitive-data care, ownership, runbooks, and a way to replay safely after correction.

Poison Messages

A poison message is a message that repeatedly fails. Without isolation, it can keep consuming resources and hide other failures.

Poison-message handling needs failure classification, retry limits, and alerts. If every failure is treated the same, operators lose signal.

Observability

At minimum, I want logs and metrics around event type, consumer group, processing time, retry count, failure reason, and dead-letter movement. Distributed tracing can help connect the original request to the event and worker.

Related Engineering Context

The Blockchain Marketplace is related because asynchronous blockchain work must account for external-provider failure, retries, duplicate processing, and eventual consistency.

Production Principle

Event-driven systems should make failure recoverable, visible, and safe to retry. If failure handling is not designed, the broker only moves uncertainty from one place to another.