Leave Event Retry and Failure Handling
Summary
Publication and consumer retries exist, but terminal handling differs by component and Leave has no dedicated dead-letter/replay workflow.
Audience
Developers, QA, architects, reliability owners, operations, and support.
Symptoms
- A Leave outbox row remains Pending/Failed.
- Payroll or Notification reflects a decision later than Leave.
- Employee profile changes are delayed.
- Workflow callback succeeds or fails independently of broker delivery.
Cause
Confirmed causes include publisher unavailability, a retained Failed outbox message, consumer infrastructure failure, duplicate/stale suppression, missing subscription coverage, downstream persistence failure, or Notification-owned delivery failure. Workflow HTTP failures are a separate cause category from broker publication.
Diagnosis
| Component | Confirmed behavior | Terminal/dead-letter boundary | Source |
|---|---|---|---|
| Leave relay | Failed publish marks outbox Failed with incremented retry; later polling includes Failed | No Leave terminal limit/dead-letter state | microservices/src/shared-kernel/Messaging/OutboxProcessor.cs |
| Employee projection | Processing failure negatively acknowledges and requeues | No dedicated dead-letter path confirmed | microservices/src/leave-service/Messaging/EmployeeProfileConsumer.cs |
| Payroll projection | Processing failure negatively acknowledges and requeues | No terminal limit/dead-letter path confirmed | microservices/src/payroll-service/Messaging/PayrollReadModelConsumer.cs |
| Notification | Infrastructure failures requeue; channel delivery has bounded retries and persisted failed outcome | Notification owns later retry handling | microservices/src/notification-service/Application/NotificationDispatcher.cs |
| Integration Service | Durable failed-event/dead-letter capability | Not applicable to Leave: no Leave subscription | microservices/src/integration-service/Domain/IntegrationEntities.cs |
| Workflow | HTTP start/state/callback failure | Separate from broker retry | microservices/src/leave-service/Application/WorkflowIntegration.cs |
Resolution
Use approved observability and consumer-owned recovery procedures. Confirm the event identity, source outbox state, broker publication evidence, consumer subscription, duplicate/stale decision, and downstream persistence outcome. Replay ownership and terminal retry policy require confirmation; this guide intentionally provides no production replay command.
Validation
Validate that the source outbox reached its expected state, the intended consumer is actually subscribed, the consumer recorded an applied/duplicate/stale/failed outcome, and downstream state matches that outcome. A processed Leave outbox row alone is insufficient validation.
Prevention
Exercise retry, duplicate, stale, multi-save, unavailable-consumer, and recovery scenarios in isolated test environments. Monitor outbox age/failure state and consumer-owned failed outcomes when operational instrumentation exists.
Source References
- microservices/src/shared-kernel/Messaging/OutboxProcessor.cs
- microservices/src/leave-service/Infrastructure/Persistence.cs
- microservices/src/leave-service/Messaging/OutboxRelayHostedService.cs
- microservices/src/leave-service/Messaging/EmployeeProfileConsumer.cs
- microservices/src/payroll-service/Messaging/PayrollReadModelConsumer.cs
- microservices/src/notification-service/Application/NotificationDispatcher.cs
- microservices/src/integration-service/Domain/IntegrationEntities.cs
- microservices/src/leave-service/Application/WorkflowIntegration.cs
Related Articles
See Also
Keywords
- Retry
- Failure handling
- Dead-letter boundary
Revision Information
- Status: Draft
- Last reviewed: 2026-07-15
- Review cycle: Quarterly