Skip to main content

Notification Troubleshooting

Summary​

Every scenario below is derived from a verified code path. The audit trail is the primary diagnostic tool because it records negative outcomes explicitly.

Audience​

DevOps engineers, support engineers, developers, QA and architects.

Symptoms​

The scenarios covered are: a notification that never arrived, a delivery that failed, a service consuming nothing, a template that rendered wrongly or not at all, a health endpoint that misleads, a service that will not start, and configuration that appears to have no effect.

Cause​

Diagnosis follows one decision path, driven by what the audit trail records.

Missing notifications​

Query the audit trail by event identifier. The recorded outcome names the cause directly. Four outcomes explain a missing notification without any failure having occurred: duplicate, stale, no recipient and suppressed.

Two subtleties are worth knowing. A no-recipient outcome most often means the payload did not deserialize into the expected shape or its key identifier was empty — which for the 32 events using service-local payload records is the signature of publisher contract drift. And a channel with no resolvable address for the recipient is skipped and audited as suppressed rather than failing.

If the event has no ledger row at all, it was never consumed.

Delivery failures​

The message carries a status, an attempt count and the last error. The attempts records carry the per-attempt provider, outcome, error text and duration.

Verified causes visible here: no sender registered for the channel; a provider exception; or the forced-failure flag having been set on a test event. Note that a message failing all attempts is logged at warning, not error.

Retry behavior appearing not to work​

Retry is bounded and in-process. All attempts happen within a single delivery, so a message that failed is not retried again later — there is no scheduled re-drive. Re-driving requires the manual retry action, which is synchronous despite the console's toast wording.

The manual retry also overwrites the attempt counter rather than accumulating it, so a retried message shows the most recent run's attempt count, not a lifetime total. The attempts table retains full history.

Broker connectivity​

The consumer logs a warning naming the host and port and retries on a fixed interval, indefinitely. Symptoms: no new notifications, the connect warning repeating in logs, and — critically — a healthy health endpoint throughout.

Absence of the bound-to-queue log line since the last restart means the consumer never connected. Its presence means it did connect at that moment; connection loss afterwards is signalled only by the warning recurring.

Template issues​

Resolution prefers the highest active version for the tenant, key and channel, then falls back to the global null-tenant row. Verified causes of a template not applying: no active row for that key and channel combination; a tenant-specific row at a lower version than expected while a higher one is active; or the row having been deactivated.

Rendering is tolerant — an unknown placeholder does not fail, so a mis-typed placeholder produces blank output rather than an error. Blank values in an otherwise correct notification indicate either a placeholder name mismatch or missing payload data.

Note that the template administration screen edits the legacy model, so changes there do not affect notifications produced by the extracted service.

Health failures​

The endpoint registers no checks, so a failure means the HTTP host itself is not responding. It never reflects a database, broker or consumer problem.

Startup failures​

Migrations run before the host serves. An unreachable database therefore fails startup. The skip flag bypasses migration — and seeding with it.

Because seeding is inside the same guard, a service started with migrations skipped against an empty database will have no default templates, and every event will record a no-template outcome.

Configuration issues​

Configuration is read from the content root, so starting the process from the wrong directory silently loads different settings. Provider selection happens once at startup, so a channel provider change requires a restart. An unrecognised email provider value falls back to console rather than failing, so a typo produces log-only email with no error.

Resolution​

Resolution steps for the recurring operational scenarios are in operational runbooks. Scenarios rooted in absent implementation — no dead-letter destination, no alerting, no preference management — cannot be resolved operationally and are recorded in UI limitations.

Validation​

After any intervention, confirm through the audit trail that a subsequent event records a processed outcome, and confirm the message status reached sent. The smoke script exercises the same paths end to end.

Classification​

Implemented — the diagnostic paths above are all source-backed.

Requires confirmation​

Escalation ownership and support-hour expectations require confirmation.

See Also​

Keywords​

  • Notification database
  • Troubleshooting
  • Draft database documentation

Source References​

  • microservices/src/notification-service/Application/NotificationDispatcher.cs
  • microservices/src/notification-service/Messaging/NotificationEventConsumer.cs
  • microservices/src/notification-service/Application/TemplateEngine.cs
  • microservices/src/notification-service/Application/EventNotificationResolver.cs
  • microservices/src/notification-service/Api/NotificationEndpoints.cs
  • microservices/src/notification-service/Application/NotificationOptions.cs
  • microservices/src/notification-service/Program.cs
  • microservices/src/notification-service/Seed/DefaultTemplateSeeder.cs

Revision Information​

  • Status: Draft
  • Last reviewed: 2026-07-21
  • Review cycle: Quarterly