Don't ignore your dead-letter queue

Every Azure Service Bus queue and topic subscription comes with a hidden sub-queue: the dead-letter queue (DLQ). Messages end up there when they can't be processed, and they stay there indefinitely. Nothing alerts you by default. In plenty of systems the first sign of trouble is a customer asking why their order never shipped, and the answer is sitting in a DLQ with thousands of other messages.

#How messages get there

• Too many delivery attempts. A message that fails processing is retried until it reaches the queue's MaxDeliveryCount (10 by default), then it's dead-lettered with the reason MaxDeliveryCountExceeded.
• Expired messages, if dead-lettering on expiration is enabled for the queue or subscription.
• Your code puts it there deliberately, for a message it knows it can never process:

catch (ValidationException ex)
{
    await args.DeadLetterMessageAsync(args.Message,
        deadLetterReason: "ValidationFailed",
        deadLetterErrorDescription: ex.Message);
    return;
}

Dead-lettering a message that's invalid straight away is better than letting it fail ten times first. Retrying won't fix bad data.

#Read it

The DLQ is just another queue. Open a receiver on the DeadLetter sub-queue:

await using var receiver = client.CreateReceiver("orders",
    new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });

var messages = await receiver.PeekMessagesAsync(maxMessages: 50);
foreach (var m in messages)
{
    Console.WriteLine($"{m.MessageId}: {m.DeadLetterReason} - {m.DeadLetterErrorDescription}");
}

PeekMessagesAsync reads without removing anything, so it's safe for investigation. The Service Bus Explorer in the Azure portal can do the same without writing code.

#Replay it

Once the cause is fixed, send the messages back to the main queue and remove them from the DLQ:

await using var receiver = client.CreateReceiver("orders",
    new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
await using var sender = client.CreateSender("orders");

foreach (var message in await receiver.ReceiveMessagesAsync(maxMessages: 50))
{
    await sender.SendMessageAsync(new ServiceBusMessage(message));   // copies body and properties
    await receiver.CompleteMessageAsync(message);
}

Replaying is only safe if your handlers cope with duplicates, because some of these messages may have been partly processed before they failed.

#Watch it

Set up an Azure Monitor alert on the dead-lettered messages count for each queue and subscription. Alert when it's above zero, or above a small threshold if a few are expected. A DLQ that only grows is a bug report nobody is reading.

#Takeaway

Treat the dead-letter queue as part of your system. Dead-letter messages you know are invalid instead of retrying them, alert when the DLQ isn't empty, and have a tested way to inspect and replay messages once the cause is fixed.