Changes between Version 8 and Version 9 of BluePrint/Messaging/Outbound

Timestamp:

09/21/12 12:37:33 (13 years ago)

Author:

Pat Tressel

Comment:

--

BluePrint/Messaging/Outbound

@@ -4 +4 @@
 == Outbound Queue Performance ==
 === Issues ===
-We would like to ship out new messages in a timely manner, and also handle retrying failed messages, but avoid unnecessary overhead, and without doing time-consuming sends in request context where they will slow down response to the user.
+We would like to send out new messages in a timely manner, without doing time-consuming sends in request context where they will slow down response to the user, but also handle retrying failed messages and avoid unnecessary overhead.
 
 Right now, inserting a message in the queue (send_by_pe_id) is unconditionally followed by kicking off the outbound sending task (which runs asynchronously, not in request context). That isn't just making an attempt to send the single new message, but rather processes the entire queue. Messages that are in the queue in order to be retried due to an earlier send failure will be retried along with the one new message.
@@ -32 +32 @@
 Since email will likely be a significant part of messaging, we might look at typical outbound email processing and retry handling. Email send failures fall into two categories:
 * Permanent failures, for which no retry will succeed. These include "no such user". In traditional email processing (read, sendmail), these failures get bounced right away. It doesn't serve any purpose to keep processing permanent failures as retries. Some we might deal with automatically, but if not, we do want to keep them around for the site admin to deal with. We could move these to a holding queue.
-* Temporary failures, for which a retry may be succeed at some point. These include no response from the host, and "user mailbox full". Traditional email processing will eventually bounce even temporary failures. (If we want different retry rates for messages that have repeatedly failed (or, say, for different urgency levels), we could use separate queues with separate periodic tasks. Messages could be demoted into less frequent queues based on number of retry failures.)
+* Temporary failures, for which a retry may be succeed at some point. These include mailer misconfiguration or outage, no response from destination host, "user mailbox full"... Traditional email processing will eventually bounce even temporary failures. (If we want different retry rates for messages that have repeatedly failed (or, say, for different urgency levels), we could use separate queues with separate periodic tasks. Messages could be demoted into less frequent queues based on number of retry failures.)
+
+=== Proposal ===
+
+Initial part:
+
+* Split outbound processing into three queues:
+  * New messages
+  * Retries
+  * Failures (immediately permanent fails and max retries exceeded)
+
+* Let site choose whether to start a new message queue processing run when a new message is posted or use a periodic task for new messages, and periodic task frequencies.
+
+* Include advice in the user guidelines for async vs. periodic new message processing, and for setting periodic task frequencies, based on the site's message behavior.
+
+* Add logging of errors. Detect obvious misconfiguration and object strenuously.
+
+Later:
+
+* Add automated failure handling, and / or a way for the admin to review and deal with the failures.
+
+* Attempt to detect transport issues that the site admin might need to deal with.
 
 == Email Error Handling ==