Skip to content

Latest commit

 

History

History
209 lines (159 loc) · 10.3 KB

README.md

File metadata and controls

209 lines (159 loc) · 10.3 KB

notarial-worker

Handles emails via GOV.UK Notify or AWS SES.

The queues are managed by pg-boss

The current naming scheme is NOTIFY_* and SES_* to handle GOV.UK Notify and AWS SES respectively. The suffixes _PROCESS and _SEND are currently used.

  • _PROCESS is used to store the parameters, and allow retrying of the message. Minimal business logic is used for these workers, they simply send the request to notarial-api/forms/emails* where the actual parsing of data and business logic is held
  • _SEND is used to send the email

The general flow is

  1. POST to /forms (handled by forms-worker)
  2. creates two messages (.sendToProcessQueue) which just takes the data and adds it to *_PROCESS queues
  3. On successful message creation, respond with success (so POST to /forms doesn’t need to be retried)
  4. Worker picks up the (sendToProcessQueue) messages and POSTs to /forms/emails/ses and /forms/emails/notify
  5. Endpoints parse the data and throw appropriately
  6. On success, sticks the compiled notify/ email body on the *_SEND queues
  7. Worker sends the emails

This allows all stages to be retried individually. If errors are thrown, or there are erroneous responses (4xx or 5xx errors), these will be stored in the database, in the output column.

A simplified diagram of this flow can be found in simplified_flow.svg.

Generally all jobs will be added to the queue with the data required for the operation, as well as metadata to allow for easy tracking of the job. All jobs will have a metadata property, which will contain the reference number. This will match with their GOV.UK Pay reference number (not to be confused wit their payment ID).

notifyProcessHandler

notifyProcessHandler

When a message on the NOTIFY_PROCESS queue is detected this worker will send a request to notarial-api/forms/emails/notify. The source of this event is notarial-api, after a user has submitted a form (POST /forms) SubmitService.submitForm

    select * from pgboss.job where name = 'NOTIFY_PROCESS';

The data stored in this job will be the user's answers, and the metadata of the form submission. For example:

{ 
  "answers": {
    "firstName": "test",
    "middleName": null, 
    "dateOfBirth": "2000-01-01"
    /** etc **/
  },
  "metadata": {
    "type": "affirmation", 
    "payment": { 
      "payId": "usfetplth9aqfm0ft598eigpkm",
      "state": { 
        "status": "created",
        "finished": false
      },
      "reference": "B-6FYZIU1M"
    },
    "reference": "B-6FYZIU1M"
  }
}

notifySendHandler

notifySendHandler

When a message on the "NOTIFY_SEND" queue is detected, this worker sends a GOV.UK notify request. The source of this event is notarial-api/forms/emails/notify, which processes the user's data.

    select * from pgboss.job where name = 'NOTIFY_SEND';

The data stored in this job will be GOV.UK Notify API options, which include personalisations, reference, template ID and email address.

For example:

 {
  "options": {
    "reference": "PF3N8EGP9L",
    "personalisation": {
      "post": "British Embassy Rome",
      "country": "Italy",
      "firstName": "test",
      /** etc **/
    },
    "metadata": {
      "reference": "PF3N8EGP9L"
    }, 
    "template": "ABC-DEF",
    "emailAddress": "pye@cautionyourblast.com"
  }
}

notifyFailureHandler

notifyFailureHandler

After messages have been sent to GOV.UK Notify, it responds with a success or error depending on if the request to their API was valid.

GOV.UK Notify will attempt to send the message, and will retry it if there was a failure. Failures may occur when the user has not provided us with the correct email address. A separate GOV.UK Notify request must be made to check for these types of failures.

NOTIFY_FAILURE_CHECK is scheduled to check for failed emails every day, and will store a digest of failed email sends.

The handler will make calls to Notify to find any messages with one of three statuses:

  • temporary-failure
  • permanent-failure
  • technical-failure

If any failed messages are found, this raises an alert which can then be caught and relayed to the relevant post.

sesProcessHandler

sesProcessHandler

When a message on the SES_PROCESS queue is detected this worker will send a request to notarial-api/forms/emails/ses. The source of this event is notarial-api, after a user has submitted a form (POST /forms) SubmitService.submitForm

    select * from pgboss.job where name = 'SES_PROCESS';

sesSendHandler

sesSendHandler

When a message on the "NOTIFY_SEND" queue is detected, this worker sends a GOV.UK notify request. The source of this event is notarial-api/forms/emails/ses, which processes the user's data.

    select * from pgboss.job where name = 'SES_SEND';

The data stored in this job will be the email body and attachments.

{
  "body": "\n<p>Dear British Embassy Rome, Use them to create a new case in Casebook and prepare the affirmation document.\n</p>...",
  "subject": "cni application, British Embassy Rome – PF3N8EGP9L",
  "metadata": {
    "reference": "PF3N8EGP9L"
  },
  "reference": "PF3N8EGP9L",
  "onComplete": {
    "job": {
      "options": { // This will match the NOTIFY_SEND job
        "personalisation": {},
        "template": "ABCDEF_EG",
        "reference": "PF3N8EGP9L",
        "emailAddress": "pye@cautionyourblast.com"
      },
      "queue": "NOTIFY_SEND"
    },
    "attachments": [
      {
        "key": "ukPassportFile",
        "type": "file",
        "title": "UK passport",
        "answer": "http://documentupload:9000/v1/files/511ffde6-ea44-4d72-967c-a5581f73fb8e.png",
        "category": "applicantDetails"
      }
    ]
  }
}

The attachments will be fetched, then added to the email body in memory (i.e. not updated in the database) before sending to SES.

Troubleshooting

When tasks fail, the error emitted will automatically be added to the jobs, and the error is logged. If the logs are incomplete, further logging may be found on the database in the output column.

See TROUBLESHOOTING.md for more information.

Environment variables

Env var Description default
QUEUE_URL Connection string of the db postgres://user:root@localhost:5432/notarial
ARCHIVE_FAILED_AFTER_DAYS In days, how long to keep failed jobs in the table pgboss.jobs, before sending it to pgboss.archive 30
DELETE_ARCHIVED_AFTER_DAYS In days, how long to keep any jobs in pgboss.archive before deleting 7
MONITOR_STATE_INTERVAL_SECONDS In seconds, how often to log the statuses of each queue 10
QUEUE_SCHEMA The schema name for pgboss to use. If it does not exist, pgboss will create the schema and related tables in this schema. pgboss
QUEUE_DRAIN_SCHEMA Moves inflight (created) messages from this schema to QUEUE_SCHEMA
NOTIFY_API_KEY Notify API key to send emails from
NOTIFY_FAILURE_CHECK_SCHEDULE Cron string determining the schedule for checking for failed notify sends 0 9 * * *
SES_SENDER_NAME The name to display when sending an email via SES Getting Married Abroad Service
SENDER_EMAIL_ADDRESS Where the email should be sent from. There must be an SES domain identity matching this email address pye@cautionyourblast.com
SUBMISSION_ADDRESS Where to send the emails to pye@cautionyourblast.com
NOTARIAL_API_CREATE_SES_EMAIL_URL URL on the notarial-api where SES emails can be created http://localhost:9000/forms/emails/ses
NOTARIAL_API_CREATE_NOTIFY_EMAIL_URL URL on the notarial-api where Notify emails can be created http://localhost:9000/forms/emails/notify
FILES_ALLOWED_ORIGINS Allowed origins where files can be downloaded from ["http://localhost:9000"]