Refactor Harvest Reporter to be a property of HarvestSource

[btylerburton]

User Story

Currently in the reporter we document:

  job_status = {
      "status": "complete",
      "date_finished": datetime.now(timezone.utc),
      "records_added": results["action"]["create"],
      "records_updated": results["action"]["update"],
      "records_deleted": results["action"]["delete"],
      "records_ignored": results["action"][None],
      "records_errored": results["status"]["error"],
      "records_validated": results["validity"][True],
  }

In order to better track the results of a harvest job, datagovteam wants to make the reporter a property of the harvest_source and have it count the results of the operation as they happen vs. creating a summary once the job is complete.

This ensures:

1. We accurately count deletes, as they are currently misrepresented since the record is already deleted from the code at the time of reporting
2. We can create a report in the case of a critical exception which fails the job

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN I am running a harvest
  WHEN a record is processed then we will count the change in its state as it happens
  THEN when we arrive at the time to report, we report the results that are available rather than doing math in process.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Create a new reporter object with the documented attributes above
• Make it a property of harvest_source
• Update the attributes as the harvest source is processed

Added:

• Update harvest job as records are processed
• Add percentage complete to jobs display page
• Update job with stats even in case of critical error