Feast Ingestion not exporting success metrics to statsd

[pmjacinto]

Expected Behavior

The dataflow step WriteSuccessMetrics should be executed and produce metrics.

Current Behavior

In the previous step, the "dummy" PTransform always seems to output zero rows.
https://github.com/feast-dev/feast/blob/v0.5.0/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryWrite.java#L93-L105

This leads to the metrics transform never getting any inputs.

Is this an expected behaviour?

---

Steps to reproduce

• Create an ingestion pipeline
• Ingest a row
• Check dataflow metrics

Specifications

• Version: 0.5.0
• Platform: GKE, Dataflow
• Subsystem: Feast Ingestion

Possible Solution

[woop]

Hi @pmjacinto

Thanks for raising this. I believe the comment that you linked to does indicate the problem: BigQueryIO does not emit successful writes, meaning we can't emit metrics. WriteSuccessMetrics is an IO agnostic step that is always generated. For the time being we emit total rows processed and failed metrics, the difference of which should give you an estimation of rows written.

[woop]

@Yanson @pmjacinto for your information, the latest addition to BQ ingestion will have success metrics again #757

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.