You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The SDGym benchmark script currently allows users to run a quality report for each dataset/synthesizer pair. But it does not allow them to run the diagnostic report.
The diagnostic report (as implemented in SDMetrics) checks for overall data validity between the real and synthetic data. For the health of a synthesizer, it is important to verify that the synthetic data it produces always has a score of 1.0.
Expected behavior
Add a parameter called compute_diagnostic_score to the benchmarking script.
(default) True: Run a diagnostic report for each synthesizer/dataset pair being evaluated and report the score. The results should appear in as a new column called Diagnostic Score. This column should appear before the Quality Score column.
False: Do not run a diagnostic report
Additional context
When run, the time it takes to compute the diagnostic report should be included in the overall Evaluate Time.
The text was updated successfully, but these errors were encountered:
Problem Description
The SDGym benchmark script currently allows users to run a quality report for each dataset/synthesizer pair. But it does not allow them to run the diagnostic report.
The diagnostic report (as implemented in SDMetrics) checks for overall data validity between the real and synthetic data. For the health of a synthesizer, it is important to verify that the synthetic data it produces always has a score of 1.0.
Expected behavior
Add a parameter called
compute_diagnostic_score
to the benchmarking script.True
: Run a diagnostic report for each synthesizer/dataset pair being evaluated and report the score. The results should appear in as a new column calledDiagnostic Score
. This column should appear before theQuality Score
column.False
: Do not run a diagnostic reportAdditional context
When run, the time it takes to compute the diagnostic report should be included in the overall
Evaluate Time
.The text was updated successfully, but these errors were encountered: