Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Allow the ability to compute diagnostic score in a benchmarking run #311

Closed
npatki opened this issue Jun 10, 2024 · 0 comments · Fixed by #317
Closed

Allow the ability to compute diagnostic score in a benchmarking run #311

npatki opened this issue Jun 10, 2024 · 0 comments · Fixed by #317
Assignees
Labels
feature request Request for a new feature
Milestone

Comments

@npatki
Copy link

npatki commented Jun 10, 2024

Problem Description

The SDGym benchmark script currently allows users to run a quality report for each dataset/synthesizer pair. But it does not allow them to run the diagnostic report.

The diagnostic report (as implemented in SDMetrics) checks for overall data validity between the real and synthetic data. For the health of a synthesizer, it is important to verify that the synthetic data it produces always has a score of 1.0.

Expected behavior

Add a parameter called compute_diagnostic_score to the benchmarking script.

  • (default) True: Run a diagnostic report for each synthesizer/dataset pair being evaluated and report the score. The results should appear in as a new column called Diagnostic Score. This column should appear before the Quality Score column.
  • False: Do not run a diagnostic report

Additional context

When run, the time it takes to compute the diagnostic report should be included in the overall Evaluate Time.

@npatki npatki added the feature request Request for a new feature label Jun 10, 2024
@lajohn4747 lajohn4747 self-assigned this Jun 28, 2024
@lajohn4747 lajohn4747 added this to the 0.8.0 milestone Jun 28, 2024
@amontanez24 amontanez24 modified the milestones: 0.8.0, 0.9.0 Aug 6, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
feature request Request for a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants