Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Benchmarking Workflow #17

Open
4 tasks
pjattke opened this issue Sep 14, 2020 · 0 comments
Open
4 tasks

Benchmarking Workflow #17

pjattke opened this issue Sep 14, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@pjattke
Copy link
Collaborator

pjattke commented Sep 14, 2020

Some aspects for improving the current SoK benchmarking setup:

  • Use for each tool a dedicated docker-entrypoint.sh script
    • Missing for SEALion
  • Implement support for running same program with different configurations (e.g., varying input size)
    • Goal should be to avoid code duplication by only having code once in repository
  • Run each program (not tool!) in a separate container to speed up benchmark execution time
    • Maybe it makes sense to somehow dynamically generate the docker-entrypoint.sh scripts based on a template (one per program) instead of generating a static individual script per {tool, program} configuration.
    • As this adds complexity to the Github Action: maybe it makes more sense to write a python script that is then executed in the Github Actions runner (instead of writing code directly in action just run the script).
  • Use AWS CLI to send SSM command to VMs
@pjattke pjattke added the enhancement New feature or request label Sep 14, 2020
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant