Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

ci(agent): Matrix CI tests across Linux, macOS and Windows #7029

Merged
merged 15 commits into from
Mar 21, 2024

Conversation

Pwuts
Copy link
Member

@Pwuts Pwuts commented Mar 20, 2024

If we want to support certain platforms, we need to include them in our automated tests. Currently, all of our CI runs on Linux (ubuntu-latest), and so it's blind to platform specific issues with Windows or macOS.

Triggered by this comment.

Changes 🏗️

  • Matrix the AutoGPT Python CI's test job across Ubuntu, macOS and Windows

    • Set up MinIO in a step rather than specifying it under jobs[test].services, because services are only supported on Linux runners

    • Add Windows version of step to install Poetry

    • Add macOS compatibility patches to 'Install Poetry (Unix)' and setup_git_auth steps

    Caveats:

    • No Docker on macOS or Windows

      • Windows comes with Docker but only supports running Windows containers, while we're mainly interested in using Linux containers for code execution and/or running auxiliary services.
      • The macOS runner doesn't come with Docker. Setting it up is possible but takes ~3-4 minutes, and the performance of the Colima engine is poor: a docker pull that takes 2 seconds on Linux takes 45 seconds on macOS.
    • No S3 service available on Windows
      It seems that running a background process isn't possible on Windows, and neither is running Linux-based Docker containers.

  • Add autogpt-agent and OS-specific flags to Codecov upload step

  • Improve caching of Python dependencies by changing the cache key

    • Include hash of poetry.lock instead of pyproject.toml in key
    • Remove date component from key; it was included to avoid getting stuck to old cached versions of packages when we were still using requirements.txt. With poetry.lock that is no longer a concern.
  • Fix skip check in test_s3_file_storage.py

PR Quality Scorecard ✨

  • Have you used the PR description template?   +2 pts
  • Is your pull request atomic, focusing on a single change?   +5 pts
  • Have you linked the GitHub issue(s) that this PR addresses?   +5 pts
  • Have you documented your changes clearly and comprehensively?   +5 pts
  • Have you changed or added a feature?   -4 pts
    • Have you added/updated corresponding documentation?   +4 pts
    • Have you added/updated corresponding integration tests?   +5 pts
  • Have you changed the behavior of AutoGPT?   -5 pts
    • Have you also run agbenchmark to verify that these changes do not regress performance?   +10 pts

@Pwuts Pwuts requested a review from a team as a code owner March 20, 2024 09:37
Copy link

codecov bot commented Mar 20, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 44.91%. Comparing base (bca5031) to head (d7b59bf).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7029      +/-   ##
==========================================
- Coverage   45.33%   44.91%   -0.43%     
==========================================
  Files         139      139              
  Lines        6511     6514       +3     
  Branches      912      915       +3     
==========================================
- Hits         2952     2926      -26     
- Misses       3408     3439      +31     
+ Partials      151      149       -2     
Flag Coverage Δ
Windows 43.67% <ø> (?)
autogpt-agent 44.91% <ø> (?)
macOS 44.88% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This comment was marked as outdated.

@Pwuts Pwuts force-pushed the ci-test/cross-platform-autogpt-ci branch from 542a1ff to 08a2aa2 Compare March 20, 2024 10:19

This comment was marked as outdated.

This comment was marked as outdated.

Copy link

netlify bot commented Mar 20, 2024

Deploy Preview for auto-gpt-docs canceled.

Name Link
🔨 Latest commit d7b59bf
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs/deploys/65fc6d658c9a520008b62ddf

@Pwuts Pwuts force-pushed the ci-test/cross-platform-autogpt-ci branch from 08a2aa2 to 374d9a0 Compare March 20, 2024 10:32
@Pwuts Pwuts requested a review from a team as a code owner March 20, 2024 11:21
@Pwuts

This comment was marked as resolved.

@Pwuts Pwuts force-pushed the ci-test/cross-platform-autogpt-ci branch from 7bb8e6b to 57bf4e2 Compare March 20, 2024 15:55
@Pwuts Pwuts force-pushed the ci-test/cross-platform-autogpt-ci branch from 59ea2c5 to e3155bd Compare March 20, 2024 16:45
@Pwuts Pwuts force-pushed the ci-test/cross-platform-autogpt-ci branch from a270657 to 8921cfc Compare March 20, 2024 16:59
@Pwuts Pwuts mentioned this pull request Mar 20, 2024
9 tasks
@Pwuts Pwuts removed the request for review from a team March 20, 2024 17:25
@Pwuts Pwuts added the ci label Mar 20, 2024
@Pwuts Pwuts force-pushed the ci-test/cross-platform-autogpt-ci branch 20 times, most recently from 13ebfc4 to a7d397c Compare March 21, 2024 10:34
@Pwuts Pwuts force-pushed the ci-test/cross-platform-autogpt-ci branch from a7d397c to 84fcc58 Compare March 21, 2024 10:41
@Pwuts Pwuts changed the title ci(agent): Add Windows and macOS test jobs ci(agent): Matrix CI tests across Linux, macOS and Windows Mar 21, 2024
@Pwuts Pwuts merged commit dde0c70 into master Mar 21, 2024
20 of 21 checks passed
@Pwuts Pwuts deleted the ci-test/cross-platform-autogpt-ci branch March 21, 2024 20:15
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants