Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Make path joining robust to OS filesystems and remote URIs throughout code base #475

Open
mcember opened this issue Jan 29, 2025 · 0 comments

Comments

@mcember
Copy link
Collaborator

mcember commented Jan 29, 2025

Currently, the code base uses a mix of os.path.join (mostly in unit tests) and posixpath.join

Os path join uses the OS defined delimiter, which isn't going to work for e.g. an S3 URI on windows. posixpath join uses the polix delimter which will work for non-filesystem URIs like S3 but not work for windows filesystems.

We have not detected these issues because our developers are not using windows, and the tests running with os.path.join are for local filesystem and not remote filesystem

Acceptance criteria

  • Make custom path join utility which is aware of the scheme of a URI to understand if it is local filesystem or some other filesystem like s3:// or gs://
  • Use this path join utility throughout code base
  • Nice to have - ensure unit and integration tests run on Windows and S3
@mcember mcember added this to the Misc. Rivulet Improvements milestone Jan 29, 2025
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant