You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the code base uses a mix of os.path.join (mostly in unit tests) and posixpath.join
Os path join uses the OS defined delimiter, which isn't going to work for e.g. an S3 URI on windows. posixpath join uses the polix delimter which will work for non-filesystem URIs like S3 but not work for windows filesystems.
We have not detected these issues because our developers are not using windows, and the tests running with os.path.join are for local filesystem and not remote filesystem
Acceptance criteria
Make custom path join utility which is aware of the scheme of a URI to understand if it is local filesystem or some other filesystem like s3:// or gs://
Use this path join utility throughout code base
Nice to have - ensure unit and integration tests run on Windows and S3
The text was updated successfully, but these errors were encountered:
Currently, the code base uses a mix of os.path.join (mostly in unit tests) and posixpath.join
Os path join uses the OS defined delimiter, which isn't going to work for e.g. an S3 URI on windows. posixpath join uses the polix delimter which will work for non-filesystem URIs like S3 but not work for windows filesystems.
We have not detected these issues because our developers are not using windows, and the tests running with os.path.join are for local filesystem and not remote filesystem
Acceptance criteria
The text was updated successfully, but these errors were encountered: