Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

WIP: Implement HTTP #468

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open

WIP: Implement HTTP #468

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
5db2d65
intial implementation of http
pjbull Sep 1, 2024
8745ef4
remove unused func
pjbull Sep 1, 2024
0d9a95f
add https support
pjbull Sep 16, 2024
af32f2d
Change dir detection to /
pjbull Sep 17, 2024
12b685a
lint
pjbull Sep 17, 2024
bf97baf
Update https tests
pjbull Feb 14, 2025
771e358
make sigs match
pjbull Feb 14, 2025
a1820f7
Add parsed_url
pjbull Feb 14, 2025
5792061
Add tests for verb methods
pjbull Feb 14, 2025
9fb5727
lint
pjbull Feb 14, 2025
9d2aaf1
test parsed_url
pjbull Feb 14, 2025
020da52
test preserved properties
pjbull Feb 16, 2025
71a73a5
spread out ports; fix warnings
pjbull Feb 16, 2025
93797a0
lint
pjbull Feb 16, 2025
a609d0a
fix full_match
pjbull Feb 17, 2025
8a06929
sleepy upload test
pjbull Feb 17, 2025
fd0ab3f
docs wip
pjbull Feb 17, 2025
cf00630
Update docs
pjbull Feb 18, 2025
2cee52b
lint
pjbull Feb 18, 2025
81903c8
improve http docs
pjbull Feb 20, 2025
acc93a0
add table
pjbull Feb 20, 2025
dff738c
lint
pjbull Feb 20, 2025
204a049
try skipping http rigs on windows in CI
pjbull Feb 20, 2025
262c574
more stable tests
pjbull Feb 27, 2025
c42dc90
test flakiness
pjbull Feb 27, 2025
a176ad5
refresh cert
pjbull Mar 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 91 additions & 82 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,88 +124,97 @@ list(root_dir.glob('**/*.txt'))

Most methods and properties from `pathlib.Path` are supported except for the ones that don't make sense in a cloud context. There are a few additional methods or properties that relate to specific cloud services or specifically for cloud paths.

| Methods + properties | `AzureBlobPath` | `S3Path` | `GSPath` |
|:-----------------------|:------------------|:-----------|:-----------|
| `absolute` | ✅ | ✅ | ✅ |
| `anchor` | ✅ | ✅ | ✅ |
| `as_uri` | ✅ | ✅ | ✅ |
| `drive` | ✅ | ✅ | ✅ |
| `exists` | ✅ | ✅ | ✅ |
| `glob` | ✅ | ✅ | ✅ |
| `is_absolute` | ✅ | ✅ | ✅ |
| `is_dir` | ✅ | ✅ | ✅ |
| `is_file` | ✅ | ✅ | ✅ |
| `is_relative_to` | ✅ | ✅ | ✅ |
| `iterdir` | ✅ | ✅ | ✅ |
| `joinpath` | ✅ | ✅ | ✅ |
| `match` | ✅ | ✅ | ✅ |
| `mkdir` | ✅ | ✅ | ✅ |
| `name` | ✅ | ✅ | ✅ |
| `open` | ✅ | ✅ | ✅ |
| `parent` | ✅ | ✅ | ✅ |
| `parents` | ✅ | ✅ | ✅ |
| `parts` | ✅ | ✅ | ✅ |
| `read_bytes` | ✅ | ✅ | ✅ |
| `read_text` | ✅ | ✅ | ✅ |
| `relative_to` | ✅ | ✅ | ✅ |
| `rename` | ✅ | ✅ | ✅ |
| `replace` | ✅ | ✅ | ✅ |
| `resolve` | ✅ | ✅ | ✅ |
| `rglob` | ✅ | ✅ | ✅ |
| `rmdir` | ✅ | ✅ | ✅ |
| `samefile` | ✅ | ✅ | ✅ |
| `stat` | ✅ | ✅ | ✅ |
| `stem` | ✅ | ✅ | ✅ |
| `suffix` | ✅ | ✅ | ✅ |
| `suffixes` | ✅ | ✅ | ✅ |
| `touch` | ✅ | ✅ | ✅ |
| `unlink` | ✅ | ✅ | ✅ |
| `with_name` | ✅ | ✅ | ✅ |
| `with_stem` | ✅ | ✅ | ✅ |
| `with_suffix` | ✅ | ✅ | ✅ |
| `write_bytes` | ✅ | ✅ | ✅ |
| `write_text` | ✅ | ✅ | ✅ |
| `as_posix` | ❌ | ❌ | ❌ |
| `chmod` | ❌ | ❌ | ❌ |
| `cwd` | ❌ | ❌ | ❌ |
| `expanduser` | ❌ | ❌ | ❌ |
| `group` | ❌ | ❌ | ❌ |
| `hardlink_to` | ❌ | ❌ | ❌ |
| `home` | ❌ | ❌ | ❌ |
| `is_block_device` | ❌ | ❌ | ❌ |
| `is_char_device` | ❌ | ❌ | ❌ |
| `is_fifo` | ❌ | ❌ | ❌ |
| `is_mount` | ❌ | ❌ | ❌ |
| `is_reserved` | ❌ | ❌ | ❌ |
| `is_socket` | ❌ | ❌ | ❌ |
| `is_symlink` | ❌ | ❌ | ❌ |
| `lchmod` | ❌ | ❌ | ❌ |
| `link_to` | ❌ | ❌ | ❌ |
| `lstat` | ❌ | ❌ | ❌ |
| `owner` | ❌ | ❌ | ❌ |
| `readlink` | ❌ | ❌ | ❌ |
| `root` | ❌ | ❌ | ❌ |
| `symlink_to` | ❌ | ❌ | ❌ |
| `as_url` | ✅ | ✅ | ✅ |
| `clear_cache` | ✅ | ✅ | ✅ |
| `cloud_prefix` | ✅ | ✅ | ✅ |
| `copy` | ✅ | ✅ | ✅ |
| `copytree` | ✅ | ✅ | ✅ |
| `download_to` | ✅ | ✅ | ✅ |
| `etag` | ✅ | ✅ | ✅ |
| `fspath` | ✅ | ✅ | ✅ |
| `is_junction` | ✅ | ✅ | ✅ |
| `is_valid_cloudpath` | ✅ | ✅ | ✅ |
| `rmtree` | ✅ | ✅ | ✅ |
| `upload_from` | ✅ | ✅ | ✅ |
| `validate` | ✅ | ✅ | ✅ |
| `walk` | ✅ | ✅ | ✅ |
| `with_segments` | ✅ | ✅ | ✅ |
| `blob` | ✅ | ❌ | ✅ |
| `bucket` | ❌ | ✅ | ✅ |
| `container` | ✅ | ❌ | ❌ |
| `key` | ❌ | ✅ | ❌ |
| `md5` | ✅ | ❌ | ✅ |
| Methods + properties | `AzureBlobPath` | `GSPath` | `HttpsPath` | `S3Path` |
|:-----------------------|:------------------|:-----------|:--------------|:-----------|
| `absolute` | ✅ | ✅ | ✅ | ✅ |
| `anchor` | ✅ | ✅ | ✅ | ✅ |
| `as_uri` | ✅ | ✅ | ✅ | ✅ |
| `drive` | ✅ | ✅ | ✅ | ✅ |
| `exists` | ✅ | ✅ | ✅ | ✅ |
| `glob` | ✅ | ✅ | ✅ | ✅ |
| `is_absolute` | ✅ | ✅ | ✅ | ✅ |
| `is_dir` | ✅ | ✅ | ✅ | ✅ |
| `is_file` | ✅ | ✅ | ✅ | ✅ |
| `is_junction` | ✅ | ✅ | ✅ | ✅ |
| `is_relative_to` | ✅ | ✅ | ✅ | ✅ |
| `iterdir` | ✅ | ✅ | ✅ | ✅ |
| `joinpath` | ✅ | ✅ | ✅ | ✅ |
| `match` | ✅ | ✅ | ✅ | ✅ |
| `mkdir` | ✅ | ✅ | ✅ | ✅ |
| `name` | ✅ | ✅ | ✅ | ✅ |
| `open` | ✅ | ✅ | ✅ | ✅ |
| `parent` | ✅ | ✅ | ✅ | ✅ |
| `parents` | ✅ | ✅ | ✅ | ✅ |
| `parts` | ✅ | ✅ | ✅ | ✅ |
| `read_bytes` | ✅ | ✅ | ✅ | ✅ |
| `read_text` | ✅ | ✅ | ✅ | ✅ |
| `relative_to` | ✅ | ✅ | ✅ | ✅ |
| `rename` | ✅ | ✅ | ✅ | ✅ |
| `replace` | ✅ | ✅ | ✅ | ✅ |
| `resolve` | ✅ | ✅ | ✅ | ✅ |
| `rglob` | ✅ | ✅ | ✅ | ✅ |
| `rmdir` | ✅ | ✅ | ✅ | ✅ |
| `samefile` | ✅ | ✅ | ✅ | ✅ |
| `stat` | ✅ | ✅ | ✅ | ✅ |
| `stem` | ✅ | ✅ | ✅ | ✅ |
| `suffix` | ✅ | ✅ | ✅ | ✅ |
| `suffixes` | ✅ | ✅ | ✅ | ✅ |
| `touch` | ✅ | ✅ | ✅ | ✅ |
| `unlink` | ✅ | ✅ | ✅ | ✅ |
| `walk` | ✅ | ✅ | ✅ | ✅ |
| `with_name` | ✅ | ✅ | ✅ | ✅ |
| `with_segments` | ✅ | ✅ | ✅ | ✅ |
| `with_stem` | ✅ | ✅ | ✅ | ✅ |
| `with_suffix` | ✅ | ✅ | ✅ | ✅ |
| `write_bytes` | ✅ | ✅ | ✅ | ✅ |
| `write_text` | ✅ | ✅ | ✅ | ✅ |
| `as_posix` | ❌ | ❌ | ❌ | ❌ |
| `chmod` | ❌ | ❌ | ❌ | ❌ |
| `cwd` | ❌ | ❌ | ❌ | ❌ |
| `expanduser` | ❌ | ❌ | ❌ | ❌ |
| `group` | ❌ | ❌ | ❌ | ❌ |
| `hardlink_to` | ❌ | ❌ | ❌ | ❌ |
| `home` | ❌ | ❌ | ❌ | ❌ |
| `is_block_device` | ❌ | ❌ | ❌ | ❌ |
| `is_char_device` | ❌ | ❌ | ❌ | ❌ |
| `is_fifo` | ❌ | ❌ | ❌ | ❌ |
| `is_mount` | ❌ | ❌ | ❌ | ❌ |
| `is_reserved` | ❌ | ❌ | ❌ | ❌ |
| `is_socket` | ❌ | ❌ | ❌ | ❌ |
| `is_symlink` | ❌ | ❌ | ❌ | ❌ |
| `lchmod` | ❌ | ❌ | ❌ | ❌ |
| `lstat` | ❌ | ❌ | ❌ | ❌ |
| `owner` | ❌ | ❌ | ❌ | ❌ |
| `readlink` | ❌ | ❌ | ❌ | ❌ |
| `root` | ❌ | ❌ | ❌ | ❌ |
| `symlink_to` | ❌ | ❌ | ❌ | ❌ |
| `as_url` | ✅ | ✅ | ✅ | ✅ |
| `clear_cache` | ✅ | ✅ | ✅ | ✅ |
| `client` | ✅ | ✅ | ✅ | ✅ |
| `cloud_prefix` | ✅ | ✅ | ✅ | ✅ |
| `copy` | ✅ | ✅ | ✅ | ✅ |
| `copytree` | ✅ | ✅ | ✅ | ✅ |
| `download_to` | ✅ | ✅ | ✅ | ✅ |
| `from_uri` | ✅ | ✅ | ✅ | ✅ |
| `fspath` | ✅ | ✅ | ✅ | ✅ |
| `full_match` | ✅ | ✅ | ✅ | ✅ |
| `is_valid_cloudpath` | ✅ | ✅ | ✅ | ✅ |
| `parser` | ✅ | ✅ | ✅ | ✅ |
| `rmtree` | ✅ | ✅ | ✅ | ✅ |
| `upload_from` | ✅ | ✅ | ✅ | ✅ |
| `validate` | ✅ | ✅ | ✅ | ✅ |
| `etag` | ✅ | ✅ | ❌ | ✅ |
| `blob` | ✅ | ✅ | ❌ | ❌ |
| `bucket` | ❌ | ✅ | ❌ | ✅ |
| `md5` | ✅ | ✅ | ❌ | ❌ |
| `container` | ✅ | ❌ | ❌ | ❌ |
| `delete` | ❌ | ❌ | ✅ | ❌ |
| `get` | ❌ | ❌ | ✅ | ❌ |
| `head` | ❌ | ❌ | ✅ | ❌ |
| `key` | ❌ | ❌ | ❌ | ✅ |
| `parsed_url` | ❌ | ❌ | ✅ | ❌ |
| `post` | ❌ | ❌ | ✅ | ❌ |
| `put` | ❌ | ❌ | ✅ | ❌ |

----

Expand Down
10 changes: 8 additions & 2 deletions cloudpathlib/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@
from .azure.azblobclient import AzureBlobClient
from .azure.azblobpath import AzureBlobPath
from .cloudpath import CloudPath, implementation_registry
from .s3.s3client import S3Client
from .gs.gspath import GSPath
from .gs.gsclient import GSClient
from .gs.gspath import GSPath
from .http.httpclient import HttpClient, HttpsClient
from .http.httppath import HttpPath, HttpsPath
from .s3.s3client import S3Client
from .s3.s3path import S3Path


Expand All @@ -27,6 +29,10 @@
"implementation_registry",
"GSClient",
"GSPath",
"HttpClient",
"HttpsClient",
"HttpPath",
"HttpsPath",
"S3Client",
"S3Path",
]
20 changes: 11 additions & 9 deletions cloudpathlib/cloudpath.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@
Generator,
List,
Optional,
Sequence,
Tuple,
Type,
TYPE_CHECKING,
Expand Down Expand Up @@ -299,11 +298,11 @@

@property
def _no_prefix(self) -> str:
return self._str[len(self.cloud_prefix) :]
return self._str[len(self.anchor) :]

@property
def _no_prefix_no_drive(self) -> str:
return self._str[len(self.cloud_prefix) + len(self.drive) :]
return self._str[len(self.anchor) + len(self.drive) :]

@overload
@classmethod
Expand Down Expand Up @@ -909,9 +908,9 @@
# absolute)
if not isinstance(other, CloudPath):
raise ValueError(f"{self} is a cloud path, but {other} is not")
if self.cloud_prefix != other.cloud_prefix:
if self.anchor != other.anchor:
raise ValueError(
f"{self} is a {self.cloud_prefix} path, but {other} is a {other.cloud_prefix} path"
f"{self} is a {self.anchor} path, but {other} is a {other.anchor} path"
)

kwargs = dict(walk_up=walk_up)
Expand Down Expand Up @@ -939,6 +938,9 @@
# strip scheme from start of pattern before testing
if pattern.startswith(self.anchor + self.drive):
pattern = pattern[len(self.anchor + self.drive) :]
elif pattern.startswith(self.anchor):

Check warning on line 941 in cloudpathlib/cloudpath.py

View check run for this annotation

Codecov / codecov/patch

cloudpathlib/cloudpath.py#L941 

Added line #L941 was not covered by tests
# for http paths, keep leading slash
pattern = pattern[len(self.anchor) - 1 :]

Check warning on line 943 in cloudpathlib/cloudpath.py

View check run for this annotation

Codecov / codecov/patch

cloudpathlib/cloudpath.py#L943 

Added line #L943 was not covered by tests

# remove drive, which is kept on normal dispatch to pathlib
return PurePosixPath(self._no_prefix_no_drive).full_match( # type: ignore[attr-defined]
Expand Down Expand Up @@ -969,7 +971,7 @@
return self._dispatch_to_path("parent")

@property
def parents(self) -> Sequence[Self]:
def parents(self) -> Tuple[Self, ...]:
return self._dispatch_to_path("parents")

@property
Expand Down Expand Up @@ -1224,7 +1226,7 @@
)
elif subpath.is_dir():
subpath.copytree(
destination / subpath.name,
destination / (subpath.name + ("" if subpath.name.endswith("/") else "/")),
force_overwrite_to_cloud=force_overwrite_to_cloud,
ignore=ignore,
)
Expand Down Expand Up @@ -1258,8 +1260,8 @@
path = path[1:]

# add prefix/anchor if it is not already
if not path.startswith(self.cloud_prefix):
path = f"{self.cloud_prefix}{path}"
if not path.startswith(self.anchor):
path = f"{self.anchor}{path}"

return self.client.CloudPath(path)

Expand Down
9 changes: 9 additions & 0 deletions cloudpathlib/http/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from .httpclient import HttpClient, HttpsClient
from .httppath import HttpPath, HttpsPath

__all__ = [
"HttpClient",
"HttpPath",
"HttpsClient",
"HttpsPath",
]
Loading
Loading