Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

added remote sync updates #399

Closed
wants to merge 8 commits into from
Closed

added remote sync updates #399

wants to merge 8 commits into from

Conversation

zackbradys
Copy link
Member

@zackbradys zackbradys commented Feb 3, 2025 •
edited
Loading

Please check below, if the PR fulfills these requirements:

  • Commit(s) and code follow the repositories guidelines.
  • Test(s) have been added or updated to support these change(s).
  • Doc(s) have been added or updated to support these change(s).

Associated Links:

Types of Changes:

  • Feature

Proposed Changes:

  • [Feature] Added support for hauler store sync to accept multiple local and remote archives of http/https
  • [Feature] Improved logging for related commands and outputs of hauler store sync
  • [Feature] defaulted local file of `hauler-manifest.yaml (new behavior) to matches other commands...

Verification/Testing of Changes:

zackbradys@Zacks-MacBook-Pro test % ls -la
total 40
drwxr-xr-x    5 zackbradys  staff    160 Feb  3 02:41 .
drwxr-xr-x  213 zackbradys  staff   6816 Feb  2 10:17 ..
-rwxr-xr-x    1 zackbradys  staff   1266 Feb  3 02:38 hauler-manifest.yaml
-rw-r--r--    1 zackbradys  staff  14970 Feb  3 02:33 rancher-cluster-templates-0.5.2.tgz
drwxr-xr-x    8 zackbradys  staff    256 Feb  3 00:29 tmp
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync         
2025-02-03 02:41:24 INF processing manifest [hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:41:24 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:41:24 INF adding 'image' [busybox] to the store
2025-02-03 02:41:30 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:41:30 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:41:31 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:41:31 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:41:33 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:41:33 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:41:33 INF adding 'chart' [rancher] to the store
2025-02-03 02:41:35 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:41:35 INF adding 'chart' [rancher] to the store
2025-02-03 02:41:36 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:41:36 INF adding 'chart' [rancher] to the store
2025-02-03 02:41:38 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:41:38 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:41:38 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:41:38 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:41:39 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:41:39 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:41:39 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:41:39 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:41:39 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:41:39 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:41:39 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:41:39 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:41:39 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:41:39 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug
2025-02-03 02:41:55 DBG running cli command [hauler store sync]
2025-02-03 02:41:55 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:41:55 DBG using temporary directory at [/var/folders/5f/lyy8pc6s62dbckp8sy31ft5r0000gn/T/hauler1037744451]
2025-02-03 02:41:55 INF processing manifest [hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:41:55 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:41:55 INF adding 'image' [busybox] to the store
2025-02-03 02:41:56 DBG multi-arch image [true]
2025-02-03 02:42:00 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:42:00 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:42:00 DBG multi-arch image [true]
2025-02-03 02:42:00 DBG platform for image [linux/amd64]
2025-02-03 02:42:00 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:42:00 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:42:01 DBG multi-arch image [false]
2025-02-03 02:42:02 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:42:02 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:42:02 INF adding 'chart' [rancher] to the store
2025-02-03 02:42:04 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:42:04 INF adding 'chart' [rancher] to the store
2025-02-03 02:42:05 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:42:05 INF adding 'chart' [rancher] to the store
2025-02-03 02:42:07 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:42:07 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:42:08 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:42:08 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:42:08 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:42:08 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:42:08 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:42:09 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:42:09 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:42:09 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml         
2025-02-03 02:43:17 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:43:17 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:43:17 INF adding 'image' [busybox] to the store
2025-02-03 02:43:23 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:43:23 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:43:24 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:43:24 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:43:26 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:43:26 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:43:26 INF adding 'chart' [rancher] to the store
2025-02-03 02:43:27 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:43:27 INF adding 'chart' [rancher] to the store
2025-02-03 02:43:29 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:43:29 INF adding 'chart' [rancher] to the store
2025-02-03 02:43:30 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:43:30 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:43:31 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:43:31 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:43:31 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:43:31 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:43:32 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:43:32 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:43:32 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:43:32 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store                                                                                  
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml
2025-02-03 02:44:13 DBG running cli command [hauler store sync]
2025-02-03 02:44:13 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:44:13 DBG using temporary directory at [/var/folders/5f/lyy8pc6s62dbckp8sy31ft5r0000gn/T/hauler3233246142]
2025-02-03 02:44:13 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:44:13 DBG detected remote manifest... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml]
2025-02-03 02:44:13 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:44:13 INF adding 'image' [busybox] to the store
2025-02-03 02:44:14 DBG multi-arch image [true]
2025-02-03 02:44:18 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:44:18 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:44:18 DBG multi-arch image [true]
2025-02-03 02:44:18 DBG platform for image [linux/amd64]
2025-02-03 02:44:19 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:44:19 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:44:19 DBG multi-arch image [false]
2025-02-03 02:44:21 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:44:21 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:44:21 INF adding 'chart' [rancher] to the store
2025-02-03 02:44:23 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:44:23 INF adding 'chart' [rancher] to the store
2025-02-03 02:44:24 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:44:24 INF adding 'chart' [rancher] to the store
2025-02-03 02:44:26 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:44:26 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:44:27 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:44:27 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:44:27 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:44:27 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:44:27 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:44:28 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:44:28 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:44:28 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store                
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info

+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % export HAULER_TEMP_DIR=/Users/zackbradys/Downloads/RGS/test/tmp
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml
2025-02-03 02:45:38 DBG running cli command [hauler store sync]
2025-02-03 02:45:38 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:45:38 DBG using temporary directory at [/Users/zackbradys/Downloads/RGS/test/tmp/hauler61401760]
2025-02-03 02:45:38 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:45:38 DBG detected remote manifest... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml]
2025-02-03 02:45:39 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:45:39 INF adding 'image' [busybox] to the store
2025-02-03 02:45:39 DBG multi-arch image [true]
2025-02-03 02:45:43 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:45:43 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:45:44 DBG multi-arch image [true]
2025-02-03 02:45:44 DBG platform for image [linux/amd64]
2025-02-03 02:45:44 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:45:44 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:45:45 DBG multi-arch image [false]
2025-02-03 02:45:46 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:45:46 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:45:46 INF adding 'chart' [rancher] to the store
2025-02-03 02:45:48 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:45:48 INF adding 'chart' [rancher] to the store
2025-02-03 02:45:50 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:45:50 INF adding 'chart' [rancher] to the store
2025-02-03 02:45:51 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:45:51 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:45:52 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:45:52 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:45:52 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:45:52 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:45:53 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:45:53 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:45:53 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:45:53 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % unset HAULER_TEMP_DIR
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug -t /Users/zackbradys/Downloads/RGS/test/tmp -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml 
2025-02-03 02:46:34 DBG running cli command [hauler store sync]
2025-02-03 02:46:34 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:46:34 DBG using temporary directory at [/Users/zackbradys/Downloads/RGS/test/tmp/hauler1717808495]
2025-02-03 02:46:34 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:46:34 DBG detected remote manifest... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml]
2025-02-03 02:46:35 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:46:35 INF adding 'image' [busybox] to the store
2025-02-03 02:46:35 DBG multi-arch image [true]
2025-02-03 02:46:39 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:46:39 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:46:39 DBG multi-arch image [true]
2025-02-03 02:46:39 DBG platform for image [linux/amd64]
2025-02-03 02:46:40 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:46:40 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:46:40 DBG multi-arch image [false]
2025-02-03 02:46:42 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:46:42 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:46:42 INF adding 'chart' [rancher] to the store
2025-02-03 02:46:43 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:46:43 INF adding 'chart' [rancher] to the store
2025-02-03 02:46:45 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:46:45 INF adding 'chart' [rancher] to the store
2025-02-03 02:46:46 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:46:46 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:46:47 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:46:47 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:46:47 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:46:47 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:46:48 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:46:48 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:46:48 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:46:48 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:46:48 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:46:48 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:46:48 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:46:48 INF successfully added 'file' [hauler/rke2-install.sh:latest]

Additional Context:

  • N/A

@zackbradys zackbradys requested a review from a team February 3, 2025 07:53
@zackbradys zackbradys self-assigned this Feb 3, 2025
@zackbradys zackbradys added enhancement New feature or request size/M Denotes an issue/PR requiring a relatively moderate amount of work labels Feb 3, 2025
@zackbradys zackbradys added this to the Hauler v1.2.0 milestone Feb 3, 2025
@zackbradys zackbradys linked an issue Feb 3, 2025 that may be closed by this pull request
[feature] Allow hauler to load and sync remote files #208
Closed
amartin120
amartin120 reviewed Feb 3, 2025
View reviewed changes
internal/flags/store.go Show resolved Hide resolved
internal/flags/sync.go Outdated Show resolved Hide resolved
cmd/hauler/cli/store/sync.go Show resolved Hide resolved
@zackbradys
Copy link
Member Author 

zackbradys@Zacks-MacBook-Pro hauler % hauler store sync -f hauler-manifest-pipeline.yaml -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml
2025-02-03 20:32:29 INF processing manifest [hauler-manifest-pipeline.yaml] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 20:32:29 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 20:32:29 INF adding image [busybox] to the store
2025-02-03 20:32:35 INF successfully added image [index.docker.io/library/busybox:latest]
2025-02-03 20:32:35 INF adding image [busybox:stable] to the store
2025-02-03 20:32:36 INF successfully added image [index.docker.io/library/busybox:stable]
2025-02-03 20:32:36 INF adding image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 20:32:38 INF successfully added image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 20:32:38 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 20:32:38 INF adding chart [rancher] to the store
2025-02-03 20:32:40 INF successfully added chart [hauler/rancher:2.10.2]
2025-02-03 20:32:40 INF adding chart [rancher] to the store
2025-02-03 20:32:41 INF successfully added chart [hauler/rancher:2.8.4]
2025-02-03 20:32:41 INF adding chart [rancher] to the store
2025-02-03 20:32:42 INF successfully added chart [hauler/rancher:2.8.3]
2025-02-03 20:32:42 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 20:32:43 INF successfully added chart [hauler/hauler-helm:1.1.1]
2025-02-03 20:32:43 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 20:32:43 INF successfully added chart [hauler/hauler-helm:1.0.6]
2025-02-03 20:32:43 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 20:32:44 INF successfully added chart [hauler/hauler-helm:1.0.4]
2025-02-03 20:32:44 INF adding chart [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 20:32:44 INF successfully added chart [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 20:32:44 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 20:32:44 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/install.sh:latest]
2025-02-03 20:32:44 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/rke2-install.sh:latest]
2025-02-03 20:32:44 INF adding file [testdata/hauler-manifest.yaml] to the store as [hauler/hauler-manifest.yaml:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/hauler-manifest.yaml:latest]
2025-02-03 20:32:44 INF adding file [testdata/hauler-manifest.yaml] to the store as [hauler/hauler-manifest-local.yaml:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/hauler-manifest-local.yaml:latest]
2025-02-03 20:32:44 INF processing completed successfully
2025-02-03 20:32:44 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 20:32:45 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 20:32:45 INF adding image [busybox] to the store
2025-02-03 20:32:50 INF successfully added image [index.docker.io/library/busybox:latest]
2025-02-03 20:32:50 INF adding image [busybox:stable] to the store
2025-02-03 20:32:51 INF successfully added image [index.docker.io/library/busybox:stable]
2025-02-03 20:32:51 INF adding image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 20:32:53 INF successfully added image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 20:32:53 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 20:32:53 INF adding chart [rancher] to the store
2025-02-03 20:32:54 INF successfully added chart [hauler/rancher:2.10.2]
2025-02-03 20:32:54 INF adding chart [rancher] to the store
2025-02-03 20:32:56 INF successfully added chart [hauler/rancher:2.8.4]
2025-02-03 20:32:56 INF adding chart [rancher] to the store
2025-02-03 20:32:57 INF successfully added chart [hauler/rancher:2.8.3]
2025-02-03 20:32:57 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 20:32:58 INF successfully added chart [hauler/hauler-helm:1.1.1]
2025-02-03 20:32:58 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 20:32:58 INF successfully added chart [hauler/hauler-helm:1.0.6]
2025-02-03 20:32:58 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 20:32:58 INF successfully added chart [hauler/hauler-helm:1.0.4]
2025-02-03 20:32:58 INF adding chart [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 20:32:58 INF successfully added chart [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 20:32:58 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 20:32:58 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 20:32:58 INF successfully added file [hauler/install.sh:latest]
2025-02-03 20:32:58 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 20:32:59 INF successfully added file [hauler/rke2-install.sh:latest]
2025-02-03 20:32:59 INF processing completed successfully

@zackbradys zackbradys requested review from amartin120 and a team February 4, 2025 04:26
@amartin120
Copy link
Contributor

Thanks for the updates and the back and forth. Please update the PR description to reflect the latest details and this should be good to go.

@zackbradys
Copy link
Member Author

updated description as requested! @amartin120

@zackbradys zackbradys closed this Feb 4, 2025
@zackbradys zackbradys deleted the remote-sync-updates branch February 5, 2025 03:47
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@amartin120 amartin120 Awaiting requested review from amartin120

Assignees

@zackbradys zackbradys

Labels
enhancement New feature or request size/M Denotes an issue/PR requiring a relatively moderate amount of work
Projects
Hauler
Status: Resolved
Milestone
Hauler v1.2.0
Development

Successfully merging this pull request may close these issues.

[feature] Allow hauler to load and sync remote files
2 participants
@zackbradys @amartin120