Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

replication for big images failed with Gateway Time-out intermittently #3446

Closed
ligc opened this issue Oct 24, 2017 · 4 comments
Closed

replication for big images failed with Gateway Time-out intermittently #3446

ligc opened this issue Oct 24, 2017 · 4 comments
Assignees
@ywk253100
Labels
kind/bug kind/customer-found

Comments

@ligc
Copy link

ligc commented Oct 24, 2017

I have some docker images with a super big file of 1GB in the images, Harbor replication for these images failed intermittently with error "Gateway Time-out", if I disable and re-enable the replication rule to trigger a re-replication, it will succeed.

The Harbor is configured with 2 replication rules to replicate the images to 2 target Harbor registries.

Harbor 1.2.0
Docker 1.12.6
docker-compose 1.9.0

2017-10-23T19:27:48Z [INFO] initializing: repository: library/harbor-test-1000, tags: [latest], source URL: http://registry:5000, destination URL: http://x3.clusters.com, insecure: false, destination user: admin
2017-10-23T19:27:48Z [INFO] initialization completed: project: library, repository: library/harbor-test-1000, tags: [latest], source URL: http://registry:5000, destination URL: http://x3.clusters.com, insecure: false, destination user: admin
2017-10-23T19:27:49Z [WARNING] the status code is 409 when creating project library on http://x3.clusters.com with user admin, try to do next step
2017-10-23T19:27:49Z [INFO] manifest of library/harbor-test-1000:latest pulled successfully from http://registry:5000: sha256:87b7558465d873599d2ab1077fb7564bc70a2ccc08604ce624070f3759c76470
2017-10-23T19:27:49Z [INFO] all blobs of library/harbor-test-1000:latest from http://registry:5000: [sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495]
2017-10-23T19:27:49Z [INFO] blobs of library/harbor-test-1000:latest need to be transferred to http://x3.clusters.com: [sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495]
2017-10-23T19:27:49Z [INFO] transferring blob sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 of library/harbor-test-1000:latest to http://x3.clusters.com ...
2017-10-23T19:27:49Z [INFO] blob sha256:3f6f06182add9ab3642b5912c8d94d53cec765491f1652331321aee97ce9ef52 of library/harbor-test-1000:latest transferred to http://x3.clusters.com completed
2017-10-23T19:27:49Z [INFO] transferring blob sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd of library/harbor-test-1000:latest to http://x3.clusters.com ...
2017-10-23T19:27:50Z [INFO] blob sha256:280aca6ddce2bc2b0904fb525b14e119111c11ca06c3c4d9e4258723da52aecd of library/harbor-test-1000:latest transferred to http://x3.clusters.com completed
2017-10-23T19:27:50Z [INFO] transferring blob sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495 of library/harbor-test-1000:latest to http://x3.clusters.com ...
2017-10-23T19:29:21Z [ERROR] [transfer.go:431]: an error occurred while pushing blob sha256:fbc94b68ab231ff9418684c3e5bc09092be65c2f2b20f36883183824c9942495 of library/harbor-test-1000:latest to http://x3.clusters.com : 504 <html>

<head><title>504 Gateway Time-out</title></head>

<body bgcolor="white">

<center><h1>504 Gateway Time-out</h1></center>

<hr><center>nginx/1.11.13</center>

</body>

</html>
@ywk253100
Copy link
Contributor

Try to add timeout settings in the configuration file of Nginx.

@ligc
Copy link
Author

ligc commented Oct 31, 2017

Thank you, @ywk253100, you suggestion works perfectly.

I added the proxy_send_timeout and proxy_read_timeout in the "location /v2/" section of common/templates/nginx/nginx.http.conf; ./prepare, restart Harbor. Then the 1000MB images could be synced correctly to the remote Harbor repositories.

    location /v2/ {
      proxy_pass http://ui/registryproxy/v2/;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      
      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_buffering off;
      proxy_request_buffering off;

      proxy_send_timeout 900;
      proxy_read_timeout 900;
    }


2017-10-30T18:20:48Z [INFO] initializing: repository: library/harbor-test-1000m, tags: [latest], source URL: http://registry:5000, destination URL: http://x2.clusters.com, insecure: false, destination user: admin
2017-10-30T18:20:48Z [INFO] initialization completed: project: library, repository: library/harbor-test-1000m, tags: [latest], source URL: http://registry:5000, destination URL: http://x2.clusters.com, insecure: false, destination user: admin
2017-10-30T18:20:48Z [WARNING] the status code is 409 when creating project library on http://x2.clusters.com with user admin, try to do next step
2017-10-30T18:20:48Z [INFO] manifest of library/harbor-test-1000m:latest pulled successfully from http://registry:5000: sha256:8eec0c51f62afcc3e979a059ce303be3afe0134c010d0c99c5b7063c61455d2b
2017-10-30T18:20:48Z [INFO] all blobs of library/harbor-test-1000m:latest from http://registry:5000: [sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34]
2017-10-30T18:20:49Z [INFO] blobs of library/harbor-test-1000m:latest need to be transferred to http://x2.clusters.com: [sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34]
2017-10-30T18:20:49Z [INFO] transferring blob sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 of library/harbor-test-1000m:latest to http://x2.clusters.com ...
2017-10-30T18:20:49Z [INFO] blob sha256:38b53c28cb5d501c1dcd09e9cbe163c52a7eba488c9c7c64bb7b10a88e4755d5 of library/harbor-test-1000m:latest transferred to http://x2.clusters.com completed
2017-10-30T18:20:49Z [INFO] transferring blob sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c of library/harbor-test-1000m:latest to http://x2.clusters.com ...
2017-10-30T18:20:50Z [INFO] blob sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93836c of library/harbor-test-1000m:latest transferred to http://x2.clusters.com completed
2017-10-30T18:20:50Z [INFO] transferring blob sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34 of library/harbor-test-1000m:latest to http://x2.clusters.com ...
2017-10-30T18:22:25Z [INFO] blob sha256:578dd27b81aad17150d6f375c1f04f55ea789288daf39909e5a6e61c87ea3b34 of library/harbor-test-1000m:latest transferred to http://x2.clusters.com completed
2017-10-30T18:22:25Z [INFO] manifest of library/harbor-test-1000m:latest exists on source registry http://registry:5000, continue manifest pushing
2017-10-30T18:22:25Z [INFO] manifest of library/harbor-test-1000m:latest has been pushed to http://x2.clusters.com
2017-10-30T18:22:25Z [INFO] no tag needs to be replicated, next state is "finished"

I am fine with closing this issue, not sure if Harbor project wants to document this as a tip of replicating big images or the network between the replicated Harbor repositories is not good enough.

@ywk253100
Copy link
Contributor

@ligc Thank you for your suggestion. I have added it to the wiki page here. And I'm closing this issue.

@moooofly
Copy link

FYI, the above resolution not works for me.

But it dose work if I add the timeout options into server block.

@lf1029698952 lf1029698952 mentioned this issue Apr 19, 2019
Closed
@rafis rafis mentioned this issue Feb 28, 2020
Closed
@YakirShriker YakirShriker mentioned this issue Jul 1, 2020
Closed
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@ywk253100 ywk253100

Labels
kind/bug kind/customer-found
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wy65701436 @ligc @ywk253100 @moooofly