Proposal: Integrate trusted cloud-native registry Harbor with Dragonfly to provide a joint image management and distribution solution to support containerized environments

[steven-zou]

The details of the proposal are defined in the Dragonfly issue dragonflyoss/dragonfly-archived#108.

This issue is created to link the proposal issue and track the related work progress of prototype development.

[steven-zou]

Todo list of Harbor:

• User can configure Dragonfly service endpoint from the web portal - 08/31

  • Depend on Dragonfly related API

• User can pre-release or promote the specified image to the Dragonfly supervise node - 09/14

• Extended: User can setup a automation policy to pre-release or promote the matched images to the Dragonfly supervise node - 10/14

[steven-zou]

Copy the details of proposal to here:

STATUS: [INPROGRESS]

Integrate trusted cloud-native registry Harbor with Dragonfly to provide a joint image management and distribution solution to support containerized environments.

Backgrounds:

Harbor: Project Harbor is an open source trusted cloud-native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity, and management. Having a registry closer to the build and run environment can improve the image transfer efficiency. Harbor supports replication of images between registries and also offers advanced security features such as user management, access control, and activity auditing. For more details, please refer to README.

Dragonfly: Dragonfly is an intelligent P2P based file distribution system. It aims to resolve issues related to low-efficiency, low-success rate and a waste of network bandwidth in file transferring process. Especially in large-scale file distribution scenarios such as application distribution, cache distribution, log distribution, image distribution, etc. For more details, please refer to README

Motivations:

With the emergence and development of Kubernetes, it's becoming possible to run and operate large-scale containerized applications and services in enterprise environments. Meanwhile, there are still existing big challenges which cannot be ignored. How to securely and effectively manage the lots of container images produced in the enterprise organizations and distribute them to the large-scale runtimes with less time and efforts when starting applications or services on demand. To address the above challenge, we should build a joint solution from the open source trust cloud-native registry Harbor and the open source intelligent P2P based file distribution system Dragonfly.

These two open sourced projects have very obviously complementary advantages to each other and the joint solution will definitely expand the scenarios of image lifecycle management and improve the securities, reliabilities, and efficiencies.

Idea:

• The integration should be a loose couple way, by calling related APIs to complete the required work. The system admin of Harbor registry can configure the related options to enable the API calling from Dragonfly side. The options may include but not limit the following ones:

  • API endpoint
  • API access token or required credentials
  • Possible calling policies or automation rules etc.

• The integrated configurations can be verified to make sure the connection between the two systems is not broken by testing or ''dry run" etc.

• The images are produced by CI/CD pipeline or any other ways and pushed to the Harbor registry. The newly pushed images can be marked with labels automatically or manually. In addition, the admin of registry can also scan the images to make sure it's secure. Of course, the admins can do any other management work if they want.

• The admin of registry can select any ready image to promote it to the supervise node of Dragonfly P2P network for the upcoming image pulling requests to improve the distribution performance. The promote action can be triggered by clicking button or auto-triggered by pre-configured rules/policies (If match some conditions, then promote it).

• Then if the containerized environments need to pull that image, the Dragonfly will help to distribute it to the nodes by layers via the P2P network.

Basic Workflow:

Architecture:

An architecture design based on the above draft idea:

The components with light blue background are the new things need to be implemented.

• The controller provides related API methods to handle the overall workflow
• The config will handle loading and saving of the related configurations based on the existing Harbor configuration service
• The policy engine handles the CRUD of policy as well as the evaluation of the policies
• Hook controller is designed to take charge of event hook related things
• Image distribution driver will define as an interface to provide the related methods to talk to the Dragonfly API
• API from Dragonfly side provides the required capabilities of publishing images to the supervise node and returning necessary status info and/or metrics if required

Followups:

• Discuss the idea and the draft design
• Confirm the Dragonfly API capabilities
• Make the development plan
• Setup the virtual team

[ghost]

Thanks for driving this @steven-zou!

/cc @bradmeiseles

[steven-zou]

Move proposal to goharbor/community#13

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.