Trivy Scan does not work (Failed to download vulnerability DB)

[ytooyama]

Expected behavior and actual behavior:
Trivy Scan does not work

Steps to reproduce the problem:
Please provide the steps to reproduce this problem.

Installation environment:

• CentOS Stream 9
• Python 3.9.13
• Docker 20.10.17
• Docker Compose 1.29.2

Installation procedure:

# wget 'https://github.com/goharbor/harbor/releases/download/v2.5.1/harbor-online-installer-v2.5.1.tgz'
# tar zvxf harbor-online-installer-v2.5.1.tgz
# cd harbor
# cp harbor.yml.tmpl harbor.yml
# vi harbor.yml

hostname: 172.16.214.190
...
# Trivy configuration
trivy:
   insecure: true
   github_token: ghp_*****  # My GitHub Token
...

Install the Harbor:

# ./install.sh --with-trivy

# cat /etc/docker/daemon.json 
{
  "insecure-registries" : ["172.16.214.190"]
}

# docker login 172.16.214.190
# docker image pull ubuntu:20.04
# docker tag alpine:latest 172.16.214.190/library/ubuntu:20.04
# docker push 172.16.214.190/library/ubuntu:20.04

Go Harbor UI. Scan the Image, Then Result

2022-06-23T06:25:05Z [INFO] [/pkg/scan/job.go:385]: {
  "uuid": "2b2c92e2-f2b6-11ec-9f1c-0242ac120009",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://trivy-adapter:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "access_credential": "[HIDDEN]",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.26.0",
  "create_time": "2022-06-23T05:34:36.395502Z",
  "update_time": "2022-06-23T05:34:36.395503Z"
}
2022-06-23T06:25:05Z [INFO] [/pkg/scan/job.go:385]: {
  "registry": {
    "url": "http://core:8080",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 1,
    "repository": "library/ubuntu",
    "tag": "20.04",
    "digest": "sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253",
    "mime_type": "application/vnd.docker.distribution.manifest.v2+json"
  }
}
2022-06-23T06:25:05Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2022-06-23T06:25:05Z [INFO] [/pkg/scan/job.go:222]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2022-06-23T06:25:07Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:12Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:17Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:22Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:27Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:32Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:37Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:42Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:47Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:53Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:25:58Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-06-23T06:26:03Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2022-06-23T06:25:08.560Z	�[34mINFO�[0m	Need to update DB
2022-06-23T06:25:08.560Z	�[34mINFO�[0m	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-06-23T06:25:08.560Z	�[34mINFO�[0m	Downloading DB...
2022-06-23T06:26:02.822Z	�[31mFATAL�[0m	DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: Get "https://ghcr.io/v2/": net/http: TLS handshake timeout
: general response handler: unexpected status code: 500, expected: 200

Versions:
Please specify the versions of following systems.

• harbor version: [2.5.1]
• docker engine version: [20.10.17]
• docker-compose version: [1.29.2]

Additional context:

• Harbor config files: You can get them by packaging harbor.yml and files in the same directory, including subdirectory.

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 172.16.214.190

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
#https:
  # https port for harbor, default is 443
  #port: 443
  # The path of cert and key files for nginx
  #certificate: /your/certificate/path
  #private_key: /your/private/key/path

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 100
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 900

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

# Trivy configuration
#
# Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases.
# It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached
# in the local file system. In addition, the database contains the update timestamp so Trivy can detect whether it
# should download a newer version from the Internet or use the cached one. Currently, the database is updated every
# 12 hours and published as a new release to GitHub.
trivy:
  # ignoreUnfixed The flag to display only fixed vulnerabilities
  ignore_unfixed: false
  # skipUpdate The flag to enable or disable Trivy DB downloads from GitHub
  #
  # You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues.
  # If the flag is enabled you have to download the `trivy-offline.tar.gz` archive manually, extract `trivy.db` and
  # `metadata.json` files and mount them in the `/home/scanner/.cache/trivy/db` path.
  skip_update: false
  #
  # The offline_scan option prevents Trivy from sending API requests to identify dependencies.
  # Scanning JAR files and pom.xml may require Internet access for better detection, but this option tries to avoid it.
  # For example, the offline mode will not try to resolve transitive dependencies in pom.xml when the dependency doesn't
  # exist in the local repositories. It means a number of detected vulnerabilities might be fewer in offline mode.
  # It would work if all the dependencies are in local.
  # This option doesn’t affect DB download. You need to specify "skip-update" as well as "offline-scan" in an air-gapped environment.
  offline_scan: false
  #
  # insecure The flag to skip verifying registry certificate
  insecure: true
  # github_token The GitHub access token to download Trivy DB
  #
  # Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough
  # for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000
  # requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult
  # https://developer.github.com/v3/#rate-limiting
  #
  # You can create a GitHub token by following the instructions in
  # https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
  #
  github_token: ghp_-snip-

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.5.0

# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# Uncomment external_redis if using external Redis server
# external_redis:
#   # support redis, redis+sentinel
#   # host for redis: <host_redis>:<port_redis>
#   # host for redis+sentinel:
#   #  <host_sentinel1>:<port_sentinel1>,<host_sentinel2>:<port_sentinel2>,<host_sentinel3>:<port_sentinel3>
#   host: redis:6379
#   password: 
#   # sentinel_master_set must be set to support redis+sentinel
#   #sentinel_master_set:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   chartmuseum_db_index: 3
#   trivy_db_index: 5
#   idle_timeout_seconds: 30

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy

# metric:
#   enabled: false
#   port: 9090
#   path: /metrics

# Trace related config
# only can enable one trace provider(jaeger or otel) at the same time,
# and when using jaeger as provider, can only enable it with agent mode or collector mode.
# if using jaeger collector mode, uncomment endpoint and uncomment username, password if needed
# if using jaeger agetn mode uncomment agent_host and agent_port
# trace:
#   enabled: true
#   # set sample_rate to 1 if you wanna sampling 100% of trace data; set 0.5 if you wanna sampling 50% of trace data, and so forth
#   sample_rate: 1
#   # # namespace used to differenciate different harbor services
#   # namespace:
#   # # attributes is a key value dict contains user defined attributes used to initialize trace provider
#   # attributes:
#   #   application: harbor
#   # # jaeger should be 1.26 or newer.
#   # jaeger:
#   #   endpoint: http://hostname:14268/api/traces
#   #   username:
#   #   password:
#   #   agent_host: hostname
#   #   # export trace data by jaeger.thrift in compact mode
#   #   agent_port: 6831
#   # otel:
#   #   endpoint: hostname:4318
#   #   url_path: /v1/traces
#   #   compression: false
#   #   insecure: true
#   #   timeout: 10s

# enable purge _upload directories
upload_purging:
  enabled: true
  # remove files in _upload directories which exist for a period of time, default is one week.
  age: 168h
  # the interval of the purge operations
  interval: 24h
  dryrun: false

• Log files: You can get them by package the /var/log/harbor/ .

[zyyw]

this issue may due to your network config environment. please refer to this for more details:

• Trivy vulnerability DB does not download #15761 (comment)
• Unable to download trivy vulnerability DB #14146

Tried fresh deploying harbor 2.5.1 with offline installer in my vm, there is no such error:

2022-06-23T06:26:03Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2022-06-23T06:25:08.560Z	�[34mINFO�[0m	Need to update DB
2022-06-23T06:25:08.560Z	�[34mINFO�[0m	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-06-23T06:25:08.560Z	�[34mINFO�[0m	Downloading DB...
2022-06-23T06:26:02.822Z	�[31mFATAL�[0m	DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: Get "https://ghcr.io/v2/": net/http: TLS handshake timeout
: general response handler: unexpected status code: 500, expected: 200

[zyyw]

@ytooyama could you try to set proxy for trivy and see if it works, i.e.

proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy

[ytooyama]

@zyyw Thanks. I'll try that.

[ytooyama]

It was already set up.

# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy

Reference information:

• https://forums.centos.org/viewtopic.php?t=74270
• DNS Not Resolving under Network [CentOS8] docker/for-linux#957 (comment)

I have set up Firewalled.

]# dnf install firewalld bind-utils
]# systemctl enable --now firewalld
]# firewall-cmd --zone=trusted --add-interface=docker0 --permanent
]# firewall-cmd --reload

On the Docker Host, DNS name resolution is available!

]# nslookup ghcr.io
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	ghcr.io
Address: 3.114.109.192

But, It seems that ghcr.io cannot respond inside the container.

]# docker run busybox nslookup ghcr.io
Server:		1.1.1.1
Address:	1.1.1.1:53

Non-authoritative answer:

*** Can't find ghcr.io: No answer

Set The dnf params (on the docker Host).

]# cat /etc/docker/daemon.json

{
  "dns": [
    "8.8.8.8", "8.8.4.4"
  ],
  "dns-opts": [
    "timeout:2"
  ]
}


]# systemctl restart docker

Didn't resolve.

]# docker run busybox nslookup ghcr.io
Server:		8.8.8.8
Address:	8.8.8.8:53

Non-authoritative answer:

*** Can't find ghcr.io: No answer


]# docker run busybox nslookup google.com
Server:		8.8.8.8
Address:	8.8.8.8:53

Non-authoritative answer:
Name:	google.com
Address: 2404:6800:4004:81f::200e

*** Can't find google.com: No answer

Hmm ...It seems that ghcr.io cannot respond inside the container.

[ytooyama]

I had a problem trying to run trivy with docker and may need to fix this issue.

aquasecurity/trivy#1908

[ytooyama]

Sorry. I was trying to run it on Docker in an OpenStack instance.
It looks like this is the same symptom, I needed to adjust the MTU.

https://platform9.com/kb/openstack/no-connectivity-to-docker-containers-within-instance

Set up again from scratch.