Skip to content
This repository has been archived by the owner on Sep 12, 2018. It is now read-only.

docker-registry does't auto reindex images when sqlite search db is empty #516

Closed
rvrignaud opened this issue Aug 8, 2014 · 2 comments
Closed

docker-registry does't auto reindex images when sqlite search db is empty #516

rvrignaud opened this issue Aug 8, 2014 · 2 comments
Milestone
Not on us - keepi...

Comments

@rvrignaud
Copy link

Hello,

I'm currently running docker-registry (0.7.3) with docker-registry-driver-swift (0.1.0).
When starting a new docker-registry with an emtpy database, search queries are returned empty.
If I push a new image, it's correctly indexed.
Is there any way to force re-indexing of all present images ?
In issue #475, @wking mentioned that docker-registry should automatically repopulate the index. Am I missing something ?
@wking
Copy link
Contributor

wking commented Aug 8, 2014

On Fri, Aug 08, 2014 at 07:58:10AM -0700, Romain Vrignaud wrote:

I'm currently running docker-registry (0.7.3) with
docker-registry-driver-swift (0.1.0).

I don't think the driver should matter.

When starting a new docker-registry with an emtpy database, search
queries are returned empty. If I push a new image, it's correctly
indexed. Is there any way to force re-indexing of all present
images ? In issue #475, @wking mentioned that docker-registry
should automatically repopulate the index. Am I missing something ?

Here's how it's supposed to work:

  1. docker_registry.lib.index.SQLAlchemyIndex.init() →
    SQLAlchemyIndex._setup_database/
  2. SQLAlchemyIndex._setup_database tries to get the version from the
    ‘version’ table (‘version = session.query(…).first()[0]’).
  3. The version table doesn't exist (because the DB is empty), so the
    session.query raises a sqlalchemy.exc.OperationalError.
  4. We catch the error and fallback to ‘version = None’
  5. ‘if version’ is false, so we call SQLAlchemyIndex._generate_index()
  6. SQLAlchemyIndex._generate_index iterates through
    Index._walk_storage() and adds the returned repositories.

You can add some debug logging to figure out where that's going
astray. The only dependencies are store.list_directory and
store.repositories. Perhaps docker-registry-driver-swift tweaks the
storage layout for repositories?

In any case, I think docker-registry-core's docker_registry.core.Base
UI is too tightly bound to the filesystem structure used by
docker_registry.drivers.file.Storage, and I'd be happier if there was
a more generic UI (e.g. Storage.repositories() which iterated through
available repository names). Storage drivers that used a backing
filesystem could inherit from docker_registry.drivers.file.Storage if
they wanted to pick up the current filesystem layout, and just replace
the disk-access methods with their own implementation.

@rvrignaud
Copy link
Author

@wking Thank you for the informations. It was a problem on swift-driver (see bacongobbler/docker-registry-driver-swift#14).

Closing

@dmp42 dmp42 added this to the Not on us - keeping as courtesy milestone Aug 12, 2014
@rvrignaud rvrignaud mentioned this issue Sep 2, 2014
Closed
sathlan added a commit to redhat-cip/docker-registry-driver-swift that referenced this issue Sep 17, 2014
@sathlan
Fix initial list_directory function.
d257117 
docker-registry has a local database which can be queried using a simple
REST API.  If you trash this database, it is supposed to be recreated
when you restart docker-registry.  This fix the recreation of the db which
happens thanks to results of the the list_directory function.  Previously
this was empty.

To test the problem and review the solution:

 1. assuming a working swift and docker registry with swift driver installed;
 2. push some images;
 3. stop docker-registry;
 4. check sqlalchemy_index_database in docker registry conf and "rm" the database (by default an sqlite file);
 5. restart docker-registry;

With the old code the database will be empty.  With the new one the db
will be filled with the already pushed images.  The main problem come
from the facts that:

    $ curl -i -X GET -H "X-Auth-Token: AUTH_token" 'http://localhost:8080/v1/AUTH_test/docker?prefix=lvl_1/&delimiter=/&format=json'

    [{"hash": "a83a7dcf68a3dd1d83584abd8e230829", "last_modified": "2014-09-17T15:26:25.585390", "bytes": 29, "name": "lvl_1/file_2", "content_type": "application/octet-stream"}, {"subdir": "lvl_1/lvl_2/"}]

and

    $ curl -i -X GET -H "X-Auth-Token: AUTH_token" 'http://localhost:8080/v1/AUTH_test/docker?path=lvl_1/&format=json'

    [{"hash": "a83a7dcf68a3dd1d83584abd8e230829", "last_modified": "2014-09-17T15:26:25.585390", "bytes": 29, "name": "lvl_1/file_2", "content_type": "application/octet-stream"}]

do not return the same thing.  The main difference being that the
subdirectories are not return when using the "path" syntax.

I tested with swift 1.18 and check that the Pseudo-hierarchical folders
and directories behaviour was the same on swift 2.2.0.

This would close bacongobbler#14 and bacongobbler#15.  The "fill it if not there" mechanism is
described in this issue  docker-archive/docker-registry#516
sathlan added a commit to redhat-cip/docker-registry-driver-swift that referenced this issue Sep 17, 2014
@sathlan
Fix initial list_directory function.
1b93877 
docker-registry has a local database which can be queried using a simple
REST API.  If you trash this database, it is supposed to be recreated
when you restart docker-registry.  This fix the recreation of the db which
happens thanks to results of the the list_directory function.  Previously
this was empty.

To test the problem and review the solution:

 1. assuming a working swift and docker registry with swift driver installed;
 2. push some images;
 3. stop docker-registry;
 4. check sqlalchemy_index_database in docker registry conf and "rm" the database (by default a sqlite file);
 5. restart docker-registry;

With the old code the database will be empty.  With the new one the db
will be filled with the already pushed images.  The main problem come
from the facts that:

    $ curl -i -X GET -H "X-Auth-Token: AUTH_token" 'http://localhost:8080/v1/AUTH_test/docker?prefix=lvl_1/&delimiter=/&format=json'

    [{"hash": "a83a7dcf68a3dd1d83584abd8e230829", "last_modified": "2014-09-17T15:26:25.585390", "bytes": 29, "name": "lvl_1/file_2", "content_type": "application/octet-stream"}, {"subdir": "lvl_1/lvl_2/"}]

and

    $ curl -i -X GET -H "X-Auth-Token: AUTH_token" 'http://localhost:8080/v1/AUTH_test/docker?path=lvl_1/&format=json'

    [{"hash": "a83a7dcf68a3dd1d83584abd8e230829", "last_modified": "2014-09-17T15:26:25.585390", "bytes": 29, "name": "lvl_1/file_2", "content_type": "application/octet-stream"}]

do not return the same thing.  The main difference being that the
subdirectories are not return when using the "path" syntax.

I tested with swift 1.8.0 and check that the Pseudo-hierarchical folders
and directories behaviour was the same on swift 2.2.0.

This would close bacongobbler#14 and bacongobbler#15.  The "fill it if not there" mechanism is
described in this issue  docker-archive/docker-registry#516
This was referenced Sep 17, 2014
Merged
Closed
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Not on us - keeping as courtesy
Development

No branches or pull requests
3 participants
@wking @dmp42 @rvrignaud