FileFields use the same mongo/gridfs collection by default

[deignacio]

The gridfs collection_name defaults to 'fs' for all FileFields. This leads to all files being stored in the same place, regardless of Field or Document. This means that there isn't a good way to hook Document.drop_collection() to also clear the referenced files in the gridfs collection.

I propose the following:

• add a "sandbox files" argument (naming obviously not final) that ensures that the collection names for these file fields are unique.

if this is True, the following things happen:

• collection_name gets defaulted to <document name>-<field name>-fs
• runtime errors if a user attempts to override collection_names to the same thing
• drop_collection() can now safely iterate over gridfs proxied fields bubble drop_collection on those collections

if False:

• collection_name stays at default to fs
• add comments in drop collection documentation that explains that you will leak files if you drop_collection()
• log a bunch of stuff on drop_collection() invocations

I am working on a patch, but wanted to get some feedback on the idea.

[rozza]

You can provide a custom collection name on instantiation via: collection_name kwarg, I'm happy to leave it to the user to implement the clean up of files as they are stored in another collection.

Arguably drop_collection shouldn't drop another collection as well as it will lead to bugs.

I'm wary of adding extra complexity for relatively small reward. A simple flag in drop_collection(inc_grid_fs=True) might be worthwhile, however, it seems to me to be far more dangerous to drop collections where data exists through user error.

The simplest solution could be to introduce a breaking change in 0.7:

1. <collection_name> is defaulted to <document name>-fs
2. <collection_name>.drop_collection() checks for FileField in the document and raises a warning that the files haven't been deleted and to log how to drop the files, warning any other data in that collection would also be deleted.

Thoughts?

[deignacio]

I think that drop_collection dropping the other collections is fine as long as we're sure that it doesn't contain files referenced by any other collections/documents. The same way chaining deletes of a document is ~okay.

As for your proposal, I think 1) is good, probably don't need <document name>-<field name> level of granularity. Although I think adding some sort of RuntimeError if you override <collection_name> to collide w/another collection would be helpful.

Can we think of any reasons why someone would want to keep all of their gridfs files in the same collection?

I'm thinking about the consequences of having the breaking change, and wonder what the migration steps would be to prevent any dataloss.

[rozza]

Well GridFS is just a convention, so other data could be stored in the collection (stranger things have been observed) and so for people with a reason for doing that (I can't think of one that I thinks valid) shouldnt be penalised. So I'm against a RunTimeError where we could explain that in the documentation.

The cost of not dropping the collection is minimal compared to the potential damage automatically dropping it could cause.

I can understand that you may want to use your own collection for each FileField but most people probably just want a dump file store. Although having it in a different DB is more useful especially when mongoDB 2.2 comes out as you can take advantage of the improved concurrency. Or means you can symlink it off to faster disks.

This thread should probably have been on the dev list!

[rozza]

https://groups.google.com/d/topic/mongoengine-dev/ezTxUdq6Z4w/discussion

[rozza]

See MongoEngine#18