Refactor importing by adding write endpoints for all items

[stefandesu]

The import script is unnecessarily complicated and only usable from the command line. There should be a way to import concepts and schemes with HTTP endpoints and the importing process should become clearer.

[stefandesu]

@nichtich In a private conversation, you said that you would like to remove the --reset parameter from the import script because it is confusing when something is deleted and it might lead to accidental data loss. How about the following:

• We leave the import script's usage exactly the same as it currently is, except that --reset and any deletion of data is removed from the script (i.e. it can be called without being afraid of deleting something).
  • The question is what the script is doing with duplicates. For schemes I would suggest to override them, for concepts I would suggest to ignore them. Not sure about mappings/concordances.
• We add a separate script for deletion.
  • In particular, we need a method to delete all concepts for a particular concept scheme. This is currently not implemented as an API endpoint. I could imagine using DELETE /voc/concepts and provide the scheme URI via an URL parameter.

I'm open for changes on all of this though. On the one hand, I would like to have a consistent behavior between different type of entities, on the other hand, we need to consider the usual use cases (in particular our own use cases) which demand slightly inconsistent behavior.

[stefandesu]

We now have write endpoints for concept schemes and concepts, so those parts of the import script can be rewritten now.

For the whole import script to be rewritten, there are a bunch more things necessary though:

• Write endpoints for concordances (see Create and save into concordances, assign mappings to a concordances #98 and Support writing concordances #99)
• Bulk deletion endpoints for certain things (in particular concepts of a certain concept scheme, e.g. by adding a DELETE /voc/concepts endpoint)
• Script for using those deletion methods locally

And probably more that I haven't though of yet.

[stefandesu]

As we are breaking the import script here, but not releasing a new major version, we should try to detect whether the new import script is called with old parameters and show an error with a link to the documentation.

[stefandesu]

The POST endpoints should also support files via upload or URL. This could then be used by the rewritten import script (which would only be a simple wrapper over the internal methods) and by the future admin interface (#39).

Edit: This is now implemented via json-anystream.

[stefandesu]

So I implemented a reset script that replaces everything the --reset parameter on the old script did and more. I also replaced the old import script with the new one and adjusted the tests to use the reset script where necessary. Everything seems to work very well. There are still some open tasks though:

• Adjustments to documentation
• Bulk deletion endpoints, at least for /voc/concepts; for now, it should be fine that it's only possible to bulk delete via command line I think, so this could be a separate issue
• Check parameters for reset script (@nichtich)
  • I'll already note here that I opted for having the type as an option (-t) because otherwise it wouldn't be possible to call the script with a single URI without type. In theory, we could differentiate between type and URIs by checking whether the string conforms to a URI, but I like it the current way actually. But please take a look.
• More tests, especially for the reset script
• Separate issue: Creating concordances and importing concordances via endpoints (Create and save into concordances, assign mappings to a concordances #98, Support writing concordances #99) -> issue created
• Adjust global commands (do we even need a global jskos-import command? I would have suggested replacing jskos-import with jskos-server-import and jskos-server-reset, or to just not offer global commands)
• Import script: Import a list of mappings for a specific concordance (in case concordance and mappings need to be imported separately)
• Look at overrideAndRemoveDuplicates.js and decide whether we need that anymore (-> I really don't know why we needed that, so I just deleted it. Good thing about source code repositories is that if we do still need it, it's easy to recover.)
• Importing annotations (honestly, I simply forgot)

[stefandesu]

I have added a bunch of tests for the import and reset script, trying to test many possibilities. There's probably still potential for more, but it should be enough for now. I will finally merge import-refactor into dev tomorrow and close this issue!