Add support for deposit to the DSpace REST API #135

markpatton · 2024-11-15T15:16:30Z

A new deposit option is added which uses the DSpace REST API.

Also:

Cleaned up DSpace configuration properties
Small fix to the duplicate JSONObject warnings when running tests.

DSpace deposit takes three calls and tries to handle resumption after an error in the last two calls.

Operational changes:

Can now use repository key DSpace to use the new direct deposit
Will need to set additional DSPACE_ variables. See documentation pr below.
The metadata set is slightly different. Need to review on stage.

Required prs:

Related prs:

Update for DSpace env variables pass-documentation#39

Test by building the pass-core, pass-ui, and pass-support prs. Then using the pass-docker pr, start up PASS with dspace support. Finally attempt to do some deposits and verify that they work. You should notice that publication date is now required.

Notes on interacting directly with the DSpace REST API:

Generate a CSRF token. Note that this endpoint returns 404.

CSRF_TOKEN=`curl -I http://localhost:9000/server/api/security/csrf -w '%header{DSPACE-XSRF-TOKEN}' -o /dev/null`

Login and get the auth token

AUTH_TOKEN=`curl -v -X POST http://localhost:9000/server/api/authn/# --data "user=xxx&password=yyy" -H "X-XSRF-TOKEN: $CSRF_TOKEN" -b "DSPACE-XSRF-COOKIE=$CSRF_TOKEN" -w '%header{Authorization}'`

Create a workspace item

export COLLECTION=8d80c978-5cf3-46bb-a5ea-dd8425f1af2f

curl -v -X POST "http://localhost:9000/server/api/submission/workspaceitems?owningCollection=$COLLECTION" --form 'file=@/home/msp/Downloads/test.pdf' --form 'file=@/home/msp/Downloads/test2.pdf' -H "Authorization: $AUTH_TOKEN" -H "X-XSRF-TOKEN: $CSRF_TOKEN" -b "DSPACE-XSRF-COOKIE=$CSRF_TOKEN"

export WSI_ID=49

See workspace items

curl -v "http://localhost:9000/server/api/submission/workspaceitems/" -H "Authorization: Bearer $AUTH_TOKEN"

Set required metadata

curl -v -X PATCH "http://localhost:9000/server/api/submission/workspaceitems/$WSI_ID" -H 'Content-Type: application/json' --data-binary "@/home/msp/work/pass/pass-support/md.patch" -H "Authorization: $AUTH_TOKEN" -H "X-XSRF-TOKEN: $CSRF_TOKEN" -b "DSPACE-XSRF-COOKIE=$CSRF_TOKEN"

Create a workflow item for workspace item

 curl -v -X POST http://localhost:9000/server/api/workflow/workflowitems -H "Content-Type:text/uri-list" --data "http://localhost:9000/server/api/submission/workspaceitems/$WSI_ID" -H "Authorization: $AUTH_TOKEN" -H "X-XSRF-TOKEN: $CSRF_TOKEN" -b "DSPACE-XSRF-COOKIE=$CSRF_TOKEN"

Example of patch

[
    {
        "op": "add",
        "path": "/sections/teste/dc.title",
        "value": [
            {
                "authority": null,
                "confidence": -1,
                "display": "AROLDO TEST WITH REST",
                "language": null,
                "otherInformation": null,
                "place": 0,
                "value": "AROLDO TEST WITH REST"
            }
        ]
    },
{
        "op": "add",
        "path": "/sections/license/granted",
        "value": "true"
    }
]





[
    {
        "op": "add",
        "path": "/sections/teste/dc.title",
        "value": [
            {
                "authority": null,
                "confidence": -1,
                "display": "AROLDO TEST WITH REST",
                "language": null,
                "otherInformation": null,
                "place": 0,
                "value": "AROLDO TEST WITH REST"
            }
        ]
    },
{
        "op": "add",
        "path": "/sections/license/granted",
        "value": "true"
    }
]

… checked

…guration properties.

rpoet-jh

Nice job, Mark! Just a few comments, but in general it looks good. I still need to test and review the other PRs, but this is a first pass at the review for this main PR.

rpoet-jh · 2025-02-05T19:59:29Z

...posit-core/src/main/java/org/eclipse/pass/deposit/provider/j10p/DspaceMetadataDomWriter.java

-        throw new RuntimeException("SourceMD with id '" + id + "' not found.");
-    }
-
-    private SourceMD createSourceMd() throws METSException {


Do the method deletions in this class affect the sword deposit for dspace? I know we are going to deprecate the sword deposit eventually, but should these changes be done at that time?

These were unused methods. It wasn't really meaningful to delete them. But I did it during another cleanup.

...services/deposit-core/src/main/java/org/eclipse/pass/deposit/provider/util/CitationUtil.java

rpoet-jh · 2025-02-05T20:03:17Z

...eposit-services/deposit-core/src/main/java/org/eclipse/pass/deposit/service/DepositTask.java

-                    deposit.setDepositStatusRef(packageStream.metadata().packageDepositStatusRef());
+
+                    // Only set deposit status ref if not already set during transport
+                    if (deposit.getDepositStatusRef() == null) {


What about the retry case? Is it possible deposit.getDepositStatusRef() has a value, then after retry it needs to be updated?

rpoet-jh · 2025-02-05T20:07:56Z

...deposit-core/src/main/java/org/eclipse/pass/deposit/support/dspace/DSpaceDepositService.java

+
+    public boolean verifyConnectivity() {
+        URI uri = URI.create(dspaceApiUrl);
+        return repositoryConnectivityService.verifyConnect(uri.getHost(), uri.getPort());


Probably want to use repositoryConnectivityService .verifyConnectByURL(dspaceApiUrl ).

I'm not sure if that url would work. I think it would give a 404.

I sort of see this as making sure the host is running and can be connected to on a port. The problem is that dspaceApiUrl is not a valid service call. It's the base for other service calls. This is what I was writing and then I did some testing. It turns out that, completely undocumented, that it is in fact a service endpoint and returns links to the other service endpoints. There is actually an health endpoint that could be used, but just the api service endpoint seems fine.

the api service endpoint seems fine -> yes, better to use this to cover cases where response code is > 500, like 502 Bad Gateway when the server is unreachable but there is a proxy in front of it.

rpoet-jh · 2025-02-05T20:23:09Z

...deposit-core/src/main/java/org/eclipse/pass/deposit/support/dspace/DSpaceDepositService.java

+            .header(AUTHORIZATION, authContext.authToken())
+            .header(X_XSRF_TOKEN, authContext.xsrfToken())
+            .header(COOKIE, DSPACE_XSRF_COOKIE + authContext.xsrfToken())
+            .body(body)


Do you know what the spring RestClient does with a post body like this, will it stream the contents of the Resources to the server? I ask for concern of memory for large files that risk of OOM exceptions?

I believe it will stream, but I will poke around.

The body is composed of resources and the default is to stream, not buffer. I tested with some larger files and I think it is ok.

...ices/deposit-core/src/main/java/org/eclipse/pass/deposit/transport/dspace/DSpaceSession.java

...deposit-core/src/main/java/org/eclipse/pass/deposit/support/dspace/DSpaceDepositService.java

rpoet-jh · 2025-02-05T20:46:32Z

...ices/deposit-core/src/main/java/org/eclipse/pass/deposit/transport/dspace/DSpaceSession.java

+
+            AuthContext authContext = dspaceDepositService.authenticate();
+
+            // Create WorkspaceItem if it does not already exist


Style comment: this method a bit long. The logic in this method could be split up into maybe three methods like createWorkspace, updateMetadata, and createWorkflow. These methods can then be called from this one.

rpoet-jh · 2025-02-07T19:45:15Z

...ices/deposit-core/src/main/java/org/eclipse/pass/deposit/transport/dspace/DSpaceSession.java

+        try {
+            DepositSubmission depositSubmission = packageStream.getDepositSubmission();
+
+            LOG.info("Processing Dspace Deposit for Submission: {}", depositSubmission.getId());


Should change to LOG.warn so it shows up in default deployed logging level.

This seems a bit odd to be at level warn instead of info. As in this isn't a warning. This is normal operation.

rpoet-jh · 2025-02-07T19:45:31Z

...ices/deposit-core/src/main/java/org/eclipse/pass/deposit/transport/dspace/DSpaceSession.java

+            String itemUuid = workspaceItemContext.read("$._embedded.workspaceitems[0]._embedded.item.uuid");
+            String accessUrl = dspaceDepositService.createAccessUrlFromItemUuid(itemUuid);
+
+            LOG.info("Completed DSpace Deposit for Submission: {}, accessUrl: {}",


Should change to LOG.warn so it shows up in default deployed logging level.

…ake the connection verification for DSpace more robust.

sonarqubecloud · 2025-02-07T20:04:12Z

Quality Gate passed

Issues
25 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.1% Duplication on New Code

See analysis details on SonarQube Cloud

markpatton marked this pull request as draft November 15, 2024 15:17

markpatton linked an issue Nov 15, 2024 that may be closed by this pull request

Add support for deposit using the DSpace REST API eclipse-pass/main#1063

Open

markpatton force-pushed the 1063-dspace-rest-deposit branch 2 times, most recently from fec504f to d4a366b Compare November 22, 2024 18:32

markpatton force-pushed the 1063-dspace-rest-deposit branch from 6ce178a to d750107 Compare December 16, 2024 14:56

markpatton force-pushed the 1063-dspace-rest-deposit branch from 5bf461e to c84a802 Compare January 2, 2025 18:16

markpatton marked this pull request as ready for review January 3, 2025 15:35

markpatton added 4 commits February 4, 2025 08:59

Fix JSONObject duplicate in classpath during tests

0b184e5

When looking up deposit support for a Repository, prefer the key

0bd7f68

Make sure that file resources have a filename so the extension can be…

5b259ff

… checked

Add support for deposit with the DSpace REST API. Adjust DSpace confi…

d34fdf5

…guration properties.

markpatton force-pushed the 1063-dspace-rest-deposit branch from 94a9f14 to d34fdf5 Compare February 4, 2025 14:47

markpatton added 2 commits February 5, 2025 10:16

Fix DSpace abstract metadata being in wrong submission form section

f87a2cc

Tighten check for missing DSpace metadata in workspace item

ffb3340

markpatton requested a review from rpoet-jh February 5, 2025 16:14

rpoet-jh reviewed Feb 5, 2025

View reviewed changes

Fix copyright statements and remove incorrect author comment

fe653d1

rpoet-jh reviewed Feb 7, 2025

View reviewed changes

Make DSpace deposit service only load if relevant property defined. M…

c252319

…ake the connection verification for DSpace more robust.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for deposit to the DSpace REST API #135

Add support for deposit to the DSpace REST API #135

markpatton commented Nov 15, 2024 •

edited

Loading

rpoet-jh left a comment

rpoet-jh Feb 5, 2025

markpatton Feb 6, 2025

rpoet-jh Feb 5, 2025

rpoet-jh Feb 5, 2025

markpatton Feb 6, 2025

markpatton Feb 7, 2025

rpoet-jh Feb 7, 2025

markpatton Feb 7, 2025

rpoet-jh Feb 5, 2025

markpatton Feb 6, 2025

markpatton Feb 7, 2025

rpoet-jh Feb 5, 2025

rpoet-jh Feb 7, 2025

markpatton Feb 7, 2025

rpoet-jh Feb 7, 2025

sonarqubecloud bot commented Feb 7, 2025


		AuthContext authContext = dspaceDepositService.authenticate();

		// Create WorkspaceItem if it does not already exist

Add support for deposit to the DSpace REST API #135

Are you sure you want to change the base?

Add support for deposit to the DSpace REST API #135

Conversation

markpatton commented Nov 15, 2024 • edited Loading

Notes on interacting directly with the DSpace REST API:

rpoet-jh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 7, 2025

Quality Gate passed

markpatton commented Nov 15, 2024 •

edited

Loading