Inventory errors when download data.json file

[FuhuXia]

Seeing errors when button "Unredacted Inventory" is clicked to download data.json file.

1. A zip file is generated but the file extension is *.json, therefore end user is not able to open the file.
2. Error message when downloading DOE data.json file

<type 'exceptions.AttributeError'> : blueprint.py : 145 : 'GroupExtra' object has no attribute 'split'

3. All datasets ends up in the errors.json when downloading GSA data.json file with same error message:

"errors": [["Invalid Required Field Value", ["The 'mediaType' field must be a string but it has a different datatype (<type 'str'>). (1 locations)"]], ["Missing Required Fields", ["The 'accessLevel' field is missing. (1 locations)", "The 'bureauCode' field is missing. (1 locations)", "The 'fn' field is missing. (1 locations)", "The 'hasEmail' field is missing. (1 locations)", "The 'identifier' field is missing. (1 locations)", "The 'modified' field is missing. (1 locations)", "The 'programCode' field is missing. (1 locations)"]]]

How to reproduce

Download json file from DOE and GSA organization.

Expected behavior

Successful zip file downloaded with data.json file in it.

Actual behavior

Multiple errors as described.

Sketch

• Diagnose blueprint.py for error mentioned above.
• Add test and code fix to ensure proper zip extension used in file name.
• Add test and code fix for "Unredacted Inventory" data.json file download.
• Add test and code fix for data.json file download when organization contains sub-agencies extra field.

[nickumia-reisys]

Is this related to a recent py3 upgrade? Was ckanext-datajson deployed to production?

These errors are probably related to the functionality added here.

[FuhuXia]

Is this related to a recent py3 upgrade? Was ckanext-datajson deployed to production?

These errors are probably related to the functionality added here.

Things were fine before the recent deployment to merge feature/py3-py2-conditional. So it should be related.

[nickumia-reisys]

ckanext-usmetadata might have the same issue because it has the same code block .... 😢

[FuhuXia]

code rolled back on production. things are back to normal.

[FuhuXia]

Will create a new issue to add tests and resolve for this scenario.
Added in the Sketch section.

[jbrown-xentity]

This might be fixed in https://github.com/GSA/ckanext-datajson/pull/104/files. Tests have been built in cypress for CKAN2.9, should consider manually testing changes and/or utilizing cypress tests: https://github.com/GSA/inventory-app/pull/359/files#diff-ab4b110143d4b67e5a8a6f90bc6d4792448cb09ba8204dd74cc575feb365fec6R41-R53

[nickumia-reisys]

1. The filename saving to *.zip is verified working based on James' last comment.
2. Not sure, the extension would have to be retested.
3. Same as 2.

[jbrown-xentity]

for 2, we would need to add multiple organizations. The top level org should have the following attributes:

• "sub-agencies": "sub-agency1,sub-agency2"

[mogul]

We are having a hard time reproducing this without data in the database. We'll work on reproducing it with a dump from production.

[jbrown-xentity]

Rollback is confirmed on production. Can download GSA data.json without issue, and can download DOE data.json without issue and it contains 400+ datasets (more than the DOE org itself, which is expected).

[jbrown-xentity]

This PR confirms Py3 is working as expected, and will be tested moving forward. Manually testing will be performed on a PR to validate CKAN2.8, pending...

[jbrown-xentity]

Testing manually on sandbox, it looks like there are a lot of issues exporting fields to DCAT-US format. Will need to investigate further before ready for current production.

[jbrown-xentity]

Documentation left for pinning datajson, ready for review.