Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

f/2840 add durable download linlk #45

Merged
merged 2 commits into from
Jun 9, 2023
Merged

f/2840 add durable download linlk #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
67480c5
add durable download linl
sansth1010 Jun 9, 2023
6be5e6c
update tests
sansth1010 Jun 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion example-app/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions src/helpers/enrich-dataset.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ describe('enrichDataset function', () => {
name: 'DCAT_Test',
description: 'Some Description',
source: 'Test Source',
id: '0_0',
id: '123a_0',
type: 'Feature Layer',
url: 'https://services1.arcgis.com/jUJYIo9tSA7EHvfZ/arcgis/rest/services/DCAT_Test/FeatureServer/0',
layer: {
Expand All @@ -55,7 +55,7 @@ describe('enrichDataset function', () => {
name: 'DCAT_Test',
description: 'Some Description',
source: 'Test Source',
id: '0_0',
id: '123a_0',
type: 'Feature Layer',
url: 'https://services1.arcgis.com/jUJYIo9tSA7EHvfZ/arcgis/rest/services/DCAT_Test/FeatureServer/0',
layer: { geometryType: 'esriGeometryPolygon' },
Expand All @@ -70,7 +70,10 @@ describe('enrichDataset function', () => {
provenance: '',
hubLandingPage: 'https://arcgis.com/maps/CALFIRE::DCAT_Test',
downloadLink: 'https://arcgis.com/datasets/CALFIRE::DCAT_Test',
agoLandingPage: 'portal.arcgis.com/home/item.html?id=0&sublayer=0',
durableUrlCSV: 'https://arcgis.com/api/download/v1/item/123a/csv?layers=0',
Copy link
Contributor

@sonofflynn89 sonofflynn89 Jun 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally get that you were just copying what existed, but I got really tripped up by the fact that these urls use arcgis.com instead of hub.arcgis.com (or my-hub-site.com, etc.). This is mainly because the download APIs are provided by us, not ArcGIS Online. Would you mind updating for consistency?

durableUrlGeoJSON: 'https://arcgis.com/api/download/v1/item/123a/geojson?layers=0',
durableUrlShapeFile: 'https://arcgis.com/api/download/v1/item/123a/shapefile?layers=0',
agoLandingPage: 'portal.arcgis.com/home/item.html?id=123a&sublayer=0',
isLayer: true,
license: '',
accessUrlGeoJSON: 'https://arcgis.com/datasets/CALFIRE::DCAT_Test.geojson?outSR=%7B%22latestWkid%22%3A3310%2C%22wkid%22%3A3310%7D',
Expand Down
10 changes: 9 additions & 1 deletion src/helpers/enrich-dataset.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ export type HubSite = {
orgBaseUrl: string,
orgTitle: string
};

type FileType = 'shapefile' | 'csv' | 'geojson';
/**
* Mapping from ElasticSearch geo_shape type
* to valid GeoJSON type
Expand Down Expand Up @@ -79,10 +79,13 @@ export function enrichDataset(dataset: HubDataset, hubsite: HubSite): Feature {

if (isLayer(dataset)) {
additionalFields.accessUrlGeoJSON = downloadLinkFor('geojson');
additionalFields.durableUrlGeoJSON = generateDurableDownloadUrl(dataset.id, siteUrl, 'geojson');
Copy link
Contributor

@sonofflynn89 sonofflynn89 Jun 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, do we want to even keep the old links then if we are planning on introducing these durable ones?

Copy link
Collaborator Author

@sansth1010 sansth1010 Jun 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now yes. Durable links will stay experimental until we finalize the new download service which hasn't been released to production yet and these durable links will only be used for testing hub feeds on DEV and QA as required by the issue 2840. Production hub feeds will be using old links for now.

Once, download service gets released to production, these old links will be removed from the provider.

additionalFields.accessUrlCSV = downloadLinkFor('csv');
additionalFields.durableUrlCSV = generateDurableDownloadUrl(dataset.id, siteUrl, 'csv');
if (_.has(dataset, 'layer.geometryType')) {
additionalFields.accessUrlKML = downloadLinkFor('kml');
additionalFields.accessUrlShapeFile = downloadLinkFor('zip');
additionalFields.durableUrlShapeFile= generateDurableDownloadUrl(dataset.id, siteUrl, 'shapefile');
}
}

Expand All @@ -100,6 +103,11 @@ export function enrichDataset(dataset: HubDataset, hubsite: HubSite): Feature {
});
};

function generateDurableDownloadUrl(datasetId: string, siteUrl: string, fileType: FileType) {
const { itemId, layerId } = parseDatasetId(datasetId);
return `https://${siteUrl}/api/download/v1/item/${itemId}/${fileType}?layers=${layerId}`;
}

function getDatasetKeyword(dataset: HubDataset): string[] {
const metaKeyword = _.get(dataset, 'metadata.metadata.dataIdInfo.searchKeys.keyword');

Expand Down