O+M 02-03-2025

[btylerburton]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• Check Production State/Actions

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• Check auto generated O&M tickets from no status column
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• DB-Solr Sync
• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/change tracking update from nightly job to weekly #4345])
• Check Catalog Solr
• Catalog Dupe Check
• Check user management requests
• Check Snyk Dashboard and cleanup

Monthly Checklist

• Invicti Scan

ad-hoc checklist

• audit/review applications on cloud foundry and determine what can be stopped and/or deleted.

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[btylerburton]

A lot of errors in the latest tracking update run:
https://github.com/GSA/catalog.data.gov/actions/runs/13173948920/job/36769144756

Ex.

2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR 2025-02-06 08:01:35,111 ERROR [ckanext.spatial.search] Geometry not valid JSON Expecting value: line 1 column 1 (char 0), not indexing :: United States, state, county, outlying area, region
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT 2025-02-06 08:01:35,111 ERROR [ckanext.spatial.search] Geometry not valid JSON Expecting value: line 1 column 1 (char 0), not indexing :: United States, state, county, outlying area, region
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR 2025-02-06 08:01:35,111 ERROR [ckanext.geodatagov] Error while indexing dataset e34b5fba-a6e4-4ee5-b4ca-be0997034f1c: AttributeError("'NoneType' object has no attribute 'get'")
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT 2025-02-06 08:01:35,111 ERROR [ckanext.geodatagov] Error while indexing dataset e34b5fba-a6e4-4ee5-b4ca-be0997034f1c: AttributeError("'NoneType' object has no attribute 'get'")
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT 2025-02-06 08:01:35,114 ERROR [ckanext.geodatagov] Traceback (most recent call last):
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-1317394[8920](https://github.com/GSA/catalog.data.gov/actions/runs/13173948920/job/36769144756#step:6:8921)-1/0] OUT File "/home/vcap/deps/1/python/lib/python3.10/site-packages/ckanext/geodatagov/cli.py", line 794, in update_tracking_solr
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT package_index.update_dict(
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT File "/home/vcap/deps/1/python/lib/python3.10/site-packages/ckan/lib/search/index.py", line 105, in update_dict
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT self.index_package(pkg_dict, defer_commit)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT File "/home/vcap/deps/1/python/lib/python3.10/site-packages/ckan/lib/search/index.py", line 284, in index_package
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT pkg_dict = item.before_dataset_index(pkg_dict)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT File "/home/vcap/deps/1/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR 2025-02-06 08:01:35,114 ERROR [ckanext.geodatagov] Traceback (most recent call last):
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT File "/home/vcap/deps/1/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 57, in index_dataset
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT shape = self.shape_from_geometry(geometry)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR File "/home/vcap/deps/1/python/lib/python3.10/site-packages/ckanext/geodatagov/cli.py", line 794, in update_tracking_solr
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT File "/home/vcap/deps/1/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR package_index.update_dict(
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT shape = shapely.geometry.shape(geometry)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR File "/home/vcap/deps/1/python/lib/python3.10/site-packages/ckan/lib/search/index.py", line 105, in update_dict
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT File "/home/vcap/deps/1/python/lib/python3.10/site-packages/shapely/geometry/geo.py", line 92, in shape
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT geom_type = ob.get("type").lower()
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR self.index_package(pkg_dict, defer_commit)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT AttributeError: 'NoneType' object has no attribute 'get'
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR File "/home/vcap/deps/1/python/lib/python3.10/site-packages/ckan/lib/search/index.py", line 284, in index_package
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR pkg_dict = item.before_dataset_index(pkg_dict)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR File "/home/vcap/deps/1/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] OUT 2025-02-06 08:01:35,114 INFO  [ckanext.geodatagov] Indexing dataset 4779/17343: 5336534c-c451-4104-a5c8-1ebbb69b81cc
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR File "/home/vcap/deps/1/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 57, in index_dataset
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR shape = self.shape_from_geometry(geometry)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR File "/home/vcap/deps/1/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR shape = shapely.geometry.shape(geometry)
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR File "/home/vcap/deps/1/python/lib/python3.10/site-packages/shapely/geometry/geo.py", line 92, in shape
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR geom_type = ob.get("type").lower()
2025-02-06T08:01:35.11+0000 [APP/TASK/ckangeodatagovtracking-update-13173948920-1/0] ERR AttributeError: 'NoneType' object has no attribute 'get'