CKAN URL Identifier

[EvilDrPurple]

Description

• Mostly waiting on Add CKAN, MuckRock source collection scrapers #106 so that the README can be updated with the identifier in mind
• Adds a CKAN URL identifier that determines whether or not each URL is hosted by CKAN

Testing

1. Checkout the branch
2. In the source_collectors/ckan folder, create a CSV file with a column labeled "url" and populate it with urls. Below is an example CSV file that may be used for testing
3. In the root project directory, create a virtual env and activate it

   python -m venv venv
   source venv/bin/activate

4. Install the requirements

   pip install -r requirements.txt

5. (Optional) Move to the ckan folder

   cd source_collectors/ckan

6. Run the program with the CSV file as the second argument

   python ckan_identifier.py urls.csv

7. Output will be located in output.csv

Sample CSV:

url
https://catalog.data.gov/dataset/operation-aquila
https://github.com/Police-Data-Accessibility-Project/data-source-identification/compare/ckan-identifier?expand=1
https://docs.ckan.org/en/2.9/api/
https://data.gov/open-gov/
https://data.boston.gov/dataset/police-districts
https://open.jacksonms.gov/dataset/police-precinct-city-of-jackson-and-wards
https://duckduckgo.com/
https://docs.pola.rs/