url-normalization

Here are 3 public repositories matching this topic...

adbar / courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

url crawler uri crawling tld recon url-parsing url-parser url-validation url-checker url-normalization

Updated Dec 30, 2024
Python

xojoc / cleanurl

Star

Remove clutter from URLs and return a canonicalized version

url url-normalization

Updated Jun 3, 2024
Python

A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.

rate-limiting http-requests error-handling html-parsing data-collection text-processing web-crawling content-extraction yaml-configuration data-scraping python-crawler modular-design metadata-storage url-normalization pdf-text-extraction structured-data-storage concurrent-crawling data-extraction-pipeline data-preservation-and-recovery

Updated Nov 18, 2024
Python

Improve this page

Add a description, image, and links to the url-normalization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the url-normalization topic, visit your repo's landing page and select "manage topics."

Learn more

Provide feedback

Saved searches

Use saved searches to filter your results more quickly