Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Rewrite of generate-environment-identifiers-dict.sh #1028

Open
1 of 4 tasks
molangning opened this issue May 30, 2024 · 0 comments
Open
1 of 4 tasks

Rewrite of generate-environment-identifiers-dict.sh #1028

molangning opened this issue May 30, 2024 · 0 comments
Labels

Comments

@molangning
Copy link
Contributor

molangning commented May 30, 2024

Describe the feature request:
The current script uses majestic's domain list, which may be missing a lot more domains as compared to the other lists in the checklist. Another issue that I find with it is that it makes a new (insecure) ssl session for every domain in the list, which is both insecure and inefficient.

More lists recommendations would be appreciated as these lists may be outdated.

Additional context:
You can use this command to interact with the sql server directly
psql -h crt.sh -p 5432 -U guest certwatch
https://groups.google.com/g/crtsh/c/sUmV0mBz8bQ/m/K-6Vymd_AAAJ

Domain list
https://hackertarget.com/top-million-site-list-download/
https://radar.cloudflare.com/domains
https://www.domcop.com/top-10-million-websites
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html
https://majestic.com/reports/majestic-million
https://builtwith.com/top-sites
https://tranco-list.eu/
https://statvoo.com/dl/top-1million-sites.csv.zip

Next steps:

  • Implement a script that pulls domains from domcorp, alexa, cloudflare, majestic and others
  • Dedupe the list/find better ways to extract environment ids
  • Change to use sql interface
  • I intend to open a pull request later
@molangning molangning added the enhancement Enhancement label May 30, 2024
@g0tmi1k g0tmi1k added help wanted and removed enhancement Enhancement labels Aug 12, 2024
@ItsIgnacioPortal ItsIgnacioPortal added the enhancement Enhancement label Dec 20, 2024
@ItsIgnacioPortal ItsIgnacioPortal changed the title [Feature request]: Rewrite of generate-environment-identifiers-dict.sh Rewrite of generate-environment-identifiers-dict.sh Dec 20, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants