Skip to content

canCrawl() returns incorrect result when matching middle of path #8

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Open
Trott opened this issue Feb 1, 2023 · 0 comments · May be fixed by #9
Open

canCrawl() returns incorrect result when matching middle of path #8

Trott opened this issue Feb 1, 2023 · 0 comments · May be fixed by #9

Comments

@Trott
Copy link
Contributor

Trott commented Feb 1, 2023

robots.txt:

User-agent: *
Disallow: /rss
Allow: /

canCrawl() thinks this means /home/rssa cannot be crawled but that is incorrect.
Trott added a commit to Trott/robots-txt-parser that referenced this issue Feb 1, 2023
@Trott
fix: match paths only at start
30ee13a 
Fixes: chrisakroyd#8
@Trott Trott linked a pull request Feb 1, 2023 that will close this issue
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: match paths only at start Trott/robots-txt-parser
1 participant
@Trott