Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Basic usage only shows a single link, but site is full of them #469

Open
mmmint opened this issue Apr 13, 2020 · 3 comments
Open

Basic usage only shows a single link, but site is full of them #469

mmmint opened this issue Apr 13, 2020 · 3 comments
Labels

Comments

@mmmint
Copy link

mmmint commented Apr 13, 2020

While trying below code I only get a single link, but the website is full of links.

Visiting http://teenage.engineering

What is happening? Thanks for any hints.

package main

import (
	"fmt"
	"github.com/gocolly/colly"
)

func main() {

	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://teenage.engineering")
}
@asciimoo
Copy link
Member

The html content is inside a noscript tag and somehow the html parsing lib doesn't handle it. I need further investigation to fix this issue, thanks for reporting.

@asciimoo asciimoo added the bug label Apr 17, 2020
@anthonygedeon
Copy link

anthonygedeon commented Oct 28, 2021

Is this bug still reproducible?

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants