fake user agent for IMDB scraping #223
Conversation
|
@pkscout |
I'll find some time to take a closer look this weekend. It's possible the header has some extra stuff in it from other web calls, and that might be causing IMDB to continue blocking it. I had that problem with the TV show scraper, so I had to make sure to blank out the header and then create a new one. |
|
@KarellenX try the updated fix. I think I stuck the call to set the header in the wrong place the fist time. I also had to clean up how and when headers were set, as we were sending a bunch of header info to every site that didn't relate to those sites, and IMDB could use that to block us again. This one worked on my test setup, at least for the dozen or so movies I tested. |
|
@pkscout IMDB ratings were scraped in the handful of movies I tested with. |
|
Great. I've added @rmrector as a requested reviewer. Hopefully he'll be by soon to merge this and then get the update pushed out. |
This should fix IMDB scraping by faking a valid user agent heading. I have tested this with the TV show scraper, and it works fine. I haven't had a chance to test this with movies, so someone should validate that this works and doesn't break some other part of the scraping. Movie scraping is more straight forward, so this shouldn't break anything.