KeyError: '/status_404'

[MaheswarReddy1194321]

-> it's giving error if we are cloning website which have error page.
becaue

def add_scheme(url):
if url[-1] == '/':
url = url.strip('/')
if yarl.URL(url).scheme:
new_url = yarl.URL(url)
err_url = yarl.URL(url + '/status_404')
else:
new_url = yarl.URL('http://' + url)
err_url = yarl.URL('http://' + url + '/status_404')
return new_url, err_url

in the above function from coner.py we are directly adding error page whithout checking whether the website has error page.
kindly, let me know if there is any mistake in my comment.

[afeena]

@MaheswarReddy1194321 sorry for the late reply

I can't reproduce your problem :( But I know we had some problems with a cloner, so it doesn't work stable all the time. You should check you meta.json and try probably to re-clone again

[afeena]

Maybe it's somehow related #215

[afeena]

And this #183

[lordlabuckdas]

As per my observation, this issue arises in 2 cases:

1. The target website does not have an error page.
2. Upon visiting a page that does not exist, we are redirected to the homepage.

The website in question (used by @MaheswarReddy1194321) comes under the 2nd case as in.yahoo.com/status_404 redirects to in.yahoo.com/?err=404&err_url=https%3a%2f%2fin.yahoo.com%2fstatus_404.