Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add ytInitialPlayerResponse checker to let user easily know problem #3027

Merged
merged 1 commit into from
Jul 25, 2024
Merged

Add ytInitialPlayerResponse checker to let user easily know problem #3027

merged 1 commit into from
Jul 25, 2024

Conversation

DGideas
Copy link
Contributor

@DGideas DGideas commented Jul 25, 2024

Today I have tried use you-get to download video from YouTube. I noticed one of my hosts from OVH Cloud FR failed to download the example video:

(venv) ubuntu@fr1:/tmp$ you-get "https://www.youtube.com/watch?v=jNQXAC9IVRw" --debug
[DEBUG] Extracting from the video page...
[DEBUG] get_content: https://www.youtube.com/watch?v=jNQXAC9IVRw
[DEBUG] Retrieving the player code...
[DEBUG] get_content: https://www.youtube.com/s/player/1f8742dc/player_ias.vflset/fr_FR/base.js
[DEBUG] Loading ytInitialPlayerResponse...
you-get: version 0.4.1718, a tiny downloader that scrapes the web.
you-get: Namespace(version=False, help=False, info=False, url=False, json=False, no_merge=False, no_caption=False, postfix=False, prefix=None, force=False, skip_existing_file_size_check=False, format=None, output_filename=None, output_dir='.', player=None, cookies=None, timeout=600, debug=True, input_file=None, password=None, playlist=False, first=None, last=None, size=None, auto_rename=False, insecure=False, http_proxy=None, extractor_proxy=None, no_proxy=False, socks_proxy=None, stream=None, itag=None, m3u8=False, URL=['https://www.youtube.com/watch?v=jNQXAC9IVRw'])
Traceback (most recent call last):
  File "/home/ubuntu/venv/bin/you-get", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/__main__.py", line 92, in main
    main(**kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1883, in main
    script_main(any_download, any_download_playlist, **kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1772, in script_main
    download_main(
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1386, in download_main
    download(url, **kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/common.py", line 1874, in any_download
    m.download(url, **kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/extractor.py", line 48, in download_by_url
    self.prepare(**kwargs)
  File "/home/ubuntu/venv/lib/python3.12/site-packages/you_get/extractors/youtube.py", line 207, in prepare
    self.title = ytInitialPlayerResponse["videoDetails"]["title"]
                 ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'videoDetails'

In advanced investigation, I noticed that the returned ytInitialPlayerResponse which raised exception have the following format:

{
    "responseContext": {...
    },
    "playabilityStatus": {...
    },
    "trackingParams": "CAAQu2kiEwiylZzn6sGHAxXVxkkHHVxgAe4=",
    "adBreakHeartbeatParams": "Q0FBJTNE",
    "frameworkUpdates": {...
    },
}

Which a normal ytInitialPlayerResponse from my another host should look like this:

{
    "responseContext": {...
    },
    "playabilityStatus": {...
    },
    "streamingData": {...
    },
    "playbackTracking": {...
    },
    "captions": {...
    },
    "videoDetails": {...
    },
    "annotations": {...
    },
    "playerConfig": {...
    },
    "storyboards": {...
    },
    "microformat": {...
    },
    "cards": {...
    },
    "trackingParams": "CAAQu2kiEwiylZzn6sGHAxXVxkkHHVxgAe4=",
    "attestation": {...
    },
    "videoQualityPromoSupportedRenderers": {...
    },
    "messages": {...
    },
    "adBreakHeartbeatParams": "Q0FBJTNE",
    "cpnInfo": {...
    },
    "frameworkUpdates": {...
    },
}

The content inner playabilityStatus has the answer why ytInitialPlayerResponse can not return video details:

    "playabilityStatus": {
        "status": "LOGIN_REQUIRED",
        "reason": "Connectez-vous pour confirmer que vous n'\u00eates pas un robot",
        ...
    }

I submit this commit to raise AssertionError to users when returned status is not okay, this error message helps user know about what happened.

After this commit, user could get a better error message like this:

(env) ubuntu@fr1:/tmp/you-get$ ./you-get "https://www.youtube.com/watch?v=jNQXAC9IVRw" --debug
/tmp/you-get/src/you_get/extractors/youtube.py:198: SyntaxWarning: invalid escape sequence '\W'
  if re.search('\Wlist=', self.url) and not kwargs.get('playlist'):
/tmp/you-get/src/you_get/extractors/youtube.py:206: SyntaxWarning: invalid escape sequence '\.'
  jsUrl = re.search('([^"]*/base\.js)"', video_page).group(1)
/tmp/you-get/src/you_get/extractors/youtube.py:214: SyntaxWarning: invalid escape sequence '\s'
  ytInitialPlayerResponse = json.loads(re.search('ytInitialPlayerResponse\s*=\s*([^\n]+?});(\n|</script>|var )', video_page).group(1))
[DEBUG] Extracting from the video page...
[DEBUG] get_content: https://www.youtube.com/watch?v=jNQXAC9IVRw
[DEBUG] Retrieving the player code...
[DEBUG] get_content: https://www.youtube.com/s/player/3400486c/player_ias.vflset/fr_FR/base.js
[DEBUG] Loading ytInitialPlayerResponse...
you-get: version 0.4.1718, a tiny downloader that scrapes the web.
you-get: Namespace(version=False, help=False, info=False, url=False, json=False, no_merge=False, no_caption=False, postfix=False, prefix=None, force=False, skip_existing_file_size_check=False, format=None, output_filename=None, output_dir='.', player=None, cookies=None, timeout=600, debug=True, input_file=None, password=None, playlist=False, first=None, last=None, size=None, auto_rename=False, insecure=False, http_proxy=None, extractor_proxy=None, no_proxy=False, socks_proxy=None, stream=None, itag=None, m3u8=False, URL=['https://www.youtube.com/watch?v=jNQXAC9IVRw'])
Traceback (most recent call last):
  File "/tmp/you-get/./you-get", line 11, in <module>
    you_get.main(repo_path=_filepath)
  File "/tmp/you-get/src/you_get/__main__.py", line 92, in main
    main(**kwargs)
  File "/tmp/you-get/src/you_get/common.py", line 1883, in main
    script_main(any_download, any_download_playlist, **kwargs)
  File "/tmp/you-get/src/you_get/common.py", line 1772, in script_main
    download_main(
  File "/tmp/you-get/src/you_get/common.py", line 1386, in download_main
    download(url, **kwargs)
  File "/tmp/you-get/src/you_get/common.py", line 1874, in any_download
    m.download(url, **kwargs)
  File "/tmp/you-get/src/you_get/extractor.py", line 48, in download_by_url
    self.prepare(**kwargs)
  File "/tmp/you-get/src/you_get/extractors/youtube.py", line 215, in prepare
    self.check_playability_response(ytInitialPlayerResponse)
  File "/tmp/you-get/src/you_get/extractors/youtube.py", line 184, in check_playability_response
    raise AssertionError(
AssertionError: Server refused to provide video details. Returned status: LOGIN_REQUIRED, reason: Connectez-vous pour confirmer que vous n'êtes pas un robot.

@soimort soimort merged commit 4ab0221 into soimort:develop Jul 25, 2024
9 checks passed
@soimort
Copy link
Owner

soimort commented Jul 25, 2024

Nice. Thanks

@DGideas DGideas deleted the patch-1 branch July 26, 2024 02:01
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants