[BUG] CORS Error When Loading Comments on TikTok

[Russell-Newton]

Context:

• Playwright Version: 1.33.0
• Operating System: Confirmed on Windows and Linux (both WSL and Docker, TikTok behaves differently across both)
• Python version: Confirmed on 3.8 and 3.10
• Browser: Confirmed on Chromium and Firefox
• Extras: Has been an issue with TikTok since January 2023

Steps to Reproduce

I've found that the easiest way to confirm the faulty behavior is with a headed browser so that you can check the console and network tabs of the inspector. You can run the following code in a terminal:

from playwright.sync_api import sync_playwright
p = sync_playwright.start()
browser = p.chromium.launch(headless=False)
browser.new_page()

And then navigate to any video on TikTok (e.g.: https://www.tiktok.com/@tiktok/video/6584647400055377158)

To be honest I'm not sure what the best pure-code way to test this would be because you need to read the network requests to see what's going on with a specific endpoint.

Describe the bug

I noticed this when trying to resolve issues in my own repository (related reports: Russell-Newton/TikTokPy#9, Russell-Newton/TikTokPy#23, Russell-Newton/TikTokPy#30).

When Playwright attempts to execute the API call that TikTok makes to grab comments on a video, it makes a request to a API endpoint that looks something like https://us.tiktok.com/api/comment/list/. This resembles the endpoints of all other API requests that Playwright has to make in order for TikTok to behave normally, but for some reason this one always sends back a CORS error.

The console always says something like:

Access to fetch at 'https://us.tiktok.com/api/comment/list/?...' from origin 'https://www.tiktok.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
init.js?cache:1

GET https://us.tiktok.com/api/comment/list/?... net::ERR_FAILED 200

In previous versions of Playwright (1.29.x), this was only an issue on Chromium, but now it seems to be an issue for Firefox too.

I would expect that this API call should work, considering it works on my regular desktop browser, and all the other API calls work.

---

All of my knowledge and experience with web development and this sort of stuff is self-taught and limited, and I've tried everything I know how to do, so I'm left to assume this is a bug with Playwright. If there's any sort of further information needed, I'll do my best to get it.

[dgozman]

I tried to reproduce this in Google Chrome, without Playwright involved. Navigating to the url above, I see a network request https://us.tiktok.com/api/comment/list/?... that has the same error as you describe:

Access to fetch at 'https://us.tiktok.com/api/comment/list/?...' from origin 'https://www.tiktok.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

This only happens the first time I open the page in Incognito window. After reload, everything works fine. I guess that's how tiktok's page works?

Since Playwright always loads your page in the fresh context, it would have the same issue. Try reloading the page with page.reload() - perhaps that would help?

[Russell-Newton]

Reloading with page.reload() and with the refresh button both don't seem to work. I've never had the issue with my regular Chrome, Firefox, Opera, etc. installations.

I've also tried messing with User-Agent information, and that doesn't seem to resolve the problem.

I suspect the reason why it didn't work for you the first time was that you didn't have the msToken cookie when the comment API request was made (it sends the value into the parameter string of the URL). From my tinkering, not having that cookie's value makes API requests impossible, so reloading the page retried the call (TikTok serves you the cookie when you load a page, it seems like the timing of it might be a bit wonky). You can notice this by checking the request headers before and after a refresh. Before the refresh, the msToken header seems to always be empty, but after the refresh it's populated.

Having that cookie in Playwright doesn't seem to change the result though.

[Russell-Newton]

It's also interesting to note, Playwright's Chromium inspection window shows that the comment request doesn't have a Remote Address field, but all the other requests do. On my regular browsers, that field is always populated. It's possible that that field gets populated on a successful request, so it might not matter.

[dgozman]

@Russell-Newton I was able to repro the issue, and I see the msToken not being sent as well. I tried to bisect what is causing this, and was not able to find anything network or cookie-specific that would cause this behavior. Most likely it's the navigator.webdriver property that is set when Playwright automates the browser, and this page probably tries to avoid being automated.

Given the above, I don't think there is a bug in Playwright that we would be fixing, so I'll close the issue.

[Russell-Newton]

That points me in hopefully the right direction. Thanks!