steamcommunity.com: breakage

[growf]

OS: macOS Sonoma 14.5 (23F79)
Browser: Google Chrome 126.0.6478.127 (Official Build) (arm64)
Extension: uBlock Origin Lite 2024.7.17.853

Following a link from a Discord server to https://steamcommunity.com/id/customurlasd/myworkshopfiles/?appid=293760 is blocked despite being a genuine page on the Steam Community Workshop. uBOL blocks the link unless the filtering mode for steamcommunity.com is set to "No filtering".

The URL isn't blocked if cut-and-pasted directly into the browser's URL bar; the block only occurs when following a link from a Discord server. I believe it is being caught up in the attempt to block fraudulent links used in Discord Nitro scams.

Issue can be worked-around by cut-and-pasting the URL in the URL bar or by disabling filtering for steamcommunity.com.

Edit: Issue appears not to be restricted to Discord links; even the link in this comment results in the page being blocked.

[MasterKia]

#155?

[growf]

#155 doesn't have the specific replication steps that this issue does; this ticket is specifically related to Discord links to steamcommunity.com

Additionally, I am not running a second ad-blocker.

[growf]

Huh. Even the link to steamcommunity.com from this ticket's description is blocked. I'll update the description to note that.

[gorhill]

Issue can be worked-around by cut-and-pasting the URL in the URL bar or by disabling filtering for steamcommunity.com

Or just forcing a reload of the page by clicking "Reload" button in browser error page.

Will have to figure out which DNR rule is causing this.

[stephenhawk8054]

@gorhill There's a regex filter in badware related to it

/^https:\/\/stea[a-z]{1,4}o[mn][a-z]{4,7}y[a-z]?\.com\//$doc,domain=com|~steamcommunity.com

Could it cause issue because DNR does not support ~ yet?

[gorhill]

DNR does support ~, the rule is here using excludedInitiatorDomains: https://github.com/uBlockOrigin/uBOL-home/blob/main/chromium/rulesets/regex/default.json#L99. It could be though that the initiator is evaluated differently in DNR.

Note that I cannot reproduce when compiling the extension locally, so it seems a filter which was removed is causing this. In the network pane, it says initiator is other.

[stephenhawk8054]

Hmm... even when I modify the link to just https://steamcommunity.com in the devtools, it's still blocked after clicking. But weird that the document when reloading is fine.

[gorhill]

Weird, I cannot reproduce with locally-compiled and sideloaded MV3 in Chromium, but can reproduce with locally-compiled and sideloaded MV3 in Firefox.

I can reproduce when sideloading both latest release for Chromium and Firefox.

Will need a way to report triggered rules at the console to figure out which rule is causing the issue.

[gorhill]

Turns out my sideloaded Chromium version was suffering from an unrelated bug related to entering all-urls in the "No filtering" field, preventing me from reproducing locally in Chromium. Removing the all-urls from the "No filtering" field did not cause the rule which disables blocking everywhere by default to be removed. This will have to be fixed separately.

Now I can reproduce locally in Chromium, and the reported rule which is causing the issue is:

[uBOL] https://github.com => https://steamcommunity.com/id/customurlasd/myworkshopfiles/?appid=293760
	[{"action":{"type":"block"},"condition":{"excludedInitiatorDomains":["steamcommunity.com"],"initiatorDomains":["com"],"regexFilter":"^https:\\/\\/stea[a-z]{1,4}o[mn][a-z]{4,7}y[a-z]?\\.com\\/","resourceTypes":["main_frame"]},"id":1000071}]

So the issue is that the initiator as seen by the DNR API is github.com.

[stephenhawk8054]

Hmm... DNR initiator is quite different than what I expected. I thought it means the context of the page. Didn't know it means the page before that.

[stephenhawk8054]

I'll use !env_mv3 as a temp solution

[gorhill]

I think using to=com|~steamcommunity.com instead of domain=com|~steamcommunity.com should work.

[stephenhawk8054]

Hmm... I'm still not clear about DNR domain. For example there's a filter in badware list like this

/\/[0-9a-f]{32}\/maq\/$/$doc,match-case,domain=com|top

If a user visits

https://example.org

then click on a link that is

https://example.com/11111111111111111111111111111111/maq/

so would the maq link be blocked? The domain here will be referred to https://example.com or https://example.org?

[gorhill]

Since the filter is targeted at document requests, using to= make more sense in that case since you want to block the request regardless of its initiator -- this will make the filter work properly in uBOL Lite while still working properly in uBO.

---

so would the maq link be blocked?

Yes, the link will be blocked when using to=com|top.

[stephenhawk8054]

Hmm... Currently there are a lot of filters with domain and doc, so are they all potentially having issues in uBOL?

---

Also there's a weird situation I'm getting, related to doc and domain while I was testing with the above example.org: currently this link is blocked properly in uBO: https://example.com/c/a?s1=1 due to the filter

.com/c/*?s1=$document,domain=com

But when I open it in uBOL, it's not blocked, although the filter is compiled to DNR normally

{"action":{"type":"block"},"condition":{"initiatorDomains":["com"],"resourceTypes":["main_frame"],"urlFilter":".com/c/*?s1="},"id":10772}

I found that when I modified the link "More information" in https://example.org to https://example.com/11111111111111111111111111111111/maq/ and clicked on it, the link was not blocked. Then I just loaded the whole link in the URL bar and it's not blocked either.

[gorhill]

But when I open it in uBOL, it's not blocked

Yes, because the initiator can be something else than what domain= specify, hence the issue here. When clicking the link at the top, initiator is github.com, hence matching domain=com|~steamcommunity.com, hence causing the unwanted block. If using to=com|~steamcommunity.com, then the destination document is what will be matched, hence steamcommunity.com, which will not match to=com|~steamcommunity.com. When loading the URL directly in the address bar, the initiator is undefined, hence not matching domain=com|~steamcommunity.com.

When using to= for filter using document, initiator is disregarded and instead the hostname of the document URL is what is matched, so you will always get the wanted result. In uBO the issue does not exist with domain= because uBO always set the initiator to the top document navigated to.

---

Currently there are a lot of filters with domain and doc, so are they all potentially having issues in uBOL?

Difficult to say given that it's arising as per DNR API internals -- we should just fix on a per-report basis to avoid filter regression.

[stephenhawk8054]

Thanks. It's more complicated than I expected now that I think about the filters which have both domain and to.

cc @Yuki2718 , these may be useful for badware list.

[gorhill]

Just to be clear, domain= is still needed for filters targeting resources other than document.

[stephenhawk8054]

Yeah I understand. The only tricky ones are document and maybe all? There are some filters like

||fitgirl-repacks.*^$all,domain=~fitgirl-repacks.site,to=~fitgirl-repacks.site

that I'm not actually sure how it will perform in uBOL.

[gorhill]

not actually sure how it will perform in uBOL

If you build uBOL from the console, the lists will be downloaded and cached in dist/build/mv3-data and once cached, a list will not be dowloaded again so you can add a filter in dist/build/mv3-data/badware.min.txt for example, then if you build again using make mv3-[chromium|firefox], your filter will make it into the DNR rules of your local build so you can test the result.

[Yuki2718]

I guess we need to change a lot of $domain to $to then. Not only Badware, I have to change Japanese filter too (confirmed the behavior is the same for AGMV3).

[stephenhawk8054]

Confirm switching from doc,domain= to doc,to= works.

cc @Yuki2718 We can gradually switch to this format

[gorhill]

Fixed in latest stable release https://github.com/uBlockOrigin/uBOL-home/releases/tag/uBOLite_2024.8.5.925.

[stephenhawk8054]

@gorhill Hmm... About domain and to issue, is removeparam affected the same? Many of the filters in "AdGuard URL Tracking Protection" is like this:

.com/*/status/$removeparam=t,domain=twitter.com|x.com

Should they be changed to .com/*/status/$removeparam=t,to=twitter.com|x.com ?

[gorhill]

Yes.

I get expected result when using ?appid=$removeparam=appid,to=example.com and either clicking or copy-pasting into address bar https://example.com/?appid=toto, and unexpected result when using ?appid=$removeparam=appid,domain=example.com.

[gorhill]

I could internally convert domain= to to= in uBOL for removeparam option. Can't do this for removeparam since the option can be used for other resources than document.

Same for csp=.

[stephenhawk8054]

@gorhill I think converting domain to to for csp is not a good idea. There are filters that target iframes in a specific site, so converting domain to to would cause incorrect targets for these filters, for example:

@@||recaptcha.net^$frame,csp=worker-src 'none',domain=lewd.ninja

[Yuki2718]

AGMV3 doesn't remove either. I'll open an issue.