Implement `$queryprune` parameter

[mtxadmin]

Prerequisites

• I verified that this is not a filter issue
  • Filter issues MUST be reported at filter issue tracker
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uBlockOrigin
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• I tried to reproduce the issue when...
  • uBlock Origin is the only extension
  • uBlock Origin with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uBlock Origin
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

In #46 about $rewrite parameter author said:

I won't implementing this filter option, I see too many issues with it. I am however open to implement a different filter option with similar purpose, but which would not suffer the issues I see with how rewrite has been designed.
...
I see a better way to implement similar option but with a more focused purpose: to remove specific query parameters from a URL:

||content.uplynk.com/ext/.m3u8?$querystrip=

Where the querystrip option would mean: "remove all query parameters matching the given lists of tokens or pattern".

Will the $querystrip parameter be realized?

A specific URL where the issue occurs

[A specific URL is MANDATORY for issue happening on a web page, even if it happens "everywhere"]

Steps to Reproduce

1. [First Step]
2. [Second Step]
3. [and so on...]

Expected behavior:

[What you expected to happen]

Actual behavior:

[What actually happened]

Your environment

• uBlock Origin version:
• Browser Name and version:
• Operating System and version:

[gorhill]

I also said:

Anyway, as said I still need more than just one case to be an argument for such filter -- the last thing I want is to add technical debt to uBO for little tangible benefits overall.

I still do not see cases being brought up to justify the feature.

[mtxadmin]

utm_* stuff (more like privacy and tracking)

Plus Aliexpress stm marketing parameters, and some others.

Yes, of course, there are some specialized URL redirecting extensions that can truncate the URL parameters. But, they are all lack subscription feature. If you want to set up blocking those trash parameters from URL on a new device, you have to install a separate extension and after then manual enter all the parameters. And after time, then you discover a new trash parameter and want to get rid of it, you have to add it on EVERY device, even those which have parents, relatives and friends. This is a way of hell to explain to inexperienced users which buttons should be clicked and which values added. Instead of adding all that to neatly autoupdated filterlist.

[liamengland1]

There are at least two cases in which the implementation of $querystrip can be used to eliminate pesky SSAI.

How $querystrip will fix this:
The video players on the below pages request files (xhr/iframe) with some ad-related parameters. If these parameters are removed, the m3u8 no longer has ads baked in server-side.

Case 1: videos on ABC-owned TV station websites (https://abcotvs.com/index.html)

Example URLS:

https://abc30.com/7285501/
https://6abc.com/7255376/
https://abc7.com/7282142/

Example from https://6abc.com/7255376/:

JSON requested:

https://content.uplynk.com/api/v3/preplay/external/10b98e7c615f43a98b180d51797e74aa/102320-wpvi-fyi4u-halloween-CC-vid.json?platform=web&ad.tag=fyi-holidays%2Challoween%2Cfyi-philly%2Cfyi-events%2Cphiladelphia%2Ccommunity&ad.vast3=1&ad.v=2&ad.tfcd=0&ad.is_lat=0&ad.npa=0&ad.correlator=49683&ad=wpvi_vod&ad.adUnit=%2Fwpvi%2F6abc.com%2Fweb%2Fcommunity&ad.pp=otv-web-desktop&ad.vid=otv-7275653&ad.description_url=https%3A%2F%2F6abc.com%2F7255376%2F&ad.sz=640x480&ad.ppid=&ad.vpi=1&accountID=10b98e7c615f43a98b180d51797e74aa&externalID=102320-wpvi-fyi4u-halloween-CC-vid&euid=_000_0_001_SF&ad.cust_params=accesslevel%3D0%26beacTyp%3Dssai%26isAuth%3D0%26aff%3Dwpvi%26lang%3Den%26ait%3Dssai%26chan%3Dmisc%26objid%3D7255376%26isAutoplay%3D1%26isDnt%3D0%26isMute%3D0%26pgtyp%3Dpost%26vdm%3Dvod%26fb_token%3D%26plt%3Ddesktop%26stp%3Dvdms%26swid%3D%26dtci_yxbk%3D%26var%3D16x9%26vps%3D640x360%26noad%3D0%26refDomain%3Dhttps%253A%252F%252F6abc.com%252F7255376%252F%26unid%3D

Response: https://gist.github.com/llacb47/b779691ae61947f3d553ab8a8c2e8b3b
If you open the playURL from that JSON in any video player, it has SSAI.

Possible querystrip filter: ||content.uplynk.com/api/v3/preplay/external/$domain=6abc.com,querystrip=/^ad\./

If the hypothetical resulting JSON is requested: https://content.uplynk.com/api/v3/preplay/external/10b98e7c615f43a98b180d51797e74aa/102320-wpvi-fyi4u-halloween-CC-vid.json?platform=web&accountID=10b98e7c615f43a98b180d51797e74aa&externalID=102320-wpvi-fyi4u-halloween-CC-vid&euid=_000_0_001_SF

Response:

{"prefix": "https://content-ause2.uplynk.com", "ads": {"breaks": [], "breakOffsets": [], "placeholderOffsets": []}, "videoView": [], "sid": "88183284ff9643bdbcd677b38be390fe", "playURL": "https://content-ause2.uplynk.com/api/v3/preplay2/712cb2a3041149c79afa3964b00021ca/447659e71fac930d104a56b5f820da5a/12UMPd9E2kAQYHf9eRchJk6JhLspdMc7aUmFVvNrTz4L.m3u8?pbs=88183284ff9643bdbcd677b38be390fe"}

By opening this playURL in any video player, it can be seen that there is no preroll.

Case 2: NBC videos

URLs:

https://www.nbc.com/late-night-with-seth-meyers/video/michael-keaton-haim/4249694
https://www.nbc.com/saturday-night-live/video/october-17-issa-rae/4246222
https://www.nbc.com/dateline/video/a-promise-to-helene/4247160

Example from https://www.nbc.com/dateline/video/a-promise-to-helene/4247160:

Page requests an iframe at this URL (probably NA geolocked)

https://player.theplatform.com/p/jujdhC/xkaAQrhkr9IU/select/media/guid/2410887629/4247160?mute=false&autoPlay=true&playbackStartPosition=0&policy=147097231&mParticleId=6082210794167379875&params=mode%3Don-demand%26uuid%3Db2372c28-9c5d-46a9-b293-866a484b2ec5%26did%3Dedb63874-27fd-1287-1a61-c829619aa2f2%26rdid%3Dedb63874-27fd-1287-1a61-c829619aa2f2%26userAgent%3DMozilla%252F5.0%2520%2528Windows%2520NT%252010.0%253B%2520rv%253A78.0%2529%2520Gecko%252F20100101%2520Firefox%252F78.0%26am_crmid%3D6082210794167379875%26am_playerv%3Dnull%26am_sdkv%3Dnull%26am_appv%3Dnull%26am_buildv%3Dnull%26am_stitcherv%3Dpoc%26uoo%3D0%26am_cpsv%3D4.0.0-2%26fw_ae%3D%26metr%3D1023%26csid%3Dnbc_tveverywhere_vod_hub%26am_extmp%3Ddefault%26am_abvrtd%3D0%26am_abtestid%3D0%26nw%3D169843%26_fw_did%3Db2372c28-9c5d-46a9-b293-866a484b2ec5%26prof%3Dnbcu_web_svp_js_https%26afid%3D136164654%26sfid%3D1676939%26policy%3D147097231%26fallbackSiteSectionId%3D9244655%26siteSectionId%3Doneapp_desktop_computer_web_ondemand%26manifest%3Dm3u%26switch%3DHLSOriginSecure%26_fw_vcid2%3D169843%3A6082210794167379875%26_fw_h_referer%3Dwww.nbc.com%26schema%3D2.0&episodetitle=A%20Promise%20to%20Helene&nbcuProfile=false&brand=NBC&show=Dateline&MVPDid=undefined#playerurl=https%3A//www.nbc.com/dateline/video/a-promise-to-helene/4247160

There are prerolls and midrolls inserted server-side. You can visit the iframe by itself to see them as well as the url provided on the NBC site.

Possible $querystrip filter: ||player.theplatform.com/p/*/select/media/guid/$subdocument,domain=nbc.com,querystrip='params'

Resulting URL:

https://player.theplatform.com/p/jujdhC/xkaAQrhkr9IU/select/media/guid/2410887629/4247160?mute=false&autoPlay=true&playbackStartPosition=0&policy=147097231&mParticleId=6082210794167379875&episodetitle=A%20Promise%20to%20Helene&nbcuProfile=false&brand=NBC&show=Dateline&MVPDid=undefined#playerurl=https%3A//www.nbc.com/dateline/video/a-promise-to-helene/4247160

No more prerolls or midrolls.

I think this is more than little tangible benefits, what do you think @gorhill ?

[gorhill]

@llacb47 Is there an extension to rewrite URLs which you can use to confirm that removing the query parameters does really remove the ads?

[mapx-]

maybe
https://addons.mozilla.org/en-US/firefox/addon/requestcontrol/

https://github.com/tumpio/requestcontrol/blob/master/_locales/en/manual.wiki#redirect-using-pattern-capturing

[liamengland1]

Yes, you can use https://einaregilsson.com/redirector/ to test. I just verified that it works as expected.

You can import the rules I used to test: https://gist.githubusercontent.com/llacb47/23a446ac1cc7763a4574a672420626fb/raw/555966752881f67859205c2ab4c579a61d8ad523/redirect-rules.json

I just found that removing parameters also gets rid of SSAI on Discovery Networks sites as well.

Test with these URLs:

https://go.discovery.com/tv-shows/growing-belushi/full-episodes/a-mission-from-god
https://www.tlc.com/tv-shows/90-day-fiance-the-other-way/full-episodes/ready-or-not
https://watch.hgtv.com/tv-shows/fixer-to-fabulous/full-episodes/dave-jennys-pick-dreary-home-gets-bright-update
https://www.sciencechannel.com/tv-shows/unearthed/full-episodes/secrets-of-the-seven-wonders
https://www.ahctv.com/tv-shows/manhunt-kill-or-capture/full-episodes/whitey-bulger-boston-mob-king

and this rule for redirector: https://gist.githubusercontent.com/llacb47/e0c78ffbb44b203da6c975796e7d6608/raw/d4ed40b979d13d6de4d130fe586cfc98383460b2/discovery-redirector.json

[liamengland1]

Any update?

[gorhill]

It's not trivial to implement with as little impact as possible on performance, so this will have to take the time it takes to implement it.

[gorhill]

It's in the latest dev build. See commit message for usage -- I do not want to provide details in release notes yet, I prefer filter list authors to experiment with usage to find out if fine tuning is necessary.

[uBlock-user]

Usage example - ||reddit.com^$queryprune=utm_, ||youtube.com^$queryprune=fbclid|gclid

[gorhill]

Avoiding queryprune from being visited at all is best, I do hope filter authors will be as carefully as possible when crafting queryprune filters as I am careful at minimizing all overhead in the code -- otherwise all the coding efforts are going to waste. So typically the query parameter of interest will be part of the filter pattern:

||reddit.com^*utm_$queryprune=|utm_
||youtube.com^*fbclid$queryprune=fbclid
||youtube.com^*gclid$queryprune=gclid

This way uBO will scan the query parameters only when the URL is found to match the targeted query parameters. Mind performance when crafting filters. Your proposed filters forces uBO to scan every URL matching reddit.com and youtube.com.

Additionally, prepending queryprune values with | when the match is of the "starts with" kind also helps.