Proposal: Consider removing https -> https/http rewrite rules?

[semenko]

We have a small subset of rules that rewrite HTTPS sites to other HTTPS/HTTP sites. A rough grepping shows ~63 rules like this.

$ grep -F "from=\"^https" default.rulesets  | wc -l
63

I propose removing these rules, and the code needed to handle them.

Benefits:

• HTTPSe still uses nonzero resources & can slow down browsing on underpowered systems (e.g. Chromebooks). We can remove blocking all HTTPS requests in Chrome, which would be a great performance benefit.
• Reduce ruleset fragility -- rewriting broken pages to fix incorrect cert names seems really fragile
• Reduce code complexity -- We can remove some of the downgrade="1" rules and eliminate the downgrade whitelist.
• Reduce rules that probably decrease security -- I imagine some of these downgrades are ancient & aren't needed anymore (e.g. the GoogleMaps whitelist that downgrades https://maps.googleapis.com/map ?) -- users also probably don't expect us to send https -> http

Downsides:

• We may increase breakage on sites that aren't very HTTPS friendly, but many are broken for other reasons (mixedcontent). e.g. https://fellowsblog.kiva.org

Thoughts?

[semenko]

FWIW, here's a rough list of files with downgrade/https matching rules.

Aeriagames.xml
Apple.xml
Austin_Fit_Magazine.com.xml
CERN.xml
CafePress.xml
Caltech.xml
Cashback.co.uk.xml
Cheezburger.xml
Comic-Con-Intl.xml
Contextly.xml
Cru.org.xml
Dafont.com.xml
DailyDot.xml
EFF.xml
Elder_Scrolls_Online.com.xml
Epic-Systems.xml
EvoTronix.xml
Fasthosts.xml
Fourmilab.xml
GO.com.xml
Ghostery_metrics_optout.xml
GoogleMaps.xml
ImageShack.us.xml
InkFrog.com.xml
Intel.xml
Intelliworks_Chat.com.xml
Jabber.ru.xml
Kat_de_Paris.xml
Kiva.org.xml
Lenovo.xml
Levo_League.com-problematic.xml
Link-Plus-Catalog.xml
Linuxaria.com.xml
Marketwatch.com.xml
Minijob-zentrale.de.xml
MobyGames.xml
MoveOn.xml
MovieTickets.com.xml
Mutterschaftsgeld.de.xml
NYTimes.xml
National-Association-of-Theatre-Owners.xml
National_Geographic.com-problematic.xml
National_Geographic.xml
Necuhb.xml
NewsBlur.xml
Nextag-self-signed.xml
Ntop.org.xml
Oak-Ridge-National-Laboratory.xml
Pair-Networks.xml
Perfect_Audience.xml
Qz.com.xml
S-msn.com.xml
SPajIT.xml
Sandbag.xml
US-Dept-of-Veterans-Affairs.xml
University-of-Alaska-Anchorage.xml
Visa.xml
Xxxbunker.com.xml
Yandex.net.xml
Yimg.com.xml
ZeniMax-Media.xml
marktplaats.nl.xml
yWorks.com.xml

[jsha]

The proposal is to disable all rulesets that require downgrades, treating the sites as too broken for us to fix currently? I am in favor. @2d1, opinions?

Blockers:

• The EFF rule rewrites one URL on our site to another in order to detect whether HTTPS Everywhere is installed and give an appropriate promo. We need to figure another way to do this - open to ideas!
• The GoogleMaps rule needs to be updated to remove the downgrades, assuming they can be removed. Blocker to this: @StevenRoddis recently made a bunch of updates to the Google rulesets, which I rolled back for the 5.0 launch. We need to roll those forward again, along with fixes to the bugs that cause the rollback. Then we can base any GoogleMaps changes off the latest version.

I think other than that none of these rules are vital enough that we can't remove them.

[semenko]

Hm, @2d1 I think better user experience could lead to wider adoption, which is of value too.

Perhaps I'll implement a "lean" (opt-in) mode in Chrome, which drops the HTTPS listeners.

Here's a wild idea: What if this significantly increased the extension's security by allowing us to ultimately remove the <all_urls> permissions request from Chrome (which enables http/https/ftp/file/chrome-extension), and request just http://*/*?

This would require careful thought (we couldn't regain the ability to read https:// data without a one-time scary warning prompt from Chrome) -- but would significantly reduce the potentially dangerous capabilities of HTTPSe itself. We could never -- even if someone wanted to do so maliciously -- downgrade HTTPS to HTTP (or otherwise read/write/modify HTTPS data).

[jsha]

I agree that it would be nice to make a security guarantee that our extension can never rewrite content that's already on HTTPS. Note that the downgrade rules may not capture all such examples: It's possible for a rule to rewrite from HTTPS to a different HTTPS URL. @semenko could you do a quick estimate of how many such rules there are?

@2d1: I hear you about coverage vs speed. But the list of downgrade rules is quite small, relatively speaking. And I think a number of them may no longer be necessary or may be able to be modified.

Also keep in mind that since Chrome and Firefox implemented MCB, we can't actually downgrade JS or CSS, which are the most likely page-breaking subresources. So we're mostly interested in rewriting top-level navigation events where there are incorrect links to an https version that doesn't exist. @semenko, perhaps we could modify your proposal to only allow downgrades of top-level navigation? Or would that still require blocking all requests?

@diracdeltas: I'm looking at the downgrade in the Apple rule,

<rule from="^https://trailers\.apple\.com/" to="http://trailers.apple.com/" downgrade="1" />  

Do you remember writing this one? Can you tell me more about it?

[fuzzyroddis]

Could it possible be a crossdomain.xml problem?

https://trailers.apple.com
HTTP/1.1 200 OK
Server: Apache
Content-Type: text/html; charset=UTF-8
Cache-Control: max-age=844
Expires: Thu, 02 Apr 2015 23:51:35 GMT
Date: Thu, 02 Apr 2015 23:37:31 GMT
Connection: keep-alive
Set-Cookie: edgelong=-122.4195
Set-Cookie: edgelat=37.7795
Set-Cookie: edgestate=CA
Set-Cookie: edgecity=SANFRANCISCO
Set-Cookie: edgezip=94102

http://trailers.apple.com
HTTP/1.1 200 OK
Server: Apache
Content-Type: text/html; charset=UTF-8
Cache-Control: max-age=447
Expires: Thu, 02 Apr 2015 23:45:13 GMT
Date: Thu, 02 Apr 2015 23:37:46 GMT
Connection: keep-alive
Set-Cookie: edgelong=-122.4195
Set-Cookie: edgelat=37.7795
Set-Cookie: edgestate=CA
Set-Cookie: edgecity=SANFRANCISCO
Set-Cookie: edgezip=94102

They look the same. If it is a crossdomain.xml problem see the exclusion rule here https://github.com/EFForg/https-everywhere/blob/691be3b4bcb6540258ba895a037866ab48a87cca/src/chrome/content/rules/Cloudfront.xml

[jsha]

@StevenRoddis I don't think it's a crossdomain problem. The HTTPS version of trailers.apple.com simply doesn't work. The question is: Would it be sufficient to simply have no rule for that subdomain? Presumably not, or the ruleset would be written that way. But I'm not sure who would have linked to the HTTPS version.

[jsha]

FYI, if you include sideways HTTPS rewrites, i.e. rewriting from one HTTPS URL to another, we have 140 relevant rulesets:

$ grep -l 'from="^https' *.xml
Aeriagames.xml
Apple.xml
Austin_Fit_Magazine.com.xml
BBC.xml
CafePress.xml
Caltech.xml
Cashback.co.uk.xml
Cato-Institute.xml
CERN.xml
Cheezburger.xml
Comic-Con-Intl.xml
Contextly.xml
Cru.org.xml
Dafont.com.xml
DailyDot.xml
default.rulesets
EFF.xml
Elder_Scrolls_Online.com.xml
Epic-Systems.xml
Epson.xml
EvoTronix.xml
Fasthosts.xml
Fourmilab.xml
Ghostery_metrics_optout.xml
GO.com.xml
GoogleMaps.xml
ImageShack.us.xml
InkFrog.com.xml
Intelliworks_Chat.com.xml
Intel.xml
Jabber.ru.xml
Kat_de_Paris.xml
Kiva.org.xml
Lenovo.xml
Levo_League.com-problematic.xml
Link-Plus-Catalog.xml
Linuxaria.com.xml
LiveJournal.xml
Marketwatch.com.xml
marktplaats.nl.xml
Minijob-zentrale.de.xml
MobyGames.xml
MoveOn.xml
MovieTickets.com.xml
Mutterschaftsgeld.de.xml
National-Association-of-Theatre-Owners.xml
National_Geographic.com-problematic.xml
National_Geographic.xml
Necuhb.xml
NewsBlur.xml
Nextag-self-signed.xml
Ntop.org.xml
NYTimes.xml
Oak-Ridge-National-Laboratory.xml
Pair-Networks.xml
Perfect_Audience.xml
Qz.com.xml
Sandbag.xml
S-msn.com.xml
SPajIT.xml
University-of-Alaska-Anchorage.xml
US-Dept-of-Veterans-Affairs.xml
Visa.xml
Xxxbunker.com.xml
Yandex.net.xml
Yandex.xml
Yimg.com.xml
yWorks.com.xml
ZeniMax-Media.xml

[jsha]

I think the next step here should be to see if we can quantify the performance impact of checking HTTPS URLs.

We could also look at short-circuiting HTTPS URLs that couldn't possibly match. To do that, at ruleset loading time we would check whether a ruleset includes downgrade or sideways URLs, and make that an attribute of the in-memory RuleSet object. When an HTTPS URL matches a ruleset without the downgrade-or-sideways attribute, we can bail on processing without evaluating any regexes.

[semenko]

Regardless of performance thoughts, the downgrading of sites from https->http strikes me as really strange (and not something our users are expecting).

@jsha unfortunately, earlier/short circuited rules won't help a ton. We spend much of our time now in context-switching overhead (from webRequest C -> handlers for blocking javascript -> C).

You can get details via:

1. Go to chrome://extensions/
2. Click: Inspect views: background page
3. Select the "Profiles" tab and start a CPU profile -- click stop after you've browsed some sites.

The resulting profile is heavy in internal bindings (extensions::) instead of stuff we control.

For a head-to-head comparison, get a good testing machine (I use a new-ish Chromebook, which is still pretty underpowered) and try:

1. Go to chrome://extensions/
2. Click: Inspect views: background page
3. In the console, enter:

chrome.webRequest.onBeforeRequest.removeListener(onBeforeRequest)     // HTTPSe is now broken
// chrome.webRequest.onBeforeRequest.hasListeners()  // Optional, should return false
chrome.webRequest.onBeforeRequest.addListener(onBeforeRequest, {urls: ["http://*/*"]}, ["blocking"]);

To revert:

chrome.webRequest.onBeforeRequest.removeListener(onBeforeRequest)
onBeforeRequest.addListener(onBeforeRequest, {urls: ["https://*/*", "http://*/*"]}, ["blocking"]);

[jsha]

Perhaps when we do addListener, instead of listening on all HTTPS URLs, we could listen only on the ~140 hosts for which we have rewrite rules that can apply to HTTPS?

[semenko]

Ah -- good call! I tried this once with just HTTP, but it has some undocumented limit (IIRC ~1000?).

I'll give it a whirl.

[semenko]

Ok, I gave this a try with only listening to the target hosts of the https rules. Unfortunately, some wildcard usage at weird places makes this hard.

e.g. for Aeriagames.xml:

<target host="www.aeriagames.com" />
<target host="aeriagames.com" />
<target host="*.aeriastatic.com" />
....
<rule from="^https?://s\.aeriastatic\.com/"
        to="https://c.aeriastatic.com/" />

Here, we'd listen on https://www.aeriagames.com, https://aeriagames.com, https://*.aeriastatic.com/ -- works great!

However, for a few sites, we're using * in weird places.

Good examples: The VA hospitals (where we have a * in the middle of a url, like subdomain..example.com), and for Google, where use use a * as a TLD (which is really weird) like some.site.google..

I'll look into this a bit more.

[jsha]

I'm in favor of removing middle-wildcarding for target hosts, and fixing up the relevant rules. I'm guessing for most of them we can enumerate the labels that go in the middle?

I assume Chrome also disallows right-hand wildcarding? Those may be more of a hassle to get rid of, since we have more rules depending on them, and the number of hostnames they expand to is larger. However, at least the set of possible hostnames they can expand to is better-defined.

One non-intuitive thing about wildcarding: Left-hand wildcards can be arbitrarily deep, but right (and I think middle) wildcards can only go one label deep.