Test CA repository does not reflect Firefox state

[fuglede]

PR #6102 gives an example of a host which whose SSL configuration will cause breakage in Firefox (due to an invalid chain) but which passes the fetch test.

cc @Hainish.

@jeremyn, @J0WI: You may want to be aware of this when reviewing; I just came across a second false positive.

[J0WI]

When you visited a site with the same cert chain before, the browser caches the chain and doesn't show any errors.
As far as I know we integrated some cert chains in our test infrastructure that tests don't fail. We may reconsider this.

[J0WI]

They are located in "test/rules/platform_certs/", but they might only be used on rules with platform CAcert or incomplete (do we have a platform for that?) specified.

[jeremyn]

This is a useful link for anyone trying to understand this problem: https://wiki.mozilla.org/CA:FAQ .

[fuglede]

@J0WI: The ones in test/rules/platform_certs/default are exactly the ones that fetch-test.sh and friends use, so those are the ones that would appear to be out of sync.

[jeremyn]

Considering @J0WI 's "We may reconsider this.":

My rule of thumb has been that a rewrite works if the https and http version of a page do the same thing: return the same content, have the same 4xx error, whatever. If a site is misconfigured to not correctly fetch an intermediate certificate, the two versions don't do the same thing and the rule is bad. So by this rule of thumb, I don't think we should include prefetched intermediate certificates in our tests.

[Hainish]

Looking into this now

[Hainish]

I wonder how this list of intermediates was generated in the first place.

The Mozilla CA Certificate Store lists this source file with all of the included root certificates. We could possibly parse this file and generate a new test/rules/platform_certs/defaults that is more current.

@jsha, do you know how this was generated in the first place?

[fuglede]

Pinging @hiviah as he added them in the first place. Having an automated fetch-and-parser sounds like it would be great.

[jsha]

There are two questions:

• How do we add new roots when they are added to Mozilla's root program? (we don't, or do it ad-hoc)
• How do we add new intermediates?

We include intermediates because some sites fail to include those intermediates, which will break curl, but not necessarily Firefox (since Firefox has different certificate caching logic). I'm not sure how @hiviah generated the original set, but for similar work in the SSL Observatory we've, in the past, generated the transitive closure of all intermediates from trusted roots. To keep the list up-to-date, I've occasionally used the ad-hoc script in test/rules/platform_certs/get-issuer.sh.

Perhaps a better long-term solution would be to use the public CT logs. Not guaranteed to be comprehensive, but given that Google pushes certs from their crawls into those logs, most likely all the intermediates we care about would show up there.

[fuglede]

We include intermediates because some sites fail to include those intermediates, which will break curl, but not necessarily Firefox (since Firefox has different certificate caching logic). I'm not sure how @hiviah generated the original set, but for similar work in the SSL Observatory we've, in the past, generated the transitive closure of all intermediates from trusted roots. To keep the list up-to-date, I've occasionally used the ad-hoc script in test/rules/platform_certs/get-issuer.sh.

I'm actually a bit surprised by this part, as I've typically been able to use curl to catch faulty Firefox behavior. Do we have an example of a host which will break curl but not Firefox?

[jsha]

I don't have one off-hand, but the case where it happens is when the site doesn't provide an intermediate and the Firefox user has previously visited another site that did provide the same intermediate, so the intermediate is cached by Firefox.

You could make an argument against including such sites, because they will break when visited in a completely fresh FF profile, but so far we have included them.

[fuglede]

Ah, right, then we're on the same note. But actually, I've avoided including such rulesets exactly because they potentially cause issues; in fact #3107 was more or less motivated by this very phenomenon.

[Hainish]

I'd also air on the side of not including hosts which do not provide their intermediates, since this would cause us to force users with a fresh FF profile on to a site they can't access. If these sites are submitted in a PR and we spot this, it may also be worth it to try to reach out to the site maintainers to notify them of the problem.

[hiviah]

I generated the original certificate sets from Firefox exported roots and generating transitive closure - with this script.

The intermediates were taken from my own observatories (Perspectives notary fork storing complete chains - notary1.qabs.cz/notary2.qabs.cz, CA certs have separate table which makes export easy)

Using CT would certainly work as well, but you'd have to sieve the certs for intermediate certs.