RFC: a mechanism to bypass CORS preflight

[wenbozhu]

CORS preflight introduces a significant delay even with browser-side caching, for examples for use cases when an application initiates its first interaction with a service whose response will bootstrap the client application.

To address this latency problem, we'd like to propose the following general solution:

1. Have the client encode the HTTP method via a special URL parameter agreed with the server that provides the resource or service. Use POST if the request has a body; or GET otherwise.
2. Have the client encode any non-whitelisted HTTP headers via a special URL parameter agreed with the server. The parameter value needs be URL encoded with the format being an HTTP/1.1 header block as 1*( : \r \n)

Since a server that does not understand the above URL parameters will not take any action, this approach in itself should not introduce any CORS related security vulnerability.

On the server side, the real HTTP method or any custom HTTP headers that the client is generating are expected to be set to the request object immediately before the request is dispatched to the application logic (commonly known as handlers, servlets etc). That is, the server run-time does not need be aware of this optimization.

Comments welcome, esp. on the security side. Thanks to Takeshi Yoshino for the initial review.

• Wenbo Zhu (Google, cloud networking)

[wanderview]

I think foreign fetch service workers would allow this. Would that be adequate for your use case?

[annevk]

This proposal sounds similar to something @mnot proposed. I would prefer we came up with something whereby an entire origin can identify it understands CORS and thereby bypass all future CORS preflights. Foreign fetch service workers are one such mechanism, though they might be somewhat heavyweight for services for which the server is essential.

[wanderview]

It seems to me foreign fetch is the primitive. It lets an origin implement their own preflight or not. It would be nice to see how that is used before formalizing a high level option, no?

[annevk]

I see it a little different. When you install a foreign fetch service worker, you have essentially opted into CORS and therefore CORS preflights are no longer needed (and won't be made for URLs that have an associated foreign fetch service worker). However, the service worker is needed, which makes it somewhat heavyweight.

I think it would be okay to still consider alternatives that do not require service workers, though I don't think we'd be in a rush to ship them.

[wenbozhu]

Any pointer to what @mnot proposed?

Re: an entire origin declaring CORS-aware, do we have to use a per-origin CORS "preflight" .. v.s. piggybacking such a capability with a response to a random path?

[annevk]

https://mnot.github.io/I-D/http-options-resources/ (I don't think this is worth doing, personally.)

I think if we did a origin-wide CORS preflight it would be to something like /.well-known/cors which we'd fetch before the first CORS preflight to a new origin (and cache negative for a while, positive for much longer). Not everyone is comfortable letting an origin opt into CORS fetches that contain credentials. That is the main sticking point.

[wenbozhu]

A origin-wide CORS preflight does not have to involve credentials-allowed ... which may still work under the current model, on a per request basis ...??

[annevk]

Yeah, that would be an alternative. Would that be good enough for Google? It is good enough for some other cases, but not all, and it is somewhat sad to have another special case.

[wenbozhu]

I think so. How to move forward with this? @tyoshino

[annevk]

So ideally we find at least two browsers willing to implement such a scheme. Then we need to figure out the particulars of the design and write a patch for this specification. Probably a new section "CORS-origin-preflight fetch" and a couple changes before doing the initial preflight (as long as request's credentials mode is not "include").

[mnot]

FWIW - an origin-wide preflight would be helpful (if limited), I think.

Right now, CORS preflights "punch through" reverse proxies (like Varnish) and CDNs (like Akamai) to the origin, because CORS caching is only designed for browsers; for example, there's no way to tell if a response can be reused for another client, or whether it's specific to the request it's associated with.

While ad hoc / proprietary ways to cache CORS preflights could be defined, the authoring experience would be pretty horrid (because it'd be up to which reverse proxy / CDN you use).

My draft was trying to define a generic way to handle OPTIONS caching, so that any other use of OPTIONS would also benefit. By making OPTIONS responses "normal" HTTP responses, you can use any existing HTTP caching mechanism on it (Cache-Control, Expires, Vary, Key, If-Modified-Since, invalidation mechanisms, etc.).

[annevk]

@mnot I see. I guess we really should have used CORS as verb rather than reuse OPTIONS. We could introduce an API-side switch for that too, but it's all rather brittle and would be annoying to propagate to all APIs that use CORS (although most should not trigger preflights, so maybe that would be okay).

[tyoshino]

I agree that the origin-wide preflight would be useful.

Not only Google's use case, but some more data is needed to rationalize the additional complexity, e.g. how many users of Chrome are using CORS preflight and how much latency could be reduced. We could figure out the data points using our JS metrics.

Wenbo, how much gain you'll have by this compared to the hack in the first post?

[wenbozhu]

It's a hack by all measures, with increased URL lengths for every request (subject to browser caching behavior), manual client logic to modify the URL, possible server-side issues due to HTTP method override etc ....

[wanderview]

From a gecko implementation side, we probably need to get a read on this from @sicking.

[sicking]

@mnot Could't a reverse proxy detect that the OPTIONS request is a CORS preflight by looking for OPTIONS requests that contain a Access-Control-Request-Method header and where the response contains a Access-Control-Max-Age header?

But yes, the caching rules for such requests might get pretty complex. Is that the problem?

Anyhow, I don't really have opinions regarding @wenbozhu's initial proposal vs. @mnot's OPTIONS-caching proposal. That seems more like a question for http gurus.

My experience is though that developers tend to loath having to wait for additional roundtrips, which, if I understand correctly, both those proposals suffer from.

Though likely doing additional roundtrips for preflights is probably more ok since developers try to avoid those anyway, and if they can be cached, it means that you just pay the cost once per URL.

Though we already have situations where certain REST APIs end up doubling the number of requests that they do due to always using unique URLs for each request. Tripling the number of requests would likely not be popular.

Regarding having a server-wide opt-in to credential-less requests, as @annevk talks about, I still think that would be an interesting idea.

Does anyone have data on how often people make CORS requests with credentials vs. without credentials?

[mnot]

@annevk the method name doesn't really make a difference; it's compatibility with existing HTTP caching infrastructure that's important.

@sicking - yes, that can be detected, but then we need to figure out how to cache it. Just a max-age sin't sufficient; we need to know if it can be reused for other users, how to invalidate it, etc. HTTP caching is quite well-developed in this respect, so it doesn't make much sense to reinvent the wheel.

I agree a server-wide opt-in is interesting; that draft was more of a thought experiment, and I'm happy to put it to the side for the moment. A server-wide opt-in minimises the round-trip problem (because it's once per origin) and it is a separate resource, so it works well with caches.

WRT credentials - I believe that the (very large) customers (that unfortunately I can't reveal) that I'm aware of for this will be wanting authentication.

If we can make it possible to use for credentialed requests, it also might help address w3ctag/design-reviews#76.

AIUI the issue is that giving an easy switch for credentials in a central place is a footgun. Would that be mitigated sufficiently by requiring something this in the well-known file?

credentials = "I understand that this makes all authenticated content on this site available to the sites below, and that this can represent a serious security issue."
credentials_origins = foo.com bar.com baz.com

First value being character-by-character case-sensitive. Would have to discuss whether credentials_origins would allow *.

[sicking]

I don't really think that a "I really know what I'm doing" flag is going to make a difference. It's just as easy to copy/paste. And the problem isn't really that people don't know what they are doing, it's how easy it is to do the wrong thing vs. how hard it is to do the right thing. In this case just setting the flag (the wrong thing), is dramatically easier than doing the right thing (auditing all your URL handlers perfectly).

I'm a bigger believer in enabling developers to opt in on a per-url or per-directory basis. That way developers at least have to tools to only enable credentials for the URLs that they are sure can handle it.

I'll comment regarding w3ctag/design-reviews#76 over there.

[mnot]

Yeah. For origins where there are multiple administrators/authors/apps, this is entirely sensible; it's just really annoying for single-party origins (which tend to get a lot of traffic, so this chattiness matters).

Perhaps if we found a way to limit it to them, it would be viable. Not sure how to do that, though.

In the meantime, we could spec up non-credentialed server-wide opt-in, leaving credentials to future extensions.

[wenbozhu]

Would it make sense to make the scope of opt-in a URL prefix, with the default being "/"?

[sicking]

To be clear, what I mean by "per-directory" is "per directory including subdirectories". So that would let you opt in for the whole origin using "/".

That said, there's still a lot of problems to solve.

• I don't believe (though happy to be proven wrong) that it's reasonable to add additional requests for already existing usage patterns. So we can't make existing APIs like XHR/<img>/etc make additional requests to /.well-known/whatever before at least making the initial request to the desired URL. Especially on http/1.1 servers. Probably on failure we could try to make requests to /.well-known/... though.
• The reason we didn't add per-directory policies originally was because some servers had very weird URL handling. Allowing /A/something to be fetched using /B/<stuff here>/A/something. I specifically think this affected Microsoft's IIS. Do we have reason to believe that no such server software exists today? Or at least that server admins that run those servers know about this quirk and wouldn't use the per-directory policies?
• I'm still not convinced that it's worth creating a server-side API for dealing with the withCredentials problem. I feel pretty confident that by far most CORS requests don't use cookies (and are safer because of it). The ones that need cookies have a solution, though it requires client-side changes. This doesn't seem different than that the request headers that UAs send by default is enough for most developers. And for ones that need other headers set need to use APIs which accept additional information in addition to the URL. (I've heard more requests for enabling custom headers and verbs in <img>, than requests for getting rid of .withCredentials).
• On the topic of .withCredentials, the privacy problem I mention on www-tag still exists. I don't know how we make a decision there.

All that said, I'm happy to look at a proposal if someone puts one together.

[mnot]

@sicking any preference on proposal flavour -- e.g., a diff to CORS, a diff to Fetch, a standalone spec?

[sicking]

Not any strong preference.

I do find the Fetch spec pretty hard to read since it's basically like source code and you have to step through all branch combinations to get a full understanding. So trying to do a mental diff in that is likely even harder.

I also don't know that a formal spec is needed at this stage. Just a complete proposal. I'm happy to ask if anything is unclear.

[annevk]

@sicking given that service workers with foreign fetch doesn't really protect the credentialed case at all and is also a single file that could be the result for copy-and-pasting, do we really not want to provide a convenient way to do this for the credentialed case?

[sicking]

Foreign fetch doesn't cause requests to be sent directly to the host though. It only allows triggering function calls in the service worker.

But yes, a registered serviceworker could use the foreignfetch API to route requests to the server. These requests could both contain cookies and perform other dangerous actions.
The same thing could be done by a HTML file opened in an <iframe>.

However it seems much less likely that someone will set up such a service worker, or such an HTML file, by accident and not realize what it does.

[annevk]

An HTML file openend in an <iframe> is not invoked automatically for each request to that server. You'd have to specifically target it. A foreign service worker is. They're not comparable. It will be quite easy to setup a service worker that does this and it can be instantiated through a static header.

[sicking]

Why does it matter what syntax is used to trigger requests? Isn't the concern whether an attacker can do harmful requests to the server?

[annevk]

It matters a lot because your argument for not having origin-wide CORS with credentials is that it's too simple. My argument is that it's not that much harder to do it through "Foreign Fetch".

[sicking]

It still seems to me that dropping in a misconfigured SW is more similar to dropping in a misconfigured HTML file (which we've seen very little of), than dropping misconfigured configuration file (which we saw a lot of with crossdomain.xml).

Static file vs. dynamic file is not the only thing affecting complexity/understanding.

[mnot]

@sicking - WRT additional requests, I was assuming that the first CORS preflight to an origin would be a normal one, containing an extra header, e.g., Access-Control-Site: 1. That would trigger an async (and fully optional) request to e.g,. /.well-known/cors-site. It is one additional request, but it's not on the critical path, and it saves potentially many more.

Presumably, you'd have a file at that location that contains a list of paths in some format. Let's say JSON for now. E.g.,

GET /.well-known/cors-site HTTP/1.1
Host: www.example.com
Origin: place.example
Access-Control-Request-Method: PUT

HTTP/1.1 200 OK
Content-Type: application/cors-map+json
Cache-Control: max-age=3600
Vary: Origin, Access-Control-Request-Method

{
    "/widgets": { 
        "Access-Control-Allow-Origin": "https://place.example http://foo.example.com:8888",
        "Access-Control-Allow-Credentials": "true",
        "Access-Control-Expose-Headers": "Foo, Bar"
    },
    "/public-data": { "Access-Control-Allow-Origin": "*" }
}

... with the longest/most specific matching path prefix winning (although one could imagine other approaches if the root object were an array).

The client could act as if any preflight that has a match here, the response would have the corresponding CORS headers (with the exception of Access-Control-Max-Age, since the cache ability of the well-known response itself would determine its freshness lifetime).

Note that CORS request headers are sent on the .well-known request; if the server wants this response to change based upon them, all it needs to do is send the appropriate Vary (and, eventually, Key).

Importantly, the administrator has to set the Access-Control-Allow-Credentials explicitly for a path, just as they'd do for "normal" CORS. Considering that they can already fire that footgun with .htaccess, is that a reasonable tradeoff?

I think the biggest risk here is that the file would get out of sync with the headers that the server sends. That could be mitigated somewhat if good tooling emerged (either driving headers from the central file, or vice versa). If that's not good enough, we probably should go back to a single, site-wide policy (so that it's only used on sites that are really homogenous).

BTW, a header like Access-Control-Site: 1 on normal CORS requests could also be taken as a hint to the server that it should HTTP/2 server push /.well-known/cors-site, removing some latency.

[bifurcation]

Yeah, @bifurcation got overwhelmed with notifications a while back.

@mnot - The idea of a site-wide policy seems sensible enough. Some thoughts below.

---

It's not clear to me why you want to go the .well-known route vs. having the original resource express the location of the policy:

Access-Control-Policy: https://example.com/cors-policy

In the case where the original resource is HTTPS, there's no reduction in security (since the redirect is secure). In the HTTP case, you're either hosed in any case or you're not. If you got creative, you could even make that work for multiple origins, saving you even more (as below).

---

I am also worried about the consistency issue. If we do this, we should be explicit that the site policy overrides anything sent by the specific resource. Note that it is safe for the resource to send only the policy (and not the other CORS headers), since older browsers will fail closed.

---

If we're going to have the origin set policy for sub-resources, it would be good to do it for more than just CORS. E.g., it would be nice to use this to set the upgrade-insecure-requests CSP directive for a whole site.

---

I would like this a bit better if we could have the policy be signed by the origin. You could re-use normal web certificates for this, and it would have a few advantages:

1. Better assurance in the HTTP case
2. You would have to have a certificate (if not HTTPS) to get this optimization
3. Policies can be distributed independently of the transport, so you could do things like having a single policy document have policies for multiple origins.

---

Net of all that, though, doing a foreign fetch service worker does seem simpler to me than revising the whole policy framework. Where are we on that idea? As an intermediate idea, you could imagine adding a service worker method to set this policy, so that you wouldn't have to do all the header munging.

[roryhewitt]

@bifurcation re this comment:

we should be explicit that the site policy overrides anything sent by the specific resource.

Is it possible in the policy to define a flag to specify whether policy overrides specific requests ("policy-wins") or vice-versa ("policy-loses")? Default to policy-wins.

Just thinking out loud.

[bifurcation]

@roryhewitt You could, but then you would need a way for the browser to tell when it needs to send a CORS request. Otherwise, the inclusion in the policy tells the browser that it already knows the result of the CORS request. Seems more complicated than it's worth.

[roryhewitt]

@bifurcation Of course. I was thinking of the concept of allowing generic entries

For instance, given the example format that @mnot supplied in an earlier comment:

{ "/public-data/*": { "Access-Control-Allow-Origin": "*" } "/public-data/secure": { "override-policy": "true" } }

So requests to any resource within the /public-data/ path can come from any origin except calls to /public-data/secure, which would have to make a preflight request.

[bifurcation]

@roryhewitt Again, you could. But I would strongly prefer that you do either one or the other. If you need crazy hierarchical policies, just do everything per-resource.

[sicking]

@bifurcation I still have some concerns that any syntax for allowing per-directory policies.

As I mentioned in my comment above, at least at the time when we designed CORS, there were servers that had some quite odd handling of certain characters.

The result was that when a client made requests to /dir1/<special characters>/dir2/script.cgi, the server would launch /dir2/script.cgi.

What happened was that the client interpreted the "special characters" as just any other characters, and thus as simply a directory name. However the server interpreted the "special characters" as ".." or "/".

This wasn't some obscure server, but one of the more prevalent servers at the time. Which is also why I'm reluctant to get into details about what the "special characters" or the server was in a public forum. Happy to provide more details over private emails.

It's certainly possible that this problem isn't common enough any more that we need to worry about it. After all, this feature is opt-in, so we can simply rely on server admins knowing what they are doing and not opting in to per-directory policies on affected servers.

But since this problem killed per-directory policies in CORS v1, I wanted us to at least be intentional about this decision if we're going back on it now.

Also, I'm still quite nervous about this supporting requests with credentials. It's a slightly smaller footgun than the widely-misused crossdomain.xml feature in flash. But only slightly smaller.

[annevk]

@sicking no need for secrecy: https://lists.w3.org/Archives/Public/public-appformats/2008May/0039.html.

[roryhewitt]

@annevk, do we know if this IIS bug has been fixed? Or indeed, whether MS considers it a bug? That post you reference was from 2008... It would be nice to know whether we could resurrect Access-Control-Policy-Path.

[annevk]

I don't know, I don't have the relevant contacts.

[sicking]

@annevk there were actually more ways to encode URLs than the one you link to, that would cause IIS to have a different interpretation than clients.

[mnot]

Hey @bifurcation / @rlbmoz / whoever you are today:

It's not clear to me why you want to go the .well-known route vs. having the original resource express the location of the policy:

Access-Control-Policy: https://example.com/cors-policy

A couple of things.

1. This header is going to need to be on every response, so it's going to create a lot of bloat on the wire (and no, we can't rely on H2 header compression to save the day).
2. I really want to be able to Server Push the policy to the client.

I suppose something like this would work almost as well:

Access-Control-Policy: /browser-policy

i.e., support (or even require) a relative URL. Even better:

ACP: /bp

The assumption behind a .well-known resource is that it'd be site-wide. If we allowed different resources on a site to specify different ACPs, path granularity in the policy format wouldn't be necessary. Indeed, I'm wondering if it's necessary even when the policy is site-wide.

I am also worried about the consistency issue. If we do this, we should be explicit that the site policy overrides anything sent by the specific resource. Note that it is safe for the resource to send only the policy (and not the other CORS headers), since older browsers will fail closed.

+1. It might be interesting to think about a request header indicating that the client is using the site policy, so that the server can omit them -- but we'd have to be careful about things like caching.

If we're going to have the origin set policy for sub-resources, it would be good to do it for more than just CORS. E.g., it would be nice to use this to set the upgrade-insecure-requests CSP directive for a whole site.

I've been asking for optional browser policy for years, so yes please. Happy to sketch up a more general mechanism if there's interest.

I would like this a bit better if we could have the policy be signed by the origin. You could re-use normal web certificates for this, and it would have a few advantages:

1. Better assurance in the HTTP case
2. You would have to have a certificate (if not HTTPS) to get this optimisation
3. Policies can be distributed independently of the transport, so you could do things like having a single policy document have policies for multiple origins.

You're going to make me use JOSE, aren't you?

I'd be OK with this if it were optional at least in the common case (e.g., delivered over HTTPS).

Net of all that, though, doing a foreign fetch service worker does seem simpler to me than revising the whole policy framework. Where are we on that idea? As an intermediate idea, you could imagine adding a service worker method to set this policy, so that you wouldn't have to do all the header munging.

I'd be happy prototyping something in a FFSW to gather data and prove a design, if it's ready. Is it?

I'm less happy if the endgame is that everyone who is unhappy with CORS is told to use FFSW.

[roryhewitt]

@annevk, @mnot Just wondering - is the use of a 'policy' (JSON format or otherwise) held in the /.well-known/ directory the 'official' way forward, or is there any chance of resurrecting/using Access-Control-Allow-Path or Access-Control-Policy-Path or one of their equivalents which directly returns a list of allowed paths where subsequent preflight requests are not required?

I am aware of the bug in IIS, but as I asked earlier, does this still exist, and are there others?

[annevk]

I don't think a manifest that blocks page load has been realized yet (there's only the Web App Manifest thing that I'm not sure actually solves a problem). If that proves possible we should go with that, but until that point there is room for alternatives, in my opinion.

We would need to get input from security folks, preferably those from Microsoft. I don't really know who they are unfortunately.

[roryhewitt]

Hmmm. OK, just thinking out loud...

Whilst I do like the idea of utilizing the /.well-known/ directory in the way that @mnot and others have suggested, it seems to me to be less 'intuitive' than using something like Access-Control-Allow-Paths: /api/*, /other/*. Of course, as has been discussed above, using the well-known location may make it easier to implement functionality other than just CORS, such as upgrade-insecure-policy.

I guess my feeling is that given that CORS is currently 'transparent' (the headers themselves indicate which methods/headers/origins are allowed), moving to a separate 'opaque' system, where this information is held in a file which is referenced in the header just feels wrong in some way. It may be 'better', but it feels like it's less backwards-compatible, and it's going to require another set of skills by developers. Additionally, it may have flexibility issues in an organization where different teams control API development and the web server itself.

However, whichever system is used, I don't think that the issue with differently formatted paths (the IIS canonicalization bug) needs to be a deal-breaker, as long as the browser processing explicitly only allows 'full' paths - IOW, a value of Access-Control-Allow-Paths: /api/,/api/goodapi/* allows only requests to a resource which is directly in the /api/ directory (but no subdirectories) or which is in the /api/goodapi/ directory or one of its sub-directories. For instance, if, given the above OPTIONS repsonse header, if I subsequently get a POST like this:

www.example.com/api/..\process.cgi

even though the UA has cached /api/ as a valid path, is it able to determine that the cached entry may not apply, and that therefore another preflight request should be made?

Since such a header (or a manifest in /.well-known/) will in theory be sent only once per browsing session, size isn't hugely important, is it?

[tyoshino]

Regarding usage of cross-origin credentialed requests, we have one for XHR here gathered for #251:
https://www.chromestatus.com/metrics/feature/timeline/popularity/1305

I'll summarize reqirements and approaches we can take here. I understand that it's useful if we can get general policy announcement infra, but looks we already have so much ideas. I'd like to figure out a way to start with small and simple change and evolve it later.
https://github.com/tyoshino/origin-wide-cors/blob/master/README.md

[annevk]

Start with just allowing for announcing that the origin understands CORS and therefore it's safe to omit preflight

Presumably this is simply to do and therefore it's also quite dangerous if the requests include credentials. We could e.g., add a "cors" token to HSTS that simply persists capability for the same time HTTPS is persisted. However, that does mean that all resources on the origin need to be capable of handling the diverse range of inputs.

The problem is with deciding what protection we are comfortable doing away with on the request side of the same-origin policy for an entire origin and how much such a decision needs to be protected from configuration copy-and-paste. Initially with CORS we decided that a whole lot of attention to detail was needed for that. It seems we're now leaning towards "just make it so".

@dveditz @rlbmoz thoughts?

[tyoshino]

Hmm, okay. I can live with having to specify the parameters in detail.

• Shall we require the Origin-Wide-CORS descriptor to be communicated only over a preflight response? I.e. the first non-simple request to an origin X. This would require people to implement preflight correctly.
• Shall we allow only the query part to be wildcarded? I'm checking with wenbozhu and not yet sure if it works for him. But limiting wildcard functionality would reduce risk of misconfiguration
• Shall we give the authority to specify the Origin-Wide-CORS only to "/"?

I've updated https://github.com/tyoshino/origin-wide-cors/blob/master/README.md accordingly. I've incorporated roryheiwitt's points and made it communicate the Origin-Wide settings over headers. Not involving any more separate fetching is basically good for reducing implementation complexity.

[tyoshino]

@mikewest just proposed an idea collection at WICG.

• https://mikewest.github.io/origin-policy/
• https://discourse.wicg.io/t/proposal-set-origin-wide-policies-via-a-manifest/1617
• CORS WICG/origin-policy#3

[mikewest]

Ugh. Sorry @tyoshino, I didn't realize you were working on this as well. The concept seemed to fit well with the origin-wide mechanisms I wanted to build for CSP; I apologize for hijacking the conversation without reviewing your ongoing work. :(

[tyoshino]

Hi Mike. No worry. I started working on it seriously just recently. I'll join it!

[annevk]

I know there is occasionally renewed interest for this, but I don't think it needs to be tracked here until we have a concrete idea and at least one implementer pushing it.