Use forwarded Host header without any changes by dmage · Pull Request #14319 · openshift/origin

dmage · 2017-05-24T12:28:30Z

The Docker Server is sensitive to the presence of the port in a URL. If Docker Server is authorised to https://example.com/ and it is redirected to https://example.com:443, it might loose the Authorization header. So we should not remove nor add the port to the Host header.

Fixes bug 1439614

dmage · 2017-05-24T19:11:34Z

[test]

openshift-bot · 2017-05-24T19:13:43Z

Evaluated for origin test up to f3340fc

liggitt · 2017-05-24T19:24:37Z

 		scheme = "https"
 	case len(req.URL.Scheme) > 0:
 		scheme = req.URL.Scheme
-	case port == "443":


removing this is wrong... it means we can redirect to http when the request is expecting https

Which reverse proxy sends X-Forwarded-Port without X-Forwarded-Proto? Guessing the protocol by the port is wrong.

we're guessing it either way at this point. in the absence of proto info, using https for port 443 is more sane

liggitt · 2017-05-24T19:32:58Z

 	forwardedProto := forwarded("Proto")
 	forwardedHost := forwarded("Host")
-	// If both X-Forwarded-Host and X-Forwarded-Port are sent, use the explicit port info
-	if forwardedPort := forwarded("Port"); len(forwardedHost) > 0 && len(forwardedPort) > 0 {


removing this is just as likely to cause problems as it is to fix them. we should only omit the port when it is the default port for the chosen scheme (80 for http, 443 for https), and even then, we don't know whether the original request explicitly set the port or not

As X-Forwarded-Host has no standard at all I use RFC 7239. It says that Forwarded host should have the original value of the "Host" header field. So if we have this header, then we know. Do you know any real reverse proxy that requires this code?

PR is amended. Now X-Forwarded-Port has priority over the port in the Host header. But if there is no port in the Host header and X-Forwarded-Port is a default port, then the port is not added to the Host header.

But if there is no port in the Host header and X-Forwarded-Port is a default port, then the port is not added to the Host header.

It should only be excluded if it is a default port that matches the scheme (80 && http, 443 && https)

liggitt · 2017-05-24T19:34:23Z

 			expectedScheme: "https",
-			expectedHost:   "example.com:443",
-		},
-		"X-Forwarded-Port overwrites X-Forwarded-Host port": {


please don't remove this test case... we should not ignore an explicitly propagated port

Do you know which proxy can behave like this? For me it sounds like a terribly broken configuration.

You can change the title to "broken configuration" and change the expected values instead of deleting the test.

liggitt · 2017-05-24T20:34:07Z

 			expectedScheme: "http",
 			expectedHost:   "example.com:80",
 		},
-		"req host with tls port": {


restore this test

liggitt · 2017-05-24T20:35:28Z

 				},
 			},
 			expectedScheme: "http",
-			expectedHost:   "route-namespace.router.default.svc.cluster.local:80",


restore this... this example was captured from live data and expects the explicit port to work

I didn't change the input, it still the same as it was captured. Why this example expects the explicit port? It's the reason of the bug with the Docker Server.

based on @miminar's explanation, this was likely captured while our router had the port-stripping bug. Can you set up a pod and access it through the current haproxy router and verify what X-Forwarded-Host is set to when accessing http://route, http://route:80, https://route and https://route:443?

X-Forwarded-Host is set to the hostname without a port, as it written in the test.
If you use docker pull route:80/foo/bar, then it'll be with the port.

can you make sure we have tests capturing all the headers we see for all four of those cases?

liggitt · 2017-05-24T20:36:34Z

 				},
 			},
 			expectedScheme: "https",
+			expectedHost:   "route-namespace.router.default.svc.cluster.local",


did you capture this via an actual request to haproxy and verify haproxy differentiates between an explicit port and an implicit one in the headers it sets?

liggitt · 2017-05-24T20:36:59Z

 			expectedScheme: "https",
+			expectedHost:   "route-namespace.router.default.svc.cluster.local",
+		},
+		"haproxy edge terminated route -> svc -> non-tls pod with the explicit port": {


same question for this testcase... was this verified with actual backend capture through haproxy?

openshift-bot · 2017-05-24T20:46:04Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/1712/) (Base Commit: 16e8fe4)

mfojtik · 2017-05-25T14:36:45Z

@miminar FYI can you have a look?

miminar · 2017-05-31T11:47:04Z

The changes seem reasonable to me. The upstream reverted support for X-Forwarded-Port because it broke some users behind ELB. I'm in favor of making us consistent with the upstream.

As far as I know, we had a problem with our router that used to strip custom port from the Host header. This has been fixed.

I'd like to spin up a fork_ami and get this tested behind both ELB, haproxy and f5 routers/balancers.

liggitt · 2017-05-31T13:41:52Z

 		host = req.URL.Host
 	}

-	port := ""


restore this

liggitt · 2017-05-31T13:47:26Z

dropping support for X-Forwarded-Port is ok with me if we can prove we preserve the user's request when accessing via the haproxy router, for both http and https protocols, with both implicit and explicit ports.

mfojtik · 2017-06-22T19:26:09Z

@liggitt @dmage we seems to be getting more and more reports about this in bugzilla, we know that this was fixed in the latest docker (not available in rhel/centos yet), but can we do something to fix this for users not able to upgrade?

i'm fine with documenting this as known issue as a fix.

https://bugzilla.redhat.com/show_bug.cgi?id=1464188

liggitt · 2017-06-29T05:12:19Z

#14319 (comment) stands...

dropping support for X-Forwarded-Port is ok with me
we should test and capture http and https requests with and without explicit ports as seen by a pod via our current haproxy router, and add/update our testcases with the inputs and expected outputs (Use forwarded Host header without any changes #14319 (comment))
restore the scheme-detection that uses https if no scheme is indicated and the port is 443 (Use forwarded Host header without any changes #14319 (comment))

liggitt · 2017-08-02T14:08:12Z

-	default:
-		scheme = "http"
+	// If X-Forwarded-Port is sent, use the explicit port info
+	if len(forwardedPort) > 0 && (len(port) != 0 || len(scheme) == 0 && forwardedPort != "80" && forwardedPort != "443") {


why is the scheme used to decide whether to use the port info? this is a confusing combination of conditions

To support this weird configuration:

X-Forwarded-Host: example.com X-Forwarded-Port: 80 X-Forwarded-Proto: https

with that configuration, this logic leaves the port as "", which would redirect to https://example.com, which implies port 443, which is wrong... we should send https://example.com:80 in that case.

stepping back, can we just remember whether the original host included an explicit port, and if it did not, insert a final step that strips default ports that match the scheme? I think all the existing logic can stay as-is in that case.

host := "" hostHadExplicitPort := false switch { case len(forwardedHost) > 0: host = forwardedHost // remember whether the X-Forwarded-Host header had an explicit port hostHadExplicitPort = hasExplicitPort(forwarded("Host")) case len(req.Host) > 0: host = req.Host hostHadExplicitPort = hasExplicitPort(host) case len(req.URL.Host) > 0: host = req.URL.Host hostHadExplicitPort = hasExplicitPort(host) } ... // Do not include a port if the original host did not explicitly set it, // and the port is a default port that matches the scheme if !hostHadExplicitPort { if (scheme == "https" && port == "443") || ("scheme" == "http" && port == "80") { if hostWithoutPort, _, err := net.SplitHostPort(host); err == nil { host = hostWithoutPort } } } return scheme, host

OK, amended this way.

liggitt · 2017-08-02T14:08:57Z

-		}
+	forwardedPort := forwarded("Port")
+
+	scheme := ""


I'd prefer to set the scheme in a single block, rather than setting it here, then using the scheme to determine the port, then using the port to determine the scheme

Me too, but you want determine the scheme by port, and I want to not add the port if it's a default port for the scheme.

liggitt · 2017-08-02T18:46:28Z

 	}

+	if !hostHadExplicitPort {
+		if scheme == "https" && port == "443" || scheme == "http" && port == "80" {


add parens for clarity

liggitt · 2017-08-02T19:11:35Z

+	// If both X-Forwarded-Host and X-Forwarded-Port are sent, use the explicit port info
+	if forwardedPort := forwarded("Port"); len(forwardedHost) > 0 && len(forwardedPort) > 0 {
+		if h, _, err := net.SplitHostPort(forwardedHost); err == nil {
+			host = net.JoinHostPort(h, forwardedPort)


it's not great to reset host from forwardedHost and forwardedPort after the switch block where we set host from the different sources. would prefer all the logic be contained in the different cases of that block.

hasExplicitHost := func(h string) bool { _, _, err := net.SplitHostPort(h) return err == nil } host := "" hostHadExplicitPort := false switch { case len(forwardedHost) > 0: host = forwardedHost hostHadExplicitPort = hasExplicitHost(host) // If both X-Forwarded-Host and X-Forwarded-Port are sent, use the explicit port info if forwardedPort := forwarded("Port"); len(forwardedPort) > 0 { if h, _, err := net.SplitHostPort(forwardedHost); err == nil { host = net.JoinHostPort(h, forwardedPort) } else { host = net.JoinHostPort(forwardedHost, forwardedPort) } } case len(req.Host) > 0: host = req.Host hostHadExplicitPort = hasExplicitHost(host) case len(req.URL.Host) > 0: host = req.URL.Host hostHadExplicitPort = hasExplicitHost(host) }

soltysh · 2017-08-03T15:13:23Z

/unassign
/assign @liggitt @miminar

miminar · 2017-08-14T11:07:54Z

/test extended_image_registry

miminar · 2017-08-14T14:57:09Z

Extended registry tests seem to be broken now because of #15763

0xmichalis · 2017-08-17T12:11:51Z

/test extended_templates

liggitt · 2017-08-24T02:43:50Z

would like to see this merged... there are two outstanding comments (https://github.com/openshift/origin/pull/14319/files#r130969215 and https://github.com/openshift/origin/pull/14319/files#r130962641) and then it LGTM

Fixes bug 1439614

dmage · 2017-09-01T14:25:55Z

@liggitt PTAL, I've addressed these comments.

openshift-ci-robot · 2017-09-01T14:33:57Z

@dmage: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/extended_image_registry	`f021f55`	link	`/test extended_image_registry`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dmage · 2017-09-01T14:49:52Z

/test cmd

liggitt · 2017-09-01T15:25:11Z

/lgtm

openshift-merge-robot · 2017-09-01T15:25:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dmage, liggitt

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these OWNERS Files:

~~pkg/util/OWNERS~~ [liggitt]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

openshift-merge-robot · 2017-09-02T02:02:25Z

Automatic merge from submit-queue (batch tested with PRs 16052, 16105, 14319, 16106, 16104)

kevin-tla · 2019-06-17T15:46:34Z

I am still experiencing this issue. Openshift is appending a port number to my route when I click on it to open it in the browser. Manually removing the port solves the problem.

Followed the trail of issues regarding the port stripping or leaving it alone and this issue seems to be the latest one.

Version:
oc v3.11.0+8de5c34
kubernetes v1.10.0+d4cacc0

dmage requested a review from liggitt May 24, 2017 12:28

dmage force-pushed the fwd-host branch from 27b5425 to f3340fc Compare May 24, 2017 12:34

dmage added the component/imageregistry label May 24, 2017

liggitt reviewed May 24, 2017

View reviewed changes

liggitt reviewed May 31, 2017

View reviewed changes

Comment thread pkg/util/httprequest/httprequest.go

host = req.URL.Host

}

port := ""

liggitt May 31, 2017

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restore this

openshift-merge-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 24, 2017

openshift-merge-robot assigned mfojtik and soltysh Jul 25, 2017

openshift-merge-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 28, 2017

dmage force-pushed the fwd-host branch from f3340fc to f235c84 Compare August 2, 2017 13:24

liggitt reviewed Aug 2, 2017

View reviewed changes

dmage force-pushed the fwd-host branch from f235c84 to 82516c1 Compare August 2, 2017 15:52

openshift-merge-robot removed the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 2, 2017

dmage force-pushed the fwd-host branch from 82516c1 to f021f55 Compare August 2, 2017 15:53

openshift-merge-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 2, 2017

liggitt reviewed Aug 2, 2017

View reviewed changes

openshift-ci-robot assigned liggitt and miminar and unassigned soltysh Aug 3, 2017

liggitt mentioned this pull request Aug 24, 2017

Improve basic auth handling for canonical ports #15945

Merged

Use forwarded Host header without any changes

3d6c78d

Fixes bug 1439614

dmage force-pushed the fwd-host branch from f021f55 to 3d6c78d Compare September 1, 2017 14:17

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 1, 2017

openshift-merge-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 1, 2017

openshift-merge-robot merged commit e2efede into openshift:master Sep 2, 2017

kevin-tla unassigned liggitt Jun 17, 2019

Uh oh!

Conversation

dmage commented May 24, 2017

Uh oh!

dmage commented May 24, 2017

Uh oh!

openshift-bot commented May 24, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liggitt May 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liggitt Aug 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-bot commented May 24, 2017

Uh oh!

mfojtik commented May 25, 2017

Uh oh!

miminar commented May 31, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liggitt commented May 31, 2017

Uh oh!

mfojtik commented Jun 22, 2017

Uh oh!

liggitt commented Jun 29, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liggitt Aug 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmage Aug 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liggitt May 31, 2017 •

edited

Loading

liggitt Aug 2, 2017 •

edited

Loading

liggitt Aug 2, 2017 •

edited

Loading

dmage Aug 2, 2017 •

edited

Loading

dmage commented Sep 1, 2017 •

edited

Loading

openshift-ci-robot commented Sep 1, 2017 •

edited

Loading