Add retry logic when assigning a random port

[michelleheh]

Thanks for submitting an issue to cloud_controller_ng. We are always trying to improve! To help us, please fill out the following template.

Issue

cf create-route --random-port fails with the following error message:

[2017-12-21 01:00:36.49 (UTC)]> cf create-route CATS-2-SPACE-9a5b0796-7aed-4730-5 TCP-2-DOMAIN-6a0e7696-c153-4d2c-6.apps.domain.com --random-port 
Creating route TCP-2-DOMAIN-6a0e7696-c153-4d2c-6.apps.domain.com for org CATS-2-ORG-fbb4569e-9342-49bf-5 / space CATS-2-SPACE-9a5b0796-7aed-4730-5 as CATS-USER-2-2017_12_21-01h00m23.867s...
FAILED
Server error, status code: 400, error code: 210005, message: The port is taken: Port 1055 is not available on this domain's router group. Try a different port, request an random port, or use a domain of a different router group.

Context

When running the routing-acceptance-tests in parallel, the test fails intermittently when there is a port collision.

Steps to Reproduce

Run RATS with the following configuration:

{
  "addresses": [
    "tcp.domain.com"
  ],
  "api": "api.sys.domain.com",
  "admin_user": "username",
  "admin_password": "password",
  "apps_domain": "apps.domain.com",
  "skip_ssl_validation": false,
  "use_http": true,
  "oauth": {
    "token_endpoint": "https://uaa.sys.domain.com",
    "client_name": "clientname",
    "client_secret": "clientsecret",
    "port": 443,
    "skip_ssl_validation": false
  },
  "cf_push_timeout": 300,
  "default_timeout": 120,
  "tcp_router_group": "default-tcp"
}

Expected result

The cf create-route succeeds, and is able to assign a random port.

Current result

See above for failure result. However, the failure happens inconsitently and will require many test runs.

Possible Fix

Put in retry logic when ports collide.

Michelle and @davewalter

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/154007745

The labels on this github issue will be updated when the story is started.

[davewalter]

FYI, we just ran into this issue again in our CI with CAPI 1.44 (plus patches).

[charleshansen]

This is ultimately an issue with the cloud controller, but is surprisingly hard to fix in the cloud controller. The retry logic didn't actually make the problem much better because once you have one port collision, you tend to get many port collisions. This issue probably needs to be addressed in RATS directly, where they can reduce usage of --random-port. I filed an issue in RATS for this cloudfoundry/routing-acceptance-tests#8

[ljfranklin]

@charleshansen while reducing usage of random-port in acceptance tests would make our CI happy, I think CI is rightly red in this case as the random-port feature is actually flaky during simultaneous requests. I get that retrying on port collisions isn't foolproof, but it seems like retrying X times on a collision would make this feature more reliable.

[charleshansen]

Ok, adding the retries. It doesn't help as much as you might think, but you should probably be safe at 6 nodes. 35762b9