[CF1] tunnel connectivity troubleshooting guide - #26582
Conversation
|
This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:
|
| Key points: | ||
|
|
||
| - Cloudflare Tunnel requires outbound connectivity to `region1.v2.argotunnel.com` and `region2.v2.argotunnel.com` (or to the equivalent `us-region1` and `us-region2` endpoints when using the US region). | ||
| - For basic connectivity, `cloudflared` only needs **one reachable IP in one region**. |
There was a problem hiding this comment.
I don't think we want to say it like this, even though it is true. We need customers to benefit from high availability and built-in resilience of the product.
So for the purpose of the troubleshooting guide customers can test against one IP in one region, but to have a setup that is working, resilient and supported by us, they must ensure cloudflared connectivity to both regions.
There was a problem hiding this comment.
@nikitacano agreed, this was just notes leftover from our meeting, not final draft content
|
|
||
| Cloudflare Tunnel requires outbound connectivity to `region1.v2.argotunnel.com` and `region2.v2.argotunnel.com` (or to the equivalent `us-region1` and `us-region2` endpoints when using only the US region). | ||
|
|
||
| For basic connectivity, `cloudflared` only needs one reachable IP in one region. For high availability and [four active replicas](/cloudflare-one/networks/connectors/cloudflare-tunnel/configure-tunnels/tunnel-availability/), Cloudflare recommends allowing connectivity to both regions (that is, both `region1.v2.argotunnel.com` and `region2.v2.argotunnel.com` or both `us-region1` and `us-region2`). |
There was a problem hiding this comment.
Same here: we absolutely require customers to have connectivity to both regions in order to consider their deployment supported.
|
@nikitacano updated the PR with a full first draft. added questions for you marked as (question) within the doc. |
| To resolve: | ||
|
|
||
| - Configure the host machine to use `1.1.1.1` as its resolver. | ||
| - If you must keep using your existing resolver, (question) then what? |
There was a problem hiding this comment.
Then investigate with your system administrator and / or ISP provider why your local resolver is returning a different response. Recursive DNS resolvers should return the same response as the authoritative DNS server since the authoritative server is always the source of truth where each hostname should point to.
There was a problem hiding this comment.
If you cannot solve this step, it means you have a problem with your local env, which is not related to CF and your local env needs to be debugged. Fix problem w/ ISP, admin, etc, because your existing resolver should resolve these hostnames.
|
|
||
| To resolve: | ||
|
|
||
| - Check for firewall rules blocking (question) what? 1.1.1.1.? Endpoints? ports? |
There was a problem hiding this comment.
... blocking DNS traffic altogether (UDP on port 53) or specific DNS queries related to Cloudflare.
|
|
||
| You have already confirmed DNS is working in the previous steps. These connectivity tests now verify whether your environment allows traffic to Cloudflare on port `7844`. By default, `cloudflared` automatically falls back to whichever protocol is available. | ||
|
|
||
| If a protocol is blocked but you force `cloudflared` to use it (for example, forcing QUIC when UDP is blocked), the tunnel will fail to connect. |
There was a problem hiding this comment.
|
|
||
| `cloudflared` will only be able to connect using `quic`. If you force `http2` in your configuration while TCP is blocked, the tunnel will fail. | ||
|
|
||
| To resolve: (question) |
There was a problem hiding this comment.
Either allow TCP on your local network firewall on port 7844 or stop forcing http2 to allow cloudflared to connect over QUIC instead.
|
|
||
| `cloudflared` will only be able to connect using `http2`. If you force `quic` while UDP is blocked, the tunnel will fail. | ||
|
|
||
| To resolve: (question) |
There was a problem hiding this comment.
Either allow UDP on local network firewall on port 7844 or stop forcing QUIC to allow cloudflared to connect over HTTP/2 instead.
|
|
||
| This usually indicates a firewall policy or upstream security control that does not allow outbound traffic (or return traffic) on port `7844`. | ||
|
|
||
| To resolve: (question) |
There was a problem hiding this comment.
Allow all traffic over port 7844 on local network firewall and, if that doesn't help, troubleshoot this with your ISP or service provider.
There was a problem hiding this comment.
The point of failure could be your firewall configuration on the ISP, cloud provider, router, or host machine. Debugging is required.
|
|
||
| To resolve: (question) | ||
|
|
||
| ### 3.4 Troubleshooting results |
There was a problem hiding this comment.
This section is good, however customer will not narrow down whether they are using stateful vs stateless firewall without taking packet captures of their traffic on the host machine. Unfortunately, netcat (nc) output will be the same in both situations. Thus, I wonder if this is too advanced for a generic user and as such we could leave it out since stateless firewalls are quite uncommon these days.
|
|
||
| - A host machine connected to the Internet where you plan to run `cloudflared`. The tests must run from the same environment where `cloudflared` will run (same network, same firewall path). | ||
|
|
||
| - A terminal session with permission to run `dig` and `nc` (netcat), or the ability to install them. |
There was a problem hiding this comment.
| - A terminal session with permission to run `dig` and `nc` (netcat), or the ability to install them. | |
| - A terminal session with permission to run `dig` and `nc` (netcat), or similar software. |
|
|
||
| For a successful and healthy deployment, `cloudflared` should have [four active replicas](/cloudflare-one/networks/connectors/cloudflare-tunnel/configure-tunnels/tunnel-availability/) with connectivity to both regions (that is, both `region1.v2.argotunnel.com` and `region2.v2.argotunnel.com`, or both `us-region1` and `us-region2`). | ||
|
|
||
| First, you need to verify that your DNS resolver returns the expected IPv4 addresses for Cloudflare Tunnel endpoints. |
There was a problem hiding this comment.
| First, you need to verify that your DNS resolver returns the expected IPv4 addresses for Cloudflare Tunnel endpoints. | |
| First, you need to verify that your DNS resolver returns the expected IP addresses for Cloudflare Tunnel endpoints. |
|
|
||
| ### 2.1. Test DNS with your current resolver | ||
|
|
||
| Depending on whether you are testing a global region or the US region, run one of the following commands: |
There was a problem hiding this comment.
add IPv6 results
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| The `ANSWER SECTION` should include the expected IPv4 addresses for Cloudflare Tunnel endpoints. |
There was a problem hiding this comment.
| The `ANSWER SECTION` should include the expected IPv4 addresses for Cloudflare Tunnel endpoints. | |
| The `ANSWER SECTION` should include the expected IP addresses for Cloudflare Tunnel endpoints. |
| - Check for firewall rules blocking (question) what? 1.1.1.1.? Endpoints? ports? | ||
| - If you are behind a managed DNS or security appliance, contact that provider to understand why queries to `region1.v2.argotunnel.com` and other Cloudflare Tunnel endpoints are blocked. | ||
|
|
||
| Once DNS resolution returns the expected IPs from at least one resolver, proceed to connectivity testing. |
There was a problem hiding this comment.
| Once DNS resolution returns the expected IPs from at least one resolver, proceed to connectivity testing. | |
| Once DNS resolution returns the expected IPs from your DNS resolver, proceed to connectivity testing. |
|
|
||
| ## 3. Test network connectivity | ||
|
|
||
| After confirming that DNS returns the correct IPs, test whether your host machine can send packets to Cloudflare on port `7844` using both UDP and TCP. |
There was a problem hiding this comment.
| After confirming that DNS returns the correct IPs, test whether your host machine can send packets to Cloudflare on port `7844` using both UDP and TCP. | |
| After confirming that your DNS resolver returns the correct IPs, test whether your host machine can send packets to Cloudflare on port `7844` using both UDP and TCP. |
deadlypants1973
left a comment
deadlypants1973
left a comment
There was a problem hiding this comment.
nikita review
| - Can the host machine send a UDP packet to Cloudflare's network? | ||
| - Can the host machine send a TCP packet to Cloudflare's network? |
There was a problem hiding this comment.
| - Can the host machine send a UDP packet to Cloudflare's network? | |
| - Can the host machine send a TCP packet to Cloudflare's network? | |
| - Can the host machine send a UDP packet to Cloudflare Tunnel endpoints? | |
| - Can the host machine send a TCP packet to Cloudflare Tunnel endpoints? |
|
|
||
| #### Both UDP and TCP succeed | ||
|
|
||
| Your firewall allows outbound traffic and return traffic to Cloudflare on port `7844`. `cloudflared` can connect using either `quic` (UDP) or `http2` (TCP). |
There was a problem hiding this comment.
| Your firewall allows outbound traffic and return traffic to Cloudflare on port `7844`. `cloudflared` can connect using either `quic` (UDP) or `http2` (TCP). | |
| Your firewall allows outbound traffic and return traffic to Cloudflare's tunnel endpoint on port `7844`. `cloudflared` can connect using either `quic` (UDP) or `http2` (TCP). If both UDP and TCP succeed and your DNS test in the previous section was successful, you can successfully deploy Cloudflare Tunnel in this environment. |
|
|
||
| `cloudflared` will only be able to connect using `quic`. If you force `http2` in your configuration while TCP is blocked, the tunnel will fail. | ||
|
|
||
| To resolve: (question) |
There was a problem hiding this comment.
| To resolve: (question) | |
| To resolve: Either allow TCP on your local network firewall on port `7844` or stop forcing `http2` to allow `cloudflared` to connect over `QUIC` instead. |
|
|
||
| #### Both UDP and TCP fail | ||
|
|
||
| Packets are being dropped somewhere between the host and Cloudflare's network. |
There was a problem hiding this comment.
| Packets are being dropped somewhere between the host and Cloudflare's network. | |
| Packets are being dropped somewhere between the host and the Cloudflare Tunnel endpoints. |
|
|
||
| This usually indicates a firewall policy or upstream security control that does not allow outbound traffic (or return traffic) on port `7844`. | ||
|
|
||
| To resolve: (question) |
There was a problem hiding this comment.
The point of failure could be your firewall configuration on the ISP, cloud provider, router, or host machine. Debugging is required.
|
|
||
| ## 4. Get help or conclusion | ||
|
|
||
| Do we want instructions on what info they need to collect when asking for support? Similar to WARP guide or just a conclusion here? |
There was a problem hiding this comment.
If either DNS or network test failed, it will likely be a problem in your local environment. You will need to debug with your administrator, ISP or cloud provider. If you believe this is CF,
- WARP list the same but no warp diag
- https://kate-fixes-tunnel-tb-guide.preview.developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/monitor-tunnels/logs/#view-logs-on-your-local-machine
- run cloudflare tunnel diag: https://kate-fixes-tunnel-tb-guide.preview.developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/troubleshoot-tunnels/diag-logs/
3 participants
Summary
Screenshots (optional)
Documentation checklist