Downloading times out; doesn't resume: "[28] Timeout was reached"

[BuildStream-Migration-Bot]

See original issue on GitLab
In GitLab by [Gitlab user @jjardon] on Feb 28, 2018, 11:45

This same issue seems to have been reported in:

• freedesktop-sdk issue: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/33, logs: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/54775812
• flatpak: Downloading times out; doesn't resume flatpak/flatpak#1196
• ostree: add some retry logic for pulling from remotes ostreedev/ostree#878
• curl: "curl: (56) Unexpected EOF" with HTTP/2.0 requests curl/curl#1618 (comment)

I'm submitting this here as we are still reproducing the issue even using latest ostree. Also in case buildstream wanted to implement something about that (retries, etc...)

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @jjardon] on Apr 3, 2018, 21:17

FYI: This is hitting us (freedesktop-sdk) hard: if there are more more than ~6 pipelines running (this is 3x6=18 jobs), we will always get this error

Other similar errors:

• https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/81
• https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/86

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @valentindavid] on Apr 17, 2018, 14:35

While I do not get the exact error code, I still can reproduce the instability with a pull big enough. When the number of requests is bigger than default limit (1000), then connections get dropped and cURL has troubles to recover from it.

Note the limit is not about concurrent requests. But the total number of requests to keep the connection alive.

It would be a bit dirty to handle the issue in buildstream.

• ostree provides cURL error code within a string.
• so far there has been 3 or 4 different error codes associated with this issue.
• buildstream already retries.
• if a ref requires downloading more than 1000 files (depending on configured limit), I am not sure it is possible at all to go around it.

It would be also better spent time to fix cURL.

There are possibility that it is not just one bug, but several bugs. But then it is the same. http2 is unstable between cURL and nginx, and users should be advised to disable http2 for the time being on artifact cache servers.

Downloading sources should not be a problem since there is one connection per download.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @jjardon] on Apr 17, 2018, 14:59

[Gitlab user @valentindavid] thanks for taking a look to this: we have disabled http2 in our cache server (See https://gitlab.com/freedesktop-sdk/infrastructure/merge_requests/15). We can still reproduce the issue

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on Apr 17, 2018, 16:45

From your assesment:

• a ref requiring 1000 or more objects sounds absolutely plausible to me
• does ostree have a configurable limit on concurrent connections for file downloads ?
  • if so, i think we should set a reasonable limit and let ostree queue its downloads
  • if not, perhaps we should fix ostree to support this in some way, configurable or otherwise ?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @valentindavid] on Apr 17, 2018, 17:09

ostree is not configurable. I am not sure that cURL has any configuration for that. We would probably need to manually handle it counting requests and using two multi in ostree.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @toscalix] on May 21, 2018, 08:44

marked this issue as related to #401

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @toscalix] on May 21, 2018, 09:23

assigned to [Gitlab user @valentindavid]

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @LaurenceUrhegyi] on May 22, 2018, 12:20

Will this issue go away when the OSTree cache is removed and replaced by the CAS cache? This is expected soon.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @toscalix] on May 22, 2018, 14:11

marked this issue as related to #387

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @toscalix] on May 22, 2018, 14:19

We have agreed that the way to go about this issue is to try the #387 solution to see if it fixes this issue. Freedesktop-sdk guys will test the #387 solution on their own.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @toscalix] on May 23, 2018, 16:05

marked this issue as related to freedesktop-sdk/freedesktop-sdk#150

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @toscalix] on May 29, 2018, 09:30

Amount of jobs per runner reduced. The impact is more limnited than before but still it is a problem when there is a lot of people building. With 15 jobs, that is 5 pipelines, the problem is there.