SIGSEGV in api.renterFilesHandler

[mtlynch]

Bug Report

I got a segfault when running the Minio Sia gateway. I suspect it's related to the fact that Minio is calling /renter/files at 1 QPS for each upload in progress, so about 6-8 QPS total.

Stack Trace or error message

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4edb73]

goroutine 77205 [running]:
bufio.(*Writer).Available(...)
        /usr/local/go/src/bufio/bufio.go:584
bufio.(*Writer).Write(0x0, 0xe07280, 0x2, 0x2, 0x220f, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:594 +0x73
net/http.(*chunkWriter).Write(0xc420074f18, 0xc450628000, 0x220f, 0x3051, 0xc44ee950a8, 0xc4b5163bc0, 0xc4b5163be0)
        /usr/local/go/src/net/http/server.go:366 +0x16c
bufio.(*Writer).Write(0xc430bf2e80, 0xc450628000, 0x220f, 0x3051, 0x0, 0xc4b5163c60, 0x515b82)
        /usr/local/go/src/bufio/bufio.go:599 +0x14f
net/http.(*response).write(0xc420074ee0, 0x220f, 0xc450628000, 0x220f, 0x3051, 0x0, 0x0, 0xc4251fcda0, 0x99, 0x100)
        /usr/local/go/src/net/http/server.go:1500 +0x150
net/http.(*response).Write(0xc420074ee0, 0xc450628000, 0x220f, 0x3051, 0x0, 0x0, 0xac45ed)
        /usr/local/go/src/net/http/server.go:1470 +0x64
encoding/json.(*Encoder).Encode(0xc4b5163d80, 0xa1e7e0, 0xc4251fcda0, 0xe0b0e0, 0xc420074ee0)
        /usr/local/go/src/encoding/json/stream.go:218 +0x1e2
github.com/NebulousLabs/Sia/api.WriteJSON(0xe11120, 0xc420074ee0, 0xa1e7e0, 0xc4251fcda0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/api.go:151 +0x10b
github.com/NebulousLabs/Sia/api.(*API).renterFilesHandler(0xc42018c120, 0xe11120, 0xc420074ee0, 0xc4202d4d00, 0x0, 0x0, 0x0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/renter.go:314 +0x9a
github.com/NebulousLabs/Sia/api.(*API).(github.com/NebulousLabs/Sia/api.renterFilesHandler)-fm(0xe11120, 0xc420074ee0, 0xc4202d4d00, 0x0, 0x0, 0x0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:72 +0x66
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc420284a40, 0xe11120, 0xc420074ee0, 0xc4202d4d00)
        /home/mike/go/src/github.com/julienschmidt/httprouter/router.go:344 +0x799
github.com/NebulousLabs/Sia/api.RequireUserAgent.func1(0xe11120, 0xc420074ee0, 0xc4202d4d00)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:173 +0xba
net/http.HandlerFunc.ServeHTTP(0xc420eb27b0, 0xe11120, 0xc420074ee0, 0xc4202d4d00)
        /usr/local/go/src/net/http/server.go:1918 +0x44
github.com/NebulousLabs/Sia/api.cleanCloseHandler.func1.1(0xc43284e4e0, 0xe0c520, 0xc420eb27b0, 0xe11120, 0xc420074ee0, 0xc4202d4d00)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:143 +0x79
created by github.com/NebulousLabs/Sia/api.cleanCloseHandler.func1
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:141 +0xb7

Environment

• Sia version: 47514b4
• OS: Ubuntu 16.04 LTS

Any useful logs

Seeing a lot of this in renter.log:

2017/11/25 14:11:59.095719 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.096310 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.096861 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.107416 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.113537 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.114201 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.115324 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.121670 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.122091 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.127702 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.128310 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.128873 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.135107 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:11:59.147630 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:12:00.962026 download.go:501: Not enough workers to finish download: insufficient hosts to recover file
2017/11/25 14:12:12.973468 download.go:374: CRITICAL: ERROR: the renter is idle, but tracking 2 active pieces; resetting to zero
2017/11/25 14:26:58.572012 repairscanner.go:323: Repairing 0 chunks

[lukechampine]

I can only suspect that cleanCloseHandler is to blame here (again). The API handler is trying to write its response to a bufio.Writer that wraps the connection, but it's nil, possibly because the http.Server reused/replaced it.

[mtlynch]

Just hit this again in RC2 (561a85d).

goroutine 951 [running]:
runtime/debug.Stack(0x446327, 0x0, 0xc4221e9c10)
        /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/NebulousLabs/Sia/build.Critical(0xc4221e9d60, 0x3, 0x3)
        /home/mike/go/src/github.com/NebulousLabs/Sia/build/critical.go:16 +0x93
github.com/NebulousLabs/Sia/persist.(*Logger).Critical(0xc420dfa420, 0xc4221e9d60, 0x3, 0x3)
        /home/mike/go/src/github.com/NebulousLabs/Sia/persist/log.go:37 +0xcc
github.com/NebulousLabs/Sia/modules/renter.(*Renter).managedDownloadIteration(0xc4201263c0, 0xc4221e9f20)
        /home/mike/go/src/github.com/NebulousLabs/Sia/modules/renter/download.go:374 +0x5b0
github.com/NebulousLabs/Sia/modules/renter.(*Renter).threadedDownloadLoop(0xc4201263c0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/modules/renter/download.go:643 +0x34e
created by github.com/NebulousLabs/Sia/modules/renter.newRenter
        /home/mike/go/src/github.com/NebulousLabs/Sia/modules/renter/renter.go:267 +0x46d
Critical error: ERROR: the renter is idle, but tracking 36 active pieces; resetting to zero
Please submit a bug report here: https://github.com/NebulousLabs/Sia/issues
goroutine 951 [running]:
runtime/debug.Stack(0x446327, 0x0, 0xc4221e9c10)
        /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/NebulousLabs/Sia/build.Critical(0xc4221e9d60, 0x3, 0x3)
        /home/mike/go/src/github.com/NebulousLabs/Sia/build/critical.go:16 +0x93
github.com/NebulousLabs/Sia/persist.(*Logger).Critical(0xc420dfa420, 0xc4221e9d60, 0x3, 0x3)
        /home/mike/go/src/github.com/NebulousLabs/Sia/persist/log.go:37 +0xcc
github.com/NebulousLabs/Sia/modules/renter.(*Renter).managedDownloadIteration(0xc4201263c0, 0xc4221e9f20)
        /home/mike/go/src/github.com/NebulousLabs/Sia/modules/renter/download.go:374 +0x5b0
github.com/NebulousLabs/Sia/modules/renter.(*Renter).threadedDownloadLoop(0xc4201263c0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/modules/renter/download.go:643 +0x34e
created by github.com/NebulousLabs/Sia/modules/renter.newRenter
        /home/mike/go/src/github.com/NebulousLabs/Sia/modules/renter/renter.go:267 +0x46d
Critical error: ERROR: the renter is idle, but tracking 16 active pieces; resetting to zero
Please submit a bug report here: https://github.com/NebulousLabs/Sia/issues
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4ec2f3]

goroutine 370762 [running]:
bufio.(*Writer).Available(...)
        /usr/local/go/src/bufio/bufio.go:584
bufio.(*Writer).Write(0x0, 0xdfe020, 0x2, 0x2, 0x5e93, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:594 +0x73
net/http.(*chunkWriter).Write(0xc4218cc1f8, 0xc4214a0000, 0x5e93, 0x6e4b, 0xc42829be28, 0xc425c4bbc0, 0xc425c4bbe0)
        /usr/local/go/src/net/http/server.go:366 +0x16c
bufio.(*Writer).Write(0xc437b04600, 0xc4214a0000, 0x5e93, 0x6e4b, 0x0, 0xc425c4bc60, 0x514302)
        /usr/local/go/src/bufio/bufio.go:599 +0x14f
net/http.(*response).write(0xc4218cc1c0, 0x5e93, 0xc4214a0000, 0x5e93, 0x6e4b, 0x0, 0x0, 0xc42581a3a0, 0x99, 0x100)
        /usr/local/go/src/net/http/server.go:1500 +0x150
net/http.(*response).Write(0xc4218cc1c0, 0xc4214a0000, 0x5e93, 0x6e4b, 0x0, 0x0, 0xabea35)
        /usr/local/go/src/net/http/server.go:1470 +0x64
encoding/json.(*Encoder).Encode(0xc425c4bd80, 0xa18c80, 0xc42581a3a0, 0xe01ea0, 0xc4218cc1c0)
        /usr/local/go/src/encoding/json/stream.go:218 +0x1e2
github.com/NebulousLabs/Sia/api.WriteJSON(0xe07ea0, 0xc4218cc1c0, 0xa18c80, 0xc42581a3a0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/api.go:151 +0x10b
github.com/NebulousLabs/Sia/api.(*API).renterFilesHandler(0xc4203ba000, 0xe07ea0, 0xc4218cc1c0, 0xc42057a700, 0x0, 0x0, 0x0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/renter.go:314 +0x9a
github.com/NebulousLabs/Sia/api.(*API).(github.com/NebulousLabs/Sia/api.renterFilesHandler)-fm(0xe07ea0, 0xc4218cc1c0, 0xc42057a700, 0x0, 0x0, 0x0)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:72 +0x66
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc420fec440, 0xe07ea0, 0xc4218cc1c0, 0xc42057a700)
        /home/mike/go/src/github.com/julienschmidt/httprouter/router.go:344 +0x799
github.com/NebulousLabs/Sia/api.RequireUserAgent.func1(0xe07ea0, 0xc4218cc1c0, 0xc42057a700)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:173 +0xba
net/http.HandlerFunc.ServeHTTP(0xc4221f0870, 0xe07ea0, 0xc4218cc1c0, 0xc42057a700)
        /usr/local/go/src/net/http/server.go:1918 +0x44
github.com/NebulousLabs/Sia/api.cleanCloseHandler.func1.1(0xc424e6c360, 0xe032a0, 0xc4221f0870, 0xe07ea0, 0xc4218cc1c0, 0xc42057a700)
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:143 +0x79
created by github.com/NebulousLabs/Sia/api.cleanCloseHandler.func1
        /home/mike/go/src/github.com/NebulousLabs/Sia/api/routes.go:141 +0xb7