Crashed when retry many times

[zyuchen2006]

Describe the bug (描述bug)

Crashed when retry many times

I1225 15:21:56.527046 42633 src/brpc/details/naming_service_thread.cpp:202] brpc::policy::FileNamingService("server_list"): added 1
W1225 15:21:56.527429 42620 src/brpc/controller.cpp:328] Retry count can't be larger than 1000, round it to 1000

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4e63700 (LWP 42631)]
0x00007ffff68b6487 in vfprintf () from /lib64/libc.so.6

(gdb) bt
#0  0x00007ffff68b6487 in vfprintf () from /lib64/libc.so.6
#1  0x00007ffff68e38a9 in vsnprintf () from /lib64/libc.so.6
#2  0x0000000000845a8d in butil::(anonymous namespace)::string_printf_impl(std::string &, const char *, typedef __va_list_tag __va_list_tag *) (
    output="[E111]Fail to connect Socket{id=0 addr=127.0.0.1:8003} (0x0x7fffdc02bfb0): Connection refused [R1][E112]Fail to select server from file://server_list lb=rr [R2][E112]Fail to select server from file://"..., 
    format=0xc7aa3f "[R%d]", args=args@entry=0x7ffff405c618) at src/butil/string_printf.cpp:52
#3  0x0000000000845cd7 in butil::string_appendf (output=output@entry=0x7fffffffded8, format=format@entry=0xc7aa3f "[R%d]") at src/butil/string_printf.cpp:100
#4  0x00000000007544f2 in brpc::Controller::SetFailed (this=this@entry=0x7fffffffdec0, error_code=<optimized out>, error_code@entry=112, reason_fmt=reason_fmt@entry=0xc7abb2 "Fail to select server from %s")
    at src/brpc/controller.cpp:433
#5  0x00000000007583c4 in brpc::Controller::IssueRPC (this=this@entry=0x7fffffffdec0, start_realtime_us=1640416916532080) at src/brpc/controller.cpp:1029
#6  0x0000000000758a03 in brpc::Controller::OnVersionedRPCReturned (this=this@entry=0x7fffffffdec0, info=..., new_bthread=<optimized out>, saved_error=<optimized out>) at src/brpc/controller.cpp:620
#7  0x00000000007590ca in brpc::Controller::HandleSendFailed (this=this@entry=0x7fffffffdec0) at src/brpc/controller.cpp:975
#8  0x00000000007583e3 in brpc::Controller::IssueRPC (this=this@entry=0x7fffffffdec0, start_realtime_us=1640416916532079) at src/brpc/controller.cpp:1030
#9  0x0000000000758a03 in brpc::Controller::OnVersionedRPCReturned (this=this@entry=0x7fffffffdec0, info=..., new_bthread=<optimized out>, saved_error=<optimized out>) at src/brpc/controller.cpp:620
#10 0x00000000007590ca in brpc::Controller::HandleSendFailed (this=this@entry=0x7fffffffdec0) at src/brpc/controller.cpp:975
#11 0x00000000007583e3 in brpc::Controller::IssueRPC (this=this@entry=0x7fffffffdec0, start_realtime_us=1640416916532077) at src/brpc/controller.cpp:1030
#12 0x0000000000758a03 in brpc::Controller::OnVersionedRPCReturned (this=this@entry=0x7fffffffdec0, info=..., new_bthread=<optimized out>, saved_error=<optimized out>) at src/brpc/controller.cpp:620
#13 0x00000000007590ca in brpc::Controller::HandleSendFailed (this=this@entry=0x7fffffffdec0) at src/brpc/controller.cpp:975

...

#2762 0x00000000007583e3 in brpc::Controller::IssueRPC (this=this@entry=0x7fffffffdec0, start_realtime_us=1640416916528033) at src/brpc/controller.cpp:1030
#2763 0x0000000000758a03 in brpc::Controller::OnVersionedRPCReturned (this=this@entry=0x7fffffffdec0, info=..., new_bthread=new_bthread@entry=true, saved_error=saved_error@entry=0) at src/brpc/controller.cpp:620
#2764 0x00000000007592b7 in brpc::Controller::HandleSocketFailed (id=..., data=0x7fffffffdec0, error_code=111, error_text=...) at src/brpc/controller.cpp:1227
#2765 0x000000000085dba7 in bthread_id_error2_verbose (id=id@entry=..., error_code=error_code@entry=111, error_text="Fail to connect Socket{id=0 addr=127.0.0.1:8003} (0x0x7fffdc02bfb0): Connection refused", 
    location=location@entry=0xc7ccaa "src/brpc/socket.cpp:487") at src/bthread/id.cpp:729
#2766 0x00000000007690ff in brpc::Socket::ReturnFailedWriteRequest (this=this@entry=0x7fffdc02bfb0, p=p@entry=0x1c9c590, error_code=error_code@entry=111, 
    error_text="Fail to connect Socket{id=0 addr=127.0.0.1:8003} (0x0x7fffdc02bfb0): Connection refused") at src/brpc/socket.cpp:487
#2767 0x0000000000769525 in brpc::Socket::ReleaseAllFailedWriteRequests (this=this@entry=0x7fffdc02bfb0, req=req@entry=0x1c9c590) at src/brpc/socket.cpp:516
#2768 0x0000000000769d88 in brpc::Socket::AfterAppConnected (err=err@entry=111, data=data@entry=0x1c9c590) at src/brpc/socket.cpp:1339
#2769 0x0000000000769f57 in brpc::Socket::CheckConnectedAndKeepWrite (fd=fd@entry=7, err=111, err@entry=0, data=data@entry=0x1c9c590) at src/brpc/socket.cpp:1389
#2770 0x000000000076a085 in brpc::Socket::KeepWriteIfConnected (fd=7, err=0, data=0x1c9c590) at src/brpc/socket.cpp:1366
#2771 0x0000000000767bd2 in brpc::Socket::HandleEpollOut (id=<optimized out>) at src/brpc/socket.cpp:1269
#2772 0x000000000075e44b in brpc::EventDispatcher::Run (this=0x1c92fb8) at src/brpc/event_dispatcher.cpp:338
#2773 0x000000000075e4f9 in brpc::EventDispatcher::RunThis (arg=<optimized out>) at src/brpc/event_dispatcher.cpp:276
#2774 0x000000000085454a in bthread::TaskGroup::task_runner (skip_remained=<optimized out>) at src/bthread/task_group.cpp:295
#2775 0x0000000000926f41 in bthread_make_fcontext ()
#2776 0x0000000000000000 in ?? ()

To Reproduce (复现方法)

ChannelOptions:

    brpc::ChannelOptions options;
    options.backup_request_ms = FLAGS_backup_timeout_ms;
    options.protocol = FLAGS_protocol;
    options.connection_type = FLAGS_connection_type;
    options.timeout_ms = 10000;
    options.max_retry = 9999;
    options.connect_timeout_ms = 5000;

Run client when there's no started server.

Expected behavior (期望行为)

No crash after reties.

Versions (各种版本)
OS:
Compiler:
brpc:
protobuf:

Additional context/screenshots (更多上下文/截图)

[serverglen]

感觉是爆栈了吧。

[zyuchen2006]

Nevermind. Fixed by set a bigger stack_size_normal.