Intermittent crashes

[theoephraim]

Unclear if this is modjs or keydb related, but I am seeing crashes.

I believe it has to do with a high volume of logging - as when I disable my logging function (which calls keydb.log) then the crashes tend to stop happening. I can trigger it fairly reliably if I enable logging and run my script with a large amount of data. If I run the same script with a small volume of calls/data, then it does not crash.

bug report details:

=== KEYDB BUG REPORT START: Cut & paste starting from here ===

1:22:M 09 Nov 2020 16:52:06.950 # KeyDB 6.0.16 crashed by signal: 11

1:22:M 09 Nov 2020 16:52:06.950 # Crashed running the instruction at: 0x7f2ad998232a

1:22:M 09 Nov 2020 16:52:06.950 # Accessing address: (nil)

1:22:M 09 Nov 2020 16:52:06.950 # Failed assertion: <no assertion failed> (<no file>:0)


------ STACK TRACE ------

EIP:

/modjs-modules/modjs.so(v8::internal::RootIndexMap::RootIndexMap(v8::internal::Isolate*)+0x3a) [0x7f2ad998232a]


Backtrace:

keydb-server *:6379(logStackTrace(ucontext_t*)+0x6b) [0x555a80ea492b]

keydb-server *:6379(sigsegvHandler(int, siginfo_t*, void*)+0xc9) [0x555a80ea50c9]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f2ae20e08a0]

/modjs-modules/modjs.so(v8::internal::RootIndexMap::RootIndexMap(v8::internal::Isolate*)+0x3a) [0x7f2ad998232a]

/modjs-modules/modjs.so(v8::internal::compiler::JSHeapBroker::JSHeapBroker(v8::internal::Isolate*, v8::internal::Zone*, bool, bool)+0x67) [0x7f2ada15e067]

/modjs-modules/modjs.so(v8::internal::compiler::PipelineData::PipelineData(v8::internal::compiler::ZoneStats*, v8::internal::Isolate*, v8::internal::OptimizedCompilationInfo*, v8::internal::compiler::PipelineStatistics*, bool)+0x1a5) [0x7f2ad9b3a265]

/modjs-modules/modjs.so(v8::internal::compiler::PipelineCompilationJob::PipelineCompilationJob(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::BailoutId, v8::internal::JavaScriptFrame*)+0x1b3) [0x7f2ad9b39ee3]

/modjs-modules/modjs.so(v8::internal::compiler::Pipeline::NewCompilationJob(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, bool, v8::internal::BailoutId, v8::internal::JavaScriptFrame*)+0xa4) [0x7f2ad9b44044]

/modjs-modules/modjs.so(+0x4b8049) [0x7f2ad9612049]

/modjs-modules/modjs.so(v8::internal::Compiler::CompileOptimized(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::ConcurrencyMode)+0x8e) [0x7f2ad9612f9e]

/modjs-modules/modjs.so(v8::internal::Runtime_CompileOptimized_Concurrent(int, unsigned long*, v8::internal::Isolate*)+0x95) [0x7f2ad9e674d5]

/modjs-modules/modjs.so(+0xb17858) [0x7f2ad9c71858]


------ INFO OUTPUT ------

�

# Server

redis_version:6.0.16

redis_git_sha1:63c47a09

redis_git_dirty:1

redis_build_id:f20f71e10920f6e0

redis_mode:standalone

os:Linux 4.19.76-linuxkit x86_64

arch_bits:64

multiplexing_api:epoll

atomicvar_api:atomic-builtin

gcc_version:7.5.0

process_id:1

run_id:ae5f90e55f15bb107b3c6f5ca6731c0ef6c0c92e

tcp_port:6379

uptime_in_seconds:10

uptime_in_days:0

hz:10

configured_hz:10

lru_clock:11105206

executable:/data/keydb-server

config_file:/etc/keydb/keydb.conf


# Clients

connected_clients:1

client_recent_max_input_buffer:2

client_recent_max_output_buffer:0

blocked_clients:0

tracking_clients:0

clients_in_timeout_table:0

current_client_thread:0

thread_0_clients:1

thread_1_clients:0


�

# Memory

used_memory:9673568

used_memory_human:9.23M

used_memory_rss:34697216

used_memory_rss_human:33.09M

used_memory_peak:9673568

used_memory_peak_human:9.23M

used_memory_peak_perc:100.34%

used_memory_overhead:9566170

used_memory_startup:9541208

used_memory_dataset:107398

used_memory_dataset_perc:81.14%

allocator_allocated:10278832

allocator_active:11214848

allocator_resident:13946880

total_system_memory:3142434816

total_system_memory_human:2.93G

used_memory_lua:41984

used_memory_lua_human:41.00K

used_memory_scripts:128

used_memory_scripts_human:128B

number_of_cached_scripts:1

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

allocator_frag_ratio:1.09

allocator_frag_bytes:936016

allocator_rss_ratio:1.24

allocator_rss_bytes:2732032

rss_overhead_ratio:2.49

rss_overhead_bytes:20750336

mem_fragmentation_ratio:3.61

mem_fragmentation_bytes:25098216

mem_not_counted_for_evict:0

mem_replication_backlog:0

mem_clients_slaves:0

mem_clients_normal:17186

mem_aof_buffer:0

mem_allocator:jemalloc-5.1.0

active_defrag_running:0

lazyfree_pending_objects:0


# Persistence

loading:0

rdb_changes_since_last_save:1452

rdb_bgsave_in_progress:0

rdb_last_save_time:1604940723

rdb_last_bgsave_status:ok

rdb_last_bgsave_time_sec:-1

rdb_current_bgsave_time_sec:-1

rdb_last_cow_size:0

aof_enabled:0

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:-1

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_last_write_status:ok

aof_last_cow_size:0

module_fork_in_progress:0

module_fork_last_cow_size:0


# Stats

total_connections_received:1

total_commands_processed:996

instantaneous_ops_per_sec:23

total_net_input_bytes:7276

total_net_output_bytes:7109

instantaneous_input_kbps:0.13

instantaneous_output_kbps:0.02

rejected_connections:0

sync_full:0

sync_partial_ok:0

sync_partial_err:0

expired_keys:0

expired_stale_perc:0.00

expired_time_cap_reached_count:0

expire_cycle_cpu_milliseconds:0

evicted_keys:0

keyspace_hits:176

keyspace_misses:26

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:0

migrate_cached_sockets:0

slave_expires_tracked_keys:0

active_defrag_hits:0

active_defrag_misses:0

active_defrag_key_hits:0

active_defrag_key_misses:0

tracking_total_keys:0

tracking_total_items:0

tracking_total_prefixes:0

unexpected_error_replies:0


# Replication

role:master

connected_slaves:0

master_replid:59caeead660634c6dc98213a1cad19123a80ec09

master_replid2:0000000000000000000000000000000000000000

master_repl_offset:0

second_repl_offset:-1

repl_backlog_active:0

repl_backlog_size:1048576

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0


# CPU

used_cpu_sys:0.105267

used_cpu_user:0.133195

used_cpu_sys_children:0.002780

used_cpu_user_children:0.000525

server_threads:2

long_lock_waits:0


# Modules

module:name=modjs,ver=1,api=1,filters=0,usedby=[],using=[],options=[]


# Commandstats

cmdstat_expiremember:calls=20,usec=265,usec_per_call=13.25

cmdstat_RLQ_TICK:calls=11,usec=55026,usec_per_call=5002.36

cmdstat_scard:calls=4,usec=37,usec_per_call=9.25

cmdstat_incr:calls=11,usec=101,usec_per_call=9.18

cmdstat_RLQ_GET_NEXT_JOBS:calls=2,usec=14301,usec_per_call=7150.50

cmdstat_hdel:calls=4,usec=42,usec_per_call=10.50

cmdstat_zadd:calls=142,usec=2391,usec_per_call=16.84

cmdstat_hmset:calls=347,usec=2855,usec_per_call=8.23

cmdstat_zrem:calls=142,usec=1159,usec_per_call=8.16

cmdstat_RLQ_JOB_COMPLETED:calls=4,usec=2634,usec_per_call=658.50

cmdstat_keydb.cron:calls=1,usec=9,usec_per_call=9.00

cmdstat_RLQ_ADD_JOBS:calls=1,usec=8820,usec_per_call=8820.00

cmdstat_hgetall:calls=166,usec=1233,usec_per_call=7.43

cmdstat_zrangebyscore:calls=27,usec=403,usec_per_call=14.93

cmdstat_hget:calls=4,usec=73,usec_per_call=18.25

cmdstat_del:calls=1,usec=5,usec_per_call=5.00

cmdstat_hmget:calls=1,usec=122,usec_per_call=122.00

cmdstat_hincrby:calls=80,usec=655,usec_per_call=8.19

cmdstat_srem:calls=14,usec=143,usec_per_call=10.21

cmdstat_flushall:calls=1,usec=1394,usec_per_call=1394.00

cmdstat_sadd:calls=9,usec=216,usec_per_call=24.00

cmdstat_info:calls=1,usec=67,usec_per_call=67.00

cmdstat_RLQ_INIT:calls=1,usec=681,usec_per_call=681.00

cmdstat_RLQ_DEFINE_LIMIT_TYPE:calls=2,usec=1156,usec_per_call=578.00


# Cluster

cluster_enabled:0


# Keyspace

db0:keys=113,expires=2,avg_ttl=48996


------ CLIENT LIST OUTPUT ------

id=360 addr=172.25.0.1:37754 fd=89 name= age=3 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=36 qbuf-free=32732 obl=0 oll=0 omem=0 events=r cmd=RLQ_JOB_COMPLETED user=default


------ REGISTERS ------

1:22:M 09 Nov 2020 16:52:06.965 #

RAX:00007f2ad080e918 RBX:00007f2ad080e918

RCX:00007f2ac0000000 RDX:0000555a80fc8140

RDI:0000000000000000 RSI:00007f2ad27fab60

RBP:00007f2ad27f8c60 RSP:00007f2ad27f8c20

R8 :0000000000000004 R9 :00000000000002a8

R10:00007f2ad27faab8 R11:0000000000000054

R12:000037be00000000 R13:000037be00000000

R14:00007f2ad087f330 R15:0000000000000000

RIP:00007f2ad998232a EFL:0000000000010202

CSGSFS:002b000000000033

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c2f) -> 00007f2ad08820c8

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c2e) -> 00007f2ad087f300

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c2d) -> 0000000000000000

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c2c) -> 000037be00000000

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c2b) -> 00007f2ad0882170

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c2a) -> 00007f2ad08820c8

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c29) -> 00007f2ada15e067

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c28) -> 00007f2ad27f8ca0

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c27) -> 0000000000000000

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c26) -> 00007f2ac4004b18

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c25) -> 0000000000000000

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c24) -> 000037be00000000

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c23) -> 00007f2ad087f300

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c22) -> 00007f2ad0882040

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c21) -> 00007f2ad99e2ed5

1:22:M 09 Nov 2020 16:52:06.965 # (00007f2ad27f8c20) -> 00007f2ad27f8c40


------ MODULES INFO OUTPUT ------


------ FAST MEMORY TEST ------

1:22:M 09 Nov 2020 16:52:06.965 # Bio thread for job type #0 terminated

1:22:M 09 Nov 2020 16:52:06.965 # Bio thread for job type #1 terminated

1:22:M 09 Nov 2020 16:52:06.966 # Bio thread for job type #2 terminated

*** Preparing to test memory region 37be00000000 (49152 bytes)

*** Preparing to test memory region 37be00040000 (4096 bytes)

*** Preparing to test memory region 37be00080000 (4096 bytes)

*** Preparing to test memory region 37be08080000 (577536 bytes)

*** Preparing to test memory region 37be08140000 (4096 bytes)

*** Preparing to test memory region 37be08180000 (671744 bytes)

*** Preparing to test memory region 37be08240000 (135168 bytes)

*** Preparing to test memory region 37be08300000 (262144 bytes)

*** Preparing to test memory region 555a81211000 (2281472 bytes)

*** Preparing to test memory region 555a814d6000 (700416 bytes)

*** Preparing to test memory region 7f2abc000000 (135168 bytes)

*** Preparing to test memory region 7f2ac4000000 (135168 bytes)

*** Preparing to test memory region 7f2ac8600000 (2097152 bytes)

*** Preparing to test memory region 7f2acb800000 (8388608 bytes)

*** Preparing to test memory region 7f2acc000000 (135168 bytes)

*** Preparing to test memory region 7f2ad0200000 (4194304 bytes)

*** Preparing to test memory region 7f2ad07ff000 (4194304 bytes)

*** Preparing to test memory region 7f2ad0c00000 (8388608 bytes)

*** Preparing to test memory region 7f2ad1400000 (2097152 bytes)

*** Preparing to test memory region 7f2ad17fc000 (8388608 bytes)

*** Preparing to test memory region 7f2ad1ffd000 (8388608 bytes)

*** Preparing to test memory region 7f2ad27fe000 (8388608 bytes)

*** Preparing to test memory region 7f2ad2fff000 (8388608 bytes)

*** Preparing to test memory region 7f2ad3800000 (8388608 bytes)

*** Preparing to test memory region 7f2ad4000000 (135168 bytes)

*** Preparing to test memory region 7f2ad8159000 (8388608 bytes)

*** Preparing to test memory region 7f2ad895a000 (8388608 bytes)

*** Preparing to test memory region 7f2ada89a000 (114688 bytes)

*** Preparing to test memory region 7f2adaabe000 (8388608 bytes)

*** Preparing to test memory region 7f2adb4ca000 (24576 bytes)

*** Preparing to test memory region 7f2adb6e8000 (8192 bytes)

*** Preparing to test memory region 7f2adbb00000 (17825792 bytes)

*** Preparing to test memory region 7f2adcfc1000 (188416 bytes)

*** Preparing to test memory region 7f2add2f7000 (4096 bytes)

*** Preparing to test memory region 7f2add541000 (4096 bytes)

*** Preparing to test memory region 7f2addfcd000 (4096 bytes)

*** Preparing to test memory region 7f2ade4fc000 (4096 bytes)

*** Preparing to test memory region 7f2adee4b000 (4096 bytes)

*** Preparing to test memory region 7f2adf4c1000 (8192 bytes)

*** Preparing to test memory region 7f2adfb03000 (4096 bytes)

*** Preparing to test memory region 7f2ae082a000 (4096 bytes)

*** Preparing to test memory region 7f2ae1224000 (8192 bytes)

*** Preparing to test memory region 7f2ae20ca000 (16384 bytes)

*** Preparing to test memory region 7f2ae22e9000 (16384 bytes)

*** Preparing to test memory region 7f2ae2c28000 (16384 bytes)

*** Preparing to test memory region 7f2ae30f4000 (12288 bytes)

*** Preparing to test memory region 7f2ae3e1e000 (114688 bytes)

*** Preparing to test memory region 7f2ae3e3f000 (4096 bytes)

.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.

[theoephraim]

Been getting another similar crash when I have logging disabled. In this case, I'm hitting a custom script with long-ish list of arguments.

=== KEYDB BUG REPORT START: Cut & paste starting from here ===

1:21:M 10 Nov 2020 00:28:08.202 # KeyDB 6.0.16 crashed by signal: 11

1:21:M 10 Nov 2020 00:28:08.202 # Crashed running the instruction at: 0x7f0962d8232a

1:21:M 10 Nov 2020 00:28:08.202 # Accessing address: (nil)

1:21:M 10 Nov 2020 00:28:08.202 # Failed assertion: <no assertion failed> (<no file>:0)


------ STACK TRACE ------

EIP:

/modjs-modules/modjs.so(v8::internal::RootIndexMap::RootIndexMap(v8::internal::Isolate*)+0x3a) [0x7f0962d8232a]


Backtrace:

keydb-server *:6379(logStackTrace(ucontext_t*)+0x6b) [0x55df0a6b592b]

keydb-server *:6379(sigsegvHandler(int, siginfo_t*, void*)+0xc9) [0x55df0a6b60c9]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f096b3538a0]

/modjs-modules/modjs.so(v8::internal::RootIndexMap::RootIndexMap(v8::internal::Isolate*)+0x3a) [0x7f0962d8232a]

/modjs-modules/modjs.so(v8::internal::compiler::JSHeapBroker::JSHeapBroker(v8::internal::Isolate*, v8::internal::Zone*, bool, bool)+0x67) [0x7f096355e067]

/modjs-modules/modjs.so(v8::internal::compiler::PipelineData::PipelineData(v8::internal::compiler::ZoneStats*, v8::internal::Isolate*, v8::internal::OptimizedCompilationInfo*, v8::internal::compiler::PipelineStatistics*, bool)+0x1a5) [0x7f0962f3a265]

/modjs-modules/modjs.so(v8::internal::compiler::PipelineCompilationJob::PipelineCompilationJob(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::BailoutId, v8::internal::JavaScriptFrame*)+0x1b3) [0x7f0962f39ee3]

/modjs-modules/modjs.so(v8::internal::compiler::Pipeline::NewCompilationJob(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, bool, v8::internal::BailoutId, v8::internal::JavaScriptFrame*)+0xa4) [0x7f0962f44044]

/modjs-modules/modjs.so(+0x4b8049) [0x7f0962a12049]

/modjs-modules/modjs.so(v8::internal::Compiler::CompileOptimized(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::ConcurrencyMode)+0x8e) [0x7f0962a12f9e]

/modjs-modules/modjs.so(v8::internal::Runtime_CompileOptimized_Concurrent(int, unsigned long*, v8::internal::Isolate*)+0x95) [0x7f09632674d5]

/modjs-modules/modjs.so(+0xb17858) [0x7f0963071858]


------ INFO OUTPUT ------

�

# Server

redis_version:6.0.16

redis_git_sha1:63c47a09

redis_git_dirty:1

redis_build_id:f20f71e10920f6e0

redis_mode:standalone

os:Linux 4.19.76-linuxkit x86_64

arch_bits:64

multiplexing_api:epoll

atomicvar_api:atomic-builtin

gcc_version:7.5.0

process_id:1

run_id:3f4c9d0159ad8159b278656376a12b37d97829af

tcp_port:6379

uptime_in_seconds:15

uptime_in_days:0

hz:10

configured_hz:10

lru_clock:11132568

executable:/data/keydb-server

config_file:/etc/keydb/keydb.conf


# Clients

connected_clients:1

client_recent_max_input_buffer:2

client_recent_max_output_buffer:0

blocked_clients:0

tracking_clients:0

clients_in_timeout_table:0

current_client_thread:0

thread_0_clients:1

thread_1_clients:0


�

# Memory

used_memory:9679336

used_memory_human:9.23M

used_memory_rss:35463168

used_memory_rss_human:33.82M

used_memory_peak:14455040

used_memory_peak_human:13.79M

used_memory_peak_perc:66.96%

used_memory_overhead:9567898

used_memory_startup:9541208

used_memory_dataset:111438

used_memory_dataset_perc:80.68%

allocator_allocated:9945448

allocator_active:11419648

allocator_resident:14475264

total_system_memory:3142434816

total_system_memory_human:2.93G

used_memory_lua:41984

used_memory_lua_human:41.00K

used_memory_scripts:128

used_memory_scripts_human:128B

number_of_cached_scripts:1

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

allocator_frag_ratio:1.15

allocator_frag_bytes:1474200

allocator_rss_ratio:1.27

allocator_rss_bytes:3055616

rss_overhead_ratio:2.45

rss_overhead_bytes:20987904

mem_fragmentation_ratio:3.69

mem_fragmentation_bytes:25846712

mem_not_counted_for_evict:0

mem_replication_backlog:0

mem_clients_slaves:0

mem_clients_normal:17186

mem_aof_buffer:0

mem_allocator:jemalloc-5.1.0

active_defrag_running:0

lazyfree_pending_objects:0


# Persistence

loading:0

rdb_changes_since_last_save:11087

rdb_bgsave_in_progress:0

rdb_last_save_time:1604968081

rdb_last_bgsave_status:ok

rdb_last_bgsave_time_sec:-1

rdb_current_bgsave_time_sec:-1

rdb_last_cow_size:0

aof_enabled:0

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:-1

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_last_write_status:ok

aof_last_cow_size:0

module_fork_in_progress:0

module_fork_last_cow_size:0


# Stats

total_connections_received:1

total_commands_processed:527

instantaneous_ops_per_sec:159

total_net_input_bytes:17900

total_net_output_bytes:3815

instantaneous_input_kbps:6.64

instantaneous_output_kbps:0.01

rejected_connections:0

sync_full:0

sync_partial_ok:0

sync_partial_err:0

expired_keys:0

expired_stale_perc:0.00

expired_time_cap_reached_count:0

expire_cycle_cpu_milliseconds:0

evicted_keys:0

keyspace_hits:60

keyspace_misses:31

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:0

migrate_cached_sockets:0

slave_expires_tracked_keys:0

active_defrag_hits:0

active_defrag_misses:0

active_defrag_key_hits:0

active_defrag_key_misses:0

tracking_total_keys:0

tracking_total_items:0

tracking_total_prefixes:0

unexpected_error_replies:0


# Replication

role:master

connected_slaves:0

master_replid:65b9227ea3d0632567886a089b477d8172417990

master_replid2:0000000000000000000000000000000000000000

master_repl_offset:0

second_repl_offset:-1

repl_backlog_active:0

repl_backlog_size:1048576

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0


# CPU

used_cpu_sys:0.121664

used_cpu_user:0.165906

used_cpu_sys_children:0.003446

used_cpu_user_children:0.000000

server_threads:2

long_lock_waits:0


# Modules

module:name=modjs,ver=1,api=1,filters=0,usedby=[],using=[],options=[]


# Commandstats

cmdstat_scard:calls=15,usec=182,usec_per_call=12.13

cmdstat_incr:calls=15,usec=163,usec_per_call=10.87

cmdstat_RLQ_ADD_JOBS:calls=14,usec=30945,usec_per_call=2210.36

cmdstat_hgetall:calls=46,usec=860,usec_per_call=18.70

cmdstat_flushall:calls=1,usec=9131,usec_per_call=9131.00

cmdstat_hmset:calls=328,usec=3801,usec_per_call=11.59

cmdstat_zrangebyscore:calls=30,usec=328,usec_per_call=10.93

cmdstat_del:calls=1,usec=6,usec_per_call=6.00

cmdstat_sadd:calls=42,usec=620,usec_per_call=14.76

cmdstat_keydb.cron:calls=1,usec=15,usec_per_call=15.00

cmdstat_RLQ_TICK:calls=15,usec=6363,usec_per_call=424.20

cmdstat_RLQ_DEFINE_LIMIT_TYPE:calls=3,usec=5188,usec_per_call=1729.33

cmdstat_info:calls=1,usec=120,usec_per_call=120.00

cmdstat_zadd:calls=14,usec=807,usec_per_call=57.64

cmdstat_RLQ_INIT:calls=1,usec=1631,usec_per_call=1631.00


# Cluster

cluster_enabled:0


# Keyspace

db0:keys=131,expires=1,avg_ttl=15


------ CLIENT LIST OUTPUT ------

id=49 addr=172.25.0.1:40538 fd=89 name= age=7 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=5279 qbuf-free=27489 obl=0 oll=0 omem=0 events=r cmd=RLQ_ADD_JOBS user=default


------ CURRENT CLIENT INFO ------

id=49 addr=172.25.0.1:40538 fd=89 name= age=7 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=5279 qbuf-free=27489 obl=0 oll=0 omem=0 events=r cmd=RLQ_ADD_JOBS user=default

argv[0]: 'RLQ_ADD_JOBS'

argv[1]: '{"priority":3,"limits":[["USER",37],["WORKSPACE",7],["ACCOUNT",5]]}'

argv[2]: 'j0'

argv[3]: '{"desc":"Job # 0 - W7/U37/A5 - P3"}'

argv[4]: 'j1'

argv[5]: '{"desc":"Job # 1 - W7/U37/A5 - P3"}'

argv[6]: 'j2'

argv[7]: '{"desc":"Job # 2 - W7/U37/A5 - P3"}'

argv[8]: 'j3'

argv[9]: '{"desc":"Job # 3 - W7/U37/A5 - P3"}'

argv[10]: 'j4'

argv[11]: '{"desc":"Job # 4 - W7/U37/A5 - P3"}'

argv[12]: 'j5'

argv[13]: '{"desc":"Job # 5 - W7/U37/A5 - P3"}'

argv[14]: 'j6'

argv[15]: '{"desc":"Job # 6 - W7/U37/A5 - P3"}'

argv[16]: 'j7'

argv[17]: '{"desc":"Job # 7 - W7/U37/A5 - P3"}'

argv[18]: 'j8'

argv[19]: '{"desc":"Job # 8 - W7/U37/A5 - P3"}'

argv[20]: 'j9'

argv[21]: '{"desc":"Job # 9 - W7/U37/A5 - P3"}'

argv[22]: 'j10'

argv[23]: '{"desc":"Job # 10 - W7/U37/A5 - P3"}'

argv[24]: 'j11'

argv[25]: '{"desc":"Job # 11 - W7/U37/A5 - P3"}'

argv[26]: 'j12'

argv[27]: '{"desc":"Job # 12 - W7/U37/A5 - P3"}'

argv[28]: 'j13'

argv[29]: '{"desc":"Job # 13 - W7/U37/A5 - P3"}'

argv[30]: 'j14'

argv[31]: '{"desc":"Job # 14 - W7/U37/A5 - P3"}'

argv[32]: 'j15'

argv[33]: '{"desc":"Job # 15 - W7/U37/A5 - P3"}'

argv[34]: 'j16'

argv[35]: '{"desc":"Job # 16 - W7/U37/A5 - P3"}'

argv[36]: 'j17'

argv[37]: '{"desc":"Job # 17 - W7/U37/A5 - P3"}'

argv[38]: 'j18'

argv[39]: '{"desc":"Job # 18 - W7/U37/A5 - P3"}'

argv[40]: 'j19'

argv[41]: '{"desc":"Job # 19 - W7/U37/A5 - P3"}'

argv[42]: 'j20'

argv[43]: '{"desc":"Job # 20 - W7/U37/A5 - P3"}'

argv[44]: 'j21'

argv[45]: '{"desc":"Job # 21 - W7/U37/A5 - P3"}'

argv[46]: 'j22'

argv[47]: '{"desc":"Job # 22 - W7/U37/A5 - P3"}'

argv[48]: 'j23'

argv[49]: '{"desc":"Job # 23 - W7/U37/A5 - P3"}'

argv[50]: 'j24'

argv[51]: '{"desc":"Job # 24 - W7/U37/A5 - P3"}'

argv[52]: 'j25'

argv[53]: '{"desc":"Job # 25 - W7/U37/A5 - P3"}'

argv[54]: 'j26'

argv[55]: '{"desc":"Job # 26 - W7/U37/A5 - P3"}'

argv[56]: 'j27'

argv[57]: '{"desc":"Job # 27 - W7/U37/A5 - P3"}'

argv[58]: 'j28'

argv[59]: '{"desc":"Job # 28 - W7/U37/A5 - P3"}'

argv[60]: 'j29'

argv[61]: '{"desc":"Job # 29 - W7/U37/A5 - P3"}'

argv[62]: 'j30'

argv[63]: '{"desc":"Job # 30 - W7/U37/A5 - P3"}'

argv[64]: 'j31'

argv[65]: '{"desc":"Job # 31 - W7/U37/A5 - P3"}'

argv[66]: 'j32'

argv[67]: '{"desc":"Job # 32 - W7/U37/A5 - P3"}'

argv[68]: 'j33'

argv[69]: '{"desc":"Job # 33 - W7/U37/A5 - P3"}'

argv[70]: 'j34'

argv[71]: '{"desc":"Job # 34 - W7/U37/A5 - P3"}'

argv[72]: 'j35'

argv[73]: '{"desc":"Job # 35 - W7/U37/A5 - P3"}'

argv[74]: 'j36'

argv[75]: '{"desc":"Job # 36 - W7/U37/A5 - P3"}'

argv[76]: 'j37'

argv[77]: '{"desc":"Job # 37 - W7/U37/A5 - P3"}'

argv[78]: 'j38'

argv[79]: '{"desc":"Job # 38 - W7/U37/A5 - P3"}'

argv[80]: 'j39'

argv[81]: '{"desc":"Job # 39 - W7/U37/A5 - P3"}'

argv[82]: 'j40'

argv[83]: '{"desc":"Job # 40 - W7/U37/A5 - P3"}'

argv[84]: 'j41'

argv[85]: '{"desc":"Job # 41 - W7/U37/A5 - P3"}'

argv[86]: 'j42'

argv[87]: '{"desc":"Job # 42 - W7/U37/A5 - P3"}'

argv[88]: 'j43'

argv[89]: '{"desc":"Job # 43 - W7/U37/A5 - P3"}'

argv[90]: 'j44'

argv[91]: '{"desc":"Job # 44 - W7/U37/A5 - P3"}'

argv[92]: 'j45'

argv[93]: '{"desc":"Job # 45 - W7/U37/A5 - P3"}'

argv[94]: 'j46'

argv[95]: '{"desc":"Job # 46 - W7/U37/A5 - P3"}'

argv[96]: 'j47'

argv[97]: '{"desc":"Job # 47 - W7/U37/A5 - P3"}'

argv[98]: 'j48'

argv[99]: '{"desc":"Job # 48 - W7/U37/A5 - P3"}'

argv[100]: 'j49'

argv[101]: '{"desc":"Job # 49 - W7/U37/A5 - P3"}'

argv[102]: 'j50'

argv[103]: '{"desc":"Job # 50 - W7/U37/A5 - P3"}'

argv[104]: 'j51'

argv[105]: '{"desc":"Job # 51 - W7/U37/A5 - P3"}'

argv[106]: 'j52'

argv[107]: '{"desc":"Job # 52 - W7/U37/A5 - P3"}'

argv[108]: 'j53'

argv[109]: '{"desc":"Job # 53 - W7/U37/A5 - P3"}'

argv[110]: 'j54'

argv[111]: '{"desc":"Job # 54 - W7/U37/A5 - P3"}'

argv[112]: 'j55'

argv[113]: '{"desc":"Job # 55 - W7/U37/A5 - P3"}'

argv[114]: 'j56'

argv[115]: '{"desc":"Job # 56 - W7/U37/A5 - P3"}'

argv[116]: 'j57'

argv[117]: '{"desc":"Job # 57 - W7/U37/A5 - P3"}'

argv[118]: 'j58'

argv[119]: '{"desc":"Job # 58 - W7/U37/A5 - P3"}'

argv[120]: 'j59'

argv[121]: '{"desc":"Job # 59 - W7/U37/A5 - P3"}'

argv[122]: 'j60'

argv[123]: '{"desc":"Job # 60 - W7/U37/A5 - P3"}'

argv[124]: 'j61'

argv[125]: '{"desc":"Job # 61 - W7/U37/A5 - P3"}'

argv[126]: 'j62'

argv[127]: '{"desc":"Job # 62 - W7/U37/A5 - P3"}'

argv[128]: 'j63'

argv[129]: '{"desc":"Job # 63 - W7/U37/A5 - P3"}'

argv[130]: 'j64'

argv[131]: '{"desc":"Job # 64 - W7/U37/A5 - P3"}'

argv[132]: 'j65'

argv[133]: '{"desc":"Job # 65 - W7/U37/A5 - P3"}'

argv[134]: 'j66'

argv[135]: '{"desc":"Job # 66 - W7/U37/A5 - P3"}'

argv[136]: 'j67'

argv[137]: '{"desc":"Job # 67 - W7/U37/A5 - P3"}'

argv[138]: 'j68'

argv[139]: '{"desc":"Job # 68 - W7/U37/A5 - P3"}'

argv[140]: 'j69'

argv[141]: '{"desc":"Job # 69 - W7/U37/A5 - P3"}'

argv[142]: 'j70'

argv[143]: '{"desc":"Job # 70 - W7/U37/A5 - P3"}'

argv[144]: 'j71'

argv[145]: '{"desc":"Job # 71 - W7/U37/A5 - P3"}'

argv[146]: 'j72'

argv[147]: '{"desc":"Job # 72 - W7/U37/A5 - P3"}'

argv[148]: 'j73'

argv[149]: '{"desc":"Job # 73 - W7/U37/A5 - P3"}'

argv[150]: 'j74'

argv[151]: '{"desc":"Job # 74 - W7/U37/A5 - P3"}'

argv[152]: 'j75'

argv[153]: '{"desc":"Job # 75 - W7/U37/A5 - P3"}'

argv[154]: 'j76'

argv[155]: '{"desc":"Job # 76 - W7/U37/A5 - P3"}'

argv[156]: 'j77'

argv[157]: '{"desc":"Job # 77 - W7/U37/A5 - P3"}'

argv[158]: 'j78'

argv[159]: '{"desc":"Job # 78 - W7/U37/A5 - P3"}'

argv[160]: 'j79'

argv[161]: '{"desc":"Job # 79 - W7/U37/A5 - P3"}'

argv[162]: 'j80'

argv[163]: '{"desc":"Job # 80 - W7/U37/A5 - P3"}'

argv[164]: 'j81'

argv[165]: '{"desc":"Job # 81 - W7/U37/A5 - P3"}'

argv[166]: 'j82'

argv[167]: '{"desc":"Job # 82 - W7/U37/A5 - P3"}'

argv[168]: 'j83'

argv[169]: '{"desc":"Job # 83 - W7/U37/A5 - P3"}'

argv[170]: 'j84'

argv[171]: '{"desc":"Job # 84 - W7/U37/A5 - P3"}'

argv[172]: 'j85'

argv[173]: '{"desc":"Job # 85 - W7/U37/A5 - P3"}'

argv[174]: 'j86'

argv[175]: '{"desc":"Job # 86 - W7/U37/A5 - P3"}'

argv[176]: 'j87'

argv[177]: '{"desc":"Job # 87 - W7/U37/A5 - P3"}'

argv[178]: 'j88'

argv[179]: '{"desc":"Job # 88 - W7/U37/A5 - P3"}'

argv[180]: 'j89'

argv[181]: '{"desc":"Job # 89 - W7/U37/A5 - P3"}'

argv[182]: 'j90'

argv[183]: '{"desc":"Job # 90 - W7/U37/A5 - P3"}'

argv[184]: 'j91'

argv[185]: '{"desc":"Job # 91 - W7/U37/A5 - P3"}'

argv[186]: 'j92'

argv[187]: '{"desc":"Job # 92 - W7/U37/A5 - P3"}'

argv[

188]: 'j93'

argv[189]: '{"desc":"Job # 93 - W7/U37/A5 - P3"}'

argv[190]: 'j94'

argv[191]: '{"desc":"Job # 94 - W7/U37/A5 - P3"}'

argv[192]: 'j95'

argv[193]: '{"desc":"Job # 95 - W7/U37/A5 - P3"}'

argv[194]: 'j96'

argv[195]: '{"desc":"Job # 96 - W7/U37/A5 - P3"}'

argv[196]: 'j97'

argv[197]: '{"desc":"Job # 97 - W7/U37/A5 - P3"}'

argv[198]: 'j98'

argv[199]: '{"desc":"Job # 98 - W7/U37/A5 - P3"}'

argv[200]: 'j99'

argv[201]: '{"desc":"Job # 99 - W7/U37/A5 - P3"}'


------ REGISTERS ------

1:21:M 10 Nov 2020 00:28:08.216 #

RAX:00007f09603ba6c8 RBX:00007f09603ba6c8

RCX:00007f0940000000 RDX:000055df0a7d9140

RDI:0000000000000000 RSI:00007f095b7fcb00

RBP:00007f095b7faaf0 RSP:00007f095b7faab0

R8 :0000000000000077 R9 :00000000000002f8

R10:00007f095b7fcab8 R11:000000000000005e

R12:0000048700000000 R13:0000048700000000

R14:00007f0958c49030 R15:0000000000000000

RIP:00007f0962d8232a EFL:0000000000010202

CSGSFS:002b000000000033

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faabf) -> 00007f0958c730c8

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faabe) -> 00007f0958c49000

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faabd) -> 0000000000000000

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faabc) -> 0000048700000000

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faabb) -> 00007f0958c73170

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faaba) -> 00007f0958c730c8

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab9) -> 00007f096355e067

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab8) -> 00007f095b7fab30

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab7) -> 0000000000000000

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab6) -> 00007f094c009438

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab5) -> 0000000000000000

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab4) -> 0000048700000000

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab3) -> 00007f0958c49000

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab2) -> 00007f0958c73040

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab1) -> 00007f0962de2ed5

1:21:M 10 Nov 2020 00:28:08.216 # (00007f095b7faab0) -> 00007f095b7faad0


------ MODULES INFO OUTPUT ------


------ FAST MEMORY TEST ------

1:21:M 10 Nov 2020 00:28:08.216 # Bio thread for job type #0 terminated

1:21:M 10 Nov 2020 00:28:08.216 # Bio thread for job type #1 terminated

1:21:M 10 Nov 2020 00:28:08.217 # Bio thread for job type #2 terminated

*** Preparing to test memory region 48700000000 (49152 bytes)

*** Preparing to test memory region 48700040000 (4096 bytes)

*** Preparing to test memory region 48700080000 (4096 bytes)

*** Preparing to test memory region 48708080000 (577536 bytes)

*** Preparing to test memory region 48708140000 (4096 bytes)

*** Preparing to test memory region 48708180000 (675840 bytes)

*** Preparing to test memory region 48708240000 (135168 bytes)

*** Preparing to test memory region 48708280000 (524288 bytes)

*** Preparing to test memory region 55df0aa22000 (2281472 bytes)

*** Preparing to test memory region 55df0afb2000 (700416 bytes)

*** Preparing to test memory region 7f094c000000 (135168 bytes)

*** Preparing to test memory region 7f0950000000 (135168 bytes)

*** Preparing to test memory region 7f0954000000 (135168 bytes)

*** Preparing to test memory region 7f0958800000 (2097152 bytes)

*** Preparing to test memory region 7f0958bff000 (2097152 bytes)

*** Preparing to test memory region 7f0958e00000 (8388608 bytes)

*** Preparing to test memory region 7f0959600000 (2097152 bytes)

*** Preparing to test memory region 7f0959a00000 (8388608 bytes)

*** Preparing to test memory region 7f095a200000 (4194304 bytes)

*** Preparing to test memory region 7f095a7fe000 (8388608 bytes)

*** Preparing to test memory region 7f095afff000 (8388608 bytes)

*** Preparing to test memory region 7f095b800000 (8388608 bytes)

*** Preparing to test memory region 7f095c000000 (135168 bytes)

*** Preparing to test memory region 7f0960156000 (4194304 bytes)

*** Preparing to test memory region 7f0960557000 (8388608 bytes)

*** Preparing to test memory region 7f0960d58000 (8388608 bytes)

*** Preparing to test memory region 7f0961559000 (8388608 bytes)

*** Preparing to test memory region 7f0961d5a000 (8388608 bytes)

*** Preparing to test memory region 7f0963c9a000 (114688 bytes)

*** Preparing to test memory region 7f0963ebe000 (8388608 bytes)

*** Preparing to test memory region 7f09648ca000 (24576 bytes)

*** Preparing to test memory region 7f0964ae8000 (8192 bytes)

*** Preparing to test memory region 7f0964f00000 (17825792 bytes)

*** Preparing to test memory region 7f0966234000 (188416 bytes)

*** Preparing to test memory region 7f096656a000 (4096 bytes)

*** Preparing to test memory region 7f09667b4000 (4096 bytes)

*** Preparing to test memory region 7f0967240000 (4096 bytes)

*** Preparing to test memory region 7f096776f000 (4096 bytes)

*** Preparing to test memory region 7f09680be000 (4096 bytes)

*** Preparing to test memory region 7f0968734000 (8192 bytes)

*** Preparing to test memory region 7f0968d76000 (4096 bytes)

*** Preparing to test memory region 7f0969a9d000 (4096 bytes)

*** Preparing to test memory region 7f096a497000 (8192 bytes)

*** Preparing to test memory region 7f096b33d000 (16384 bytes)

*** Preparing to test memory region 7f096b55c000 (16384 bytes)

*** Preparing to test memory region 7f096be9b000 (16384 bytes)

*** Preparing to test memory region 7f096c367000 (12288 bytes)

*** Preparing to test memory region 7f096d091000 (114688 bytes)

*** Preparing to test memory region 7f096d0b2000 (4096 bytes)

.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.

[myproblemchild]

I've been experiencing similar crashes when using modJS with Redis. I didn't find a reliable way to reproduce. Logging isn't too helpful because there seems to be some buffering in action and a few log lines might get lost (so you can't even be sure where exactly your script failed).

Having some minimal script for reproduction of crashes would be really helpful here but I'm not sure anyone has time for that.