DropTable fails during RocksdbCloud single longevity test. Since the first time the DropTable fails, it keeps retrying and failing for two days.
Detailed log file is on eloq@store-2:/mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/logs/node-6389/eloqkv.log.INFO.20250906-174059.2981528. The file is stored to eloq@eloq-super-server1/home/eloq/store_handler_issue_17/eloqkv.log.INFO.20250906-174059.2981528.
10838 [2025-09-06T17:54:05.118448 I 2981604] txn: 4345906833 process acquire_all_intent_op_ 10839 [2025-09-06T17:54:05.121887 I 2981604] txn: 4345906833 process prepare_log_op_
10840 [2025-09-06T17:54:05.133042 I 2981604] txn: 4345906833 process post_all_intent_op_
10841 [2025-09-06T17:54:05.135954 I 2981604] txn: 4345906833 process acquire_all_lock_op_
10842 [2025-09-06T17:54:05.136749 I 2981604] txn: 4345906833 process commit_log_op_ 10843 [2025-09-06T17:54:05.142589 I 2981604] txn: 4345906833 process upsert_kv_table_op_
10844 [2025-09-06T17:54:05.143138 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10845 [2025-09-06T17:54:05.144135 E 2981616] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
10846 [2025-09-06T17:54:05.144574 W 2981678] DataStoreHandler: Failed to do DropKvTable.
10847 [2025-09-06T17:54:05.938580 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10848 [2025-09-06T17:54:05.939280 E 2981618] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
10849 [2025-09-06T17:54:05.939636 W 2981678] DataStoreHandler: Failed to do DropKvTable.
10850 [2025-09-06T17:54:07.942220 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10851 [2025-09-06T17:54:07.943130 E 2981628] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached 10852 [2025-09-06T17:54:07.943652 W 2981678] DataStoreHandler: Failed to do DropKvTable.
10853 [2025-09-06T17:54:11.949412 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10854 [2025-09-06T17:54:11.950407 E 2981617] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
465661 [2025-09-08T12:58:31.341657 W 2981678] DataStoreHandler: Failed to do DropKvTable.
465662 [2025-09-08T12:58:33.276012 I 2981614] Total memtables size: 0 MB
465663 [2025-09-08T12:58:33.276227 I 2981614] Total SSTables size used in replay: 0 MB
465664 [2025-09-08T12:58:33.276286 I 2981614] Total replay log size: 0 MB
465665 [2025-09-08T12:58:33.353834 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
465666 [2025-09-08T12:58:33.354503 E 2981616] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
465667 [2025-09-08T12:58:33.354805 W 2981678] DataStoreHandler: Failed to do DropKvTable.
465668 [2025-09-08T12:58:33.442333 I 2981713] [Global dead lock detector]: gather lock dependancy information
465669 [2025-09-08T12:58:37.360330 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
465670 [2025-09-08T12:58:37.361240 E 2981617] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
DropTable fails during RocksdbCloud single longevity test. Since the first time the DropTable fails, it keeps retrying and failing for two days.
Detailed log file is on
eloq@store-2:/mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/logs/node-6389/eloqkv.log.INFO.20250906-174059.2981528. The file is stored toeloq@eloq-super-server1/home/eloq/store_handler_issue_17/eloqkv.log.INFO.20250906-174059.2981528.