Revert "upgrade rabbitmq always to the latest management version"

[Didayolo]

Reverting:

• upgrade rabbitmq always to the latest management version #1099

I've tested it too quickly. Submissions does not work anymore both on my local instance and on the test server.

codabench-rabbit-1       | BOOT FAILED
codabench-rabbit-1       | ===========
codabench-rabbit-1       | Error during startup: {error,
codabench-rabbit-1       |                           {schema_integrity_check_failed,
codabench-rabbit-1       |                               [{table_attributes_mismatch,rabbit_user,
codabench-rabbit-1       |                                    [username,password_hash,tags,
codabench-rabbit-1       |                                     hashing_algorithm,limits],
codabench-rabbit-1       |                                    [username,password_hash,tags,
codabench-rabbit-1       |                                     hashing_algorithm]},
codabench-rabbit-1       |                                {table_missing,rabbit_topic_permission},
codabench-rabbit-1       |                                {table_attributes_mismatch,rabbit_vhost,
codabench-rabbit-1       |                                    [virtual_host,limits,metadata],
codabench-rabbit-1       |                                    [virtual_host,dummy]},
codabench-rabbit-1       |                                {table_missing,rabbit_index_route},
codabench-rabbit-1       |                                {table_attributes_mismatch,
codabench-rabbit-1       |                                    rabbit_durable_exchange,
codabench-rabbit-1       |                                    [name,type,durable,auto_delete,internal,
codabench-rabbit-1       |                                     arguments,scratches,policy,
codabench-rabbit-1       |                                     operator_policy,decorators,options],
codabench-rabbit-1       |                                    [name,type,durable,auto_delete,internal,
codabench-rabbit-1       |                                     arguments,scratches,policy,decorators]},
codabench-rabbit-1       |                                {table_attributes_mismatch,rabbit_exchange,
codabench-rabbit-1       |                                    [name,type,durable,auto_delete,internal,
codabench-rabbit-1       |                                     arguments,scratches,policy,
codabench-rabbit-1       |                                     operator_policy,decorators,options],
codabench-rabbit-1       |                                    [name,type,durable,auto_delete,internal,
codabench-rabbit-1       |                                     arguments,scratches,policy,decorators]},
codabench-rabbit-1       |                                {table_attributes_mismatch,
codabench-rabbit-1       |                                    rabbit_durable_queue,
codabench-rabbit-1       |                                    [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       |                                     arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       |                                     recoverable_slaves,policy,operator_policy,
codabench-rabbit-1       |                                     gm_pids,decorators,state,policy_version,
codabench-rabbit-1       |                                     slave_pids_pending_shutdown,vhost,options,
codabench-rabbit-1       |                                     type,type_state],
codabench-rabbit-1       |                                    [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       |                                     arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       |                                     recoverable_slaves,policy,gm_pids,
codabench-rabbit-1       |                                     decorators,state,policy_version]},
codabench-rabbit-1       |                                {table_attributes_mismatch,rabbit_queue,
codabench-rabbit-1       |                                    [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       |                                     arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       |                                     recoverable_slaves,policy,operator_policy,
codabench-rabbit-1       |                                     gm_pids,decorators,state,policy_version,
codabench-rabbit-1       |                                     slave_pids_pending_shutdown,vhost,options,
codabench-rabbit-1       |                                     type,type_state],
codabench-rabbit-1       |                                    [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       |                                     arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       |                                     recoverable_slaves,policy,gm_pids,
codabench-rabbit-1       |                                     decorators,state,policy_version]}]}}
codabench-rabbit-1       | 
codabench-rabbit-1       | 
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>   crasher:
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     initial call: application_master:init/4
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     pid: <0.229.0>
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     registered_name: []
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     exception exit: {{schema_integrity_check_failed,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                          [{table_attributes_mismatch,rabbit_user,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [username,password_hash,tags,hashing_algorithm,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                limits],
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [username,password_hash,tags,hashing_algorithm]},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                           {table_missing,rabbit_topic_permission},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                           {table_attributes_mismatch,rabbit_vhost,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [virtual_host,limits,metadata],
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [virtual_host,dummy]},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                           {table_missing,rabbit_index_route},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                           {table_attributes_mismatch,rabbit_durable_exchange,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,type,durable,auto_delete,internal,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,scratches,policy,operator_policy,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                decorators,options],
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,type,durable,auto_delete,internal,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,scratches,policy,decorators]},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                           {table_attributes_mismatch,rabbit_exchange,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,type,durable,auto_delete,internal,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,scratches,policy,operator_policy,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                decorators,options],
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,type,durable,auto_delete,internal,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,scratches,policy,decorators]},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                           {table_attributes_mismatch,rabbit_durable_queue,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                recoverable_slaves,policy,operator_policy,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                gm_pids,decorators,state,policy_version,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                slave_pids_pending_shutdown,vhost,options,type,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                type_state],
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                recoverable_slaves,policy,gm_pids,decorators,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                state,policy_version]},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                           {table_attributes_mismatch,rabbit_queue,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                recoverable_slaves,policy,operator_policy,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                gm_pids,decorators,state,policy_version,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                slave_pids_pending_shutdown,vhost,options,type,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                type_state],
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                               [name,durable,auto_delete,exclusive_owner,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                arguments,pid,slave_pids,sync_slave_pids,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                recoverable_slaves,policy,gm_pids,decorators,
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                                state,policy_version]}]},
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>                      {rabbit,start,[normal,[]]}}
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>       in function  application_master:init/4 (application_master.erl, line 142)
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     ancestors: [<0.228.0>]
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     message_queue_len: 1
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     messages: [{'EXIT',<0.230.0>,normal}]
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     links: [<0.228.0>,<0.44.0>]
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     dictionary: []
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     trap_exit: true
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     status: running
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     heap_size: 2586
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     stack_size: 28
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>     reductions: 197
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0>   neighbours:
codabench-rabbit-1       | 2023-08-22 13:54:19.005245+00:00 [error] <0.229.0> 
codabench-rabbit-1       | 2023-08-22 13:54:19.008063+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_user,[username,password_hash,tags,hashing_algorithm,limits],[username,password_hash,tags,hashing_algorithm]},{table_missing,rabbit_topic_permission},{table_attributes_mismatch,rabbit_vhost,[virtual_host,limits,metadata],[virtual_host,dummy]},{table_missing,rabbit_index_route},{table_attributes_mismatch,rabbit_durable_exchange,[name,type,durable,auto_delete,internal,arguments,scratches,policy,operator_policy,decorators,options],[name,type,durable,auto_delete,internal,arguments,scratches,policy,decorators]},{table_attributes_mismatch,rabbit_exchange,[name,type,durable,auto_delete,internal,arguments,scratches,policy,operator_policy,decorators,options],[name,type,durable,auto_delete,internal,arguments,scratches,policy,decorators]},{table_attributes_mismatch,rabbit_durable_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options,type,type_state],[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,gm_pids,decorators,state,policy_version]},{table_attributes_mismatch,rabbit_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options,type,type_state],[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,gm_pids,decorators,state,policy_version]}]},{rabbit,start,[normal,[]]}}
codabench-rabbit-1       | {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_user,[username,password_hash,tags,hashing_algorithm,limits],[username,password_hash,tags,hashing_algorithm]},{table_missing,rabbit_topic_permission},{table_attributes_mismatch,rabbit_vhost,[virtual_host,limits,metadata],[virtual_host,dummy]},{table_missing,rabbit_index_route},{table_attributes_mismatch,rabbit_durable_exchange,[name,type,durable,auto_delete,internal,arguments,scratches,policy,operator_policy,decorators,options],[name,type,durable,auto_delete,internal,arguments,scratches,policy,decorators]},{table_attributes_mismatch,rabbit_exchange,[name,type,durable,auto_delete,internal,arguments,scratches,policy,operator_policy,decorators,options],[name,type,durable,auto_delete,internal,arguments,scratches,policy,decorators]},{table_attributes_mismatch,rabbit_durable_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options,type,type_state],[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,gm_pids,decorators,state,policy_version]},{table_attributes_mismatch,rabbit_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options,type,type_state],[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,gm_pids,decorators,state,policy_version]}]},{rabbit,start,[normal,[]]}}}"}
codabench-rabbit-1       | Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_user,[username,password_hash,tags,hashing_algorithm,limits],[username,password_hash,tags,hashing_algorithm]},{table_missing,rabbit_topic_permission},{table_attributes_mismatch,rabbit_vhost,[virtual_host,limits,metadata],[virtual_host,dummy]},{table_missing,rabbit_index_route},{table_attributes_mismatch,rabbit_durable_exchange,[name,type,durable,auto_delete,internal,arguments,scratches,policy,operator_policy,decorators,options],[name,type,durable,auto_delete,internal,arguments,scratches,policy,decorators]},{table_attributes_mismatch,rabbit_exchange,[name,type,durable,auto_delete,internal,arguments,scratches,policy,operator_policy,decorators,options],[name,type,durable,auto_delete,internal,arguments,scratches,policy,decorators]},{table_attributes_mismatch,rabbit_durable_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recove
codabench-rabbit-1       | 
codabench-rabbit-1       | Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

codabench-site_worker-1  | [2023-08-21 16:43:15,266: ERROR/Beat] beat: Connection error: failed to resolve broker hostname. Trying again in 32.0 seconds...
codabench-site_worker-1  | [2023-08-21 16:43:45,521: ERROR/MainProcess] consumer: Cannot connect to amqp://rabbit-codabench-test:**@rabbit:5672//: [Errno 111] Connection refused.
codabench-site_worker-1  | Trying again in 32.00 seconds...
codabench-site_worker-1  | 
codabench-site_worker-1  | 
codabench-site_worker-1  | [2023-08-21 16:43:47,317: ERROR/Beat] beat: Connection error: [Errno 111] Connection refused. Trying again in 32.0 seconds...
codabench-site_worker-1  | [2023-08-21 16:44:17,604: ERROR/MainProcess] consumer: Cannot connect to amqp://rabbit-codabench-test:**@rabbit:5672//: failed to resolve broker hostname.
codabench-site_worker-1  | Trying again in 32.00 seconds...
codabench-site_worker-1  | 
codabench-site_worker-1  | 
codabench-site_worker-1  | [2023-08-21 16:44:19,390: ERROR/Beat] beat: Connection error: failed to resolve broker hostname. Trying again in 32.0 seconds...
codabench-site_worker-1  | [2023-08-21 16:44:49,687: ERROR/MainProcess] consumer: Cannot connect to amqp://rabbit-codabench-test:**@rabbit:5672//: failed to resolve broker hostname.
codabench-site_worker-1  | Trying again in 32.00 seconds...
codabench-site_worker-1  | 
codabench-site_worker-1  | 
codabench-site_worker-1  | [2023-08-21 16:44:51,463: ERROR/Beat] beat: Connection error: failed to resolve broker hostname. Trying again in 32.0 seconds...
codabench-site_worker-1  | [2023-08-21 16:45:21,775: ERROR/MainProcess] consumer: Cannot connect to amqp://rabbit-codabench-test:**@rabbit:5672//: failed to resolve broker hostname.
codabench-site_worker-1  | Trying again in 32.00 seconds...
codabench-site_worker-1  | 
codabench-site_worker-1  | 
codabench-site_worker-1  | [2023-08-21 16:45:23,534: ERROR/Beat] beat: Connection error: failed to resolve broker hostname. Trying again in 32.0 seconds...
codabench-site_worker-1  | [2023-08-21 16:45:53,855: ERROR/MainProcess] consumer: Cannot connect to amqp://rabbit-codabench-test:**@rabbit:5672//: failed to resolve broker hostname.
codabench-site_worker-1  | Trying again in 32.00 seconds...
codabench-site_worker-1  | 
codabench-site_worker-1  | 
codabench-site_worker-1  | [2023-08-21 16:45:55,601: ERROR/Beat] beat: Connection error: failed to resolve broker hostname. Trying again in 32.0 seconds...
codabench-site_worker-1  | [2023-08-21 16:46:25,918: ERROR/MainProcess] consumer: Cannot connect to amqp://rabbit-codabench-test:**@rabbit:5672//: failed to resolve broker hostname.
codabench-site_worker-1  | Trying again in 32.00 seconds...
codabench-site_worker-1  | 
codabench-site_worker-1  | 
codabench-site_worker-1  | [2023-08-21 16:46:27,675: ERROR/Beat] beat: Connection error: failed to resolve broker hostname. Trying again in 32.0 seconds...
codabench-site_worker-1  | [2023-08-21 16:46:57,993: ERROR/MainProcess] consumer: Cannot connect to amqp://rabbit-codabench-test:**@rabbit:5672//: failed to resolve broker hostname.
codabench-site_worker-1  | Trying again in 32.00 seconds...

[Didayolo]

@dtuantran

[dtuantran]

Maybe it's because your rabbitmq didn't stop proprely, then the rabbit database was crashed. You can remove all data in var/rabbit/ then execute docker compose down docker compose up -d again, it should works.