EdgeHub restarts automatically, but all other modules do not restart, leading to miscommunication between the custom module and the EdgeHub module.(Interruption of data receiving after 30 days after automatic server certificate renewal)

[malavvvakharia]

I'm not entirely certain, but it seems that the issue might be related to network configuration or a certificate renewal. As a result, EdgeHub may have restarted, preventing it from receiving the route or subscription for custom modules. Additionally, since the other modules were not restarted, some custom modules may not have been able to connect to EdgeHub either.

1. Error of route/subscription issue after restart of Edge Hub:

"No subscription for receiving messages found for testdevice/module2"

Logs:

<6> 2026-02-12 07:21:24.008 +00:00 [INF] - Stopping the protocol heads...
<6> 2026-02-12 07:21:24.011 +00:00 [INF] - Closing protocol heads - (MQTT, AMQP, HTTP)
<6> 2026-02-12 07:21:24.016 +00:00 [INF] - Stopping MQTT protocol head
<6> 2026-02-12 07:21:24.029 +00:00 [INF] - Closing link Events for testdevice/IoTEdgeMetricsCollector
<6> 2026-02-12 07:21:24.029 +00:00 [INF] - Setting proxy inactive for testdevice/IoTEdgeMetricsCollector.
<6> 2026-02-12 07:21:24.029 +00:00 [INF] - Removing device connection for device testdevice/IoTEdgeMetricsCollector with removeCloudConnection flag 'True'.
<6> 2026-02-12 07:21:24.042 +00:00 [INF] - Closing HTTP head
<6> 2026-02-12 07:21:24.071 +00:00 [INF] - Closing receiver in cloud proxy 295f5b27-2d88-43a8-920e for testdevice/IoTEdgeMetricsCollector
<6> 2026-02-12 07:21:24.072 +00:00 [INF] - Closed cloud proxy 295f5b27-2d88-43a8-920e for testdevice/IoTEdgeMetricsCollector
<6> 2026-02-12 07:21:24.072 +00:00 [INF] - Device connection removed for device testdevice/IoTEdgeMetricsCollector
<6> 2026-02-12 07:21:24.072 +00:00 [INF] - Closing link Events for testdevice/module1
<6> 2026-02-12 07:21:24.073 +00:00 [INF] - Remove device connection for device testdevice/IoTEdgeMetricsCollector
<6> 2026-02-12 07:21:24.085 +00:00 [INF] - Closing link Events for testdevice/module3
<6> 2026-02-12 07:21:24.102 +00:00 [INF] - Closing link Events for testdevice/module2
<6> 2026-02-12 07:21:24.103 +00:00 [INF] - Closing link TwinSending for testdevice/module1
<6> 2026-02-12 07:21:24.103 +00:00 [INF] - Closing link TwinReceiving for testdevice/module1
<6> 2026-02-12 07:21:24.103 +00:00 [INF] - Closing link MethodSending for testdevice/module1
<6> 2026-02-12 07:21:24.103 +00:00 [INF] - Closing link MethodReceiving for testdevice/module1
<6> 2026-02-12 07:21:24.103 +00:00 [INF] - Setting proxy inactive for testdevice/module1.
<6> 2026-02-12 07:21:24.103 +00:00 [INF] - Removing device connection for device testdevice/module1 with removeCloudConnection flag 'True'.
<6> 2026-02-12 07:21:24.117 +00:00 [INF] - Closing link TwinSending for testdevice/module2
<6> 2026-02-12 07:21:24.117 +00:00 [INF] - Closing link TwinReceiving for testdevice/module2
<6> 2026-02-12 07:21:24.117 +00:00 [INF] - Closing link ModuleMessages for testdevice/module2
<6> 2026-02-12 07:21:24.117 +00:00 [INF] - Setting proxy inactive for testdevice/module2.
<6> 2026-02-12 07:21:24.117 +00:00 [INF] - Removing device connection for device testdevice/module2 with removeCloudConnection flag 'True'.
<6> 2026-02-12 07:21:24.118 +00:00 [INF] - Closing link TwinSending for testdevice/module3
<6> 2026-02-12 07:21:24.118 +00:00 [INF] - Closing link TwinReceiving for testdevice/module3
<6> 2026-02-12 07:21:24.118 +00:00 [INF] - Setting proxy inactive for testdevice/module3.
<6> 2026-02-12 07:21:24.118 +00:00 [INF] - Removing device connection for device testdevice/module3 with removeCloudConnection flag 'True'.
<6> 2026-02-12 07:21:24.155 +00:00 [INF] - Closing receiver in cloud proxy 34d647e7-6704-46d5-aaad-c1836956abda for testdevice/module1
<6> 2026-02-12 07:21:24.155 +00:00 [INF] - Closed cloud proxy 34d647e7-6704-46d5-aaad-c1836956abda for testdevice/module1
<6> 2026-02-12 07:21:24.155 +00:00 [INF] - Device connection removed for device testdevice/module1
<6> 2026-02-12 07:21:24.155 +00:00 [INF] - Remove device connection for device testdevice/module1
<6> 2026-02-12 07:21:24.174 +00:00 [INF] - Connection closed CorrelationId d59a41bc-687c-44ba-a11f-9cf9fb2d0ebc
<6> 2026-02-12 07:21:24.177 +00:00 [INF] - Connection closed CorrelationId 04f84ada-f5ae-46d1-8d3b-48c3832fa49c
<6> 2026-02-12 07:21:24.178 +00:00 [INF] - Closing receiver in cloud proxy d4a7f0e3-dcd9-41f5-a847-e95a475448c1 for testdevice/module2
<6> 2026-02-12 07:21:24.179 +00:00 [INF] - Closed cloud proxy d4a7f0e3-dcd9-41f5-a847-e95a475448c1 for testdevice/module2
<6> 2026-02-12 07:21:24.179 +00:00 [INF] - Device connection removed for device testdevice/module2
<6> 2026-02-12 07:21:24.179 +00:00 [INF] - Remove device connection for device testdevice/module2
<6> 2026-02-12 07:21:24.188 +00:00 [INF] - Closing receiver in cloud proxy 44d9076c-748d-4597-9d73-62b98105d266 for testdevice/module3
<6> 2026-02-12 07:21:24.188 +00:00 [INF] - Closed cloud proxy 44d9076c-748d-4597-9d73-62b98105d266 for testdevice/module3
<6> 2026-02-12 07:21:24.188 +00:00 [INF] - Device connection removed for device testdevice/module3
<6> 2026-02-12 07:21:24.189 +00:00 [INF] - Remove device connection for device testdevice/module3
<6> 2026-02-12 07:21:24.196 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "https"://"testdevice""""/$iothub/websocket""" - 101 null null 2591840222.9543ms
<6> 2026-02-12 07:21:24.200 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "https"://"testdevice""""/$iothub/websocket""" - 101 null null 2591840111.798ms
<6> 2026-02-12 07:21:24.226 +00:00 [INF] - Connection closed CorrelationId a8b2162a-efe9-4559-b16f-dd43878d51d3
<6> 2026-02-12 07:21:24.226 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "https"://"testdevice""""/$iothub/websocket""" - 101 null null 2591836717.0365ms
<4> 2026-02-12 07:21:24.215 +00:00 [WRN] - Websockets close failed CorrelationId d59a41bc-687c-44ba-a11f
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'HttpRequestStream'.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.<ValidateState>g__ThrowObjectDisposedException|14_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Stream.ReadAtLeastAsyncCore(Memory`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.CloseAsyncPrivate(WebSocketCloseStatus closeStatus, String statusDescription, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.ServerWebSocketTransport.CloseInternalAsync(TimeSpan timeout) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/ServerWebSocketTransport.cs:line 244
<6> 2026-02-12 07:21:24.251 +00:00 [INF] - Closed HTTP head
<6> 2026-02-12 07:21:24.432 +00:00 [INF] - Updated reported properties for testdevice/$edgeHub
<6> 2026-02-12 07:21:29.830 +00:00 [INF] - Updated reported properties for testdevice/$edgeHub
<6> 2026-02-12 07:22:09.153 +00:00 [INF] - Stopped MQTT protocol head
<6> 2026-02-12 07:22:09.153 +00:00 [INF] - Protocol heads stopped.
<6> 2026-02-12 07:22:09.188 +00:00 [INF] - Shutdown complete.
<6> 2026-02-12 07:22:09.191 +00:00 [INF] - Termination requested, initiating shutdown.
<6> 2026-02-12 07:22:09.191 +00:00 [INF] - Waiting for cleanup to finish
<6> 2026-02-12 07:22:09.191 +00:00 [INF] - Done with cleanup. Shutting down.
2026-02-12 07:22:20  Starting Edge Hub
2026-02-12 07:22:20  Starting Edge Hub
<6> 2026-02-12 16:24:35.712 +00:00 [INF] - Cloud connection for testdevice/module2 is True
<6> 2026-02-12 16:24:35.713 +00:00 [INF] - Connection status for testdevice/module2 changed to ConnectionEstablished
<6> 2026-02-12 16:24:35.713 +00:00 [INF] - Client testdevice/module2 connected to cloud, processing existing subscriptions.
<6> 2026-02-12 16:24:35.713 +00:00 [INF] - Skipping testdevice/module2 for subscription processing, as it is currently being processed.
<6> 2026-02-12 16:24:35.713 +00:00 [INF] - Created cloud proxy for client testdevice/module2 via AMQP over WebSockets, with client operation timeout 20 seconds.
<6> 2026-02-12 16:24:35.713 +00:00 [INF] - Initialized cloud proxy eae17b27-5b08-4f73-85aa for testdevice/module2
<6> 2026-02-12 16:24:35.713 +00:00 [INF] - Created cloud connection for client testdevice/module2
<6> 2026-02-12 16:24:39.480 +00:00 [INF] - Updated reported properties for testdevice/$edgeHub
<6> 2026-02-12 16:24:39.746 +00:00 [INF] - Updated reported properties for testdevice/module2
<4> 2026-02-12 16:25:26.169 +00:00 [WRN] - No subscription for receiving messages found for testdevice/module2
<6> 2026-02-12 16:25:33.697 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-12 16:25:33.698 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 1.062ms
<4> 2026-02-12 16:26:26.168 +00:00 [WRN] - No subscription for receiving messages found for testdevice/module2
<6> 2026-02-12 16:27:24.910 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-12 16:27:26.166 +00:00 [WRN] - No subscription for receiving messages found for testdevice/module2
<4> 2026-02-12 16:28:26.169 +00:00 [WRN] - No subscription for receiving messages found for testdevice/module2
<4> 2026-02-12 16:29:26.172 +00:00 [WRN] - No subscription for receiving messages found for testdevice/module2
<4> 2026-02-12 16:30:26.173 +00:00 [WRN] - No subscription for receiving messages found for testdevice/module2

2. Error of connection to one module not found after restart of Edge Hub:

Module testdevice/module1 is not connected

<6> 2026-02-02 09:17:56.009 +00:00 [INF] - Restarting process to perform server certificate renewal.
<6> 2026-02-02 09:17:56.010 +00:00 [INF] - Stopping the protocol heads...
<6> 2026-02-02 09:17:56.020 +00:00 [INF] - Closing protocol heads - (MQTT, AMQP, HTTP)
<6> 2026-02-02 09:17:56.024 +00:00 [INF] - Stopping MQTT protocol head
<6> 2026-02-02 09:17:56.052 +00:00 [INF] - Closing HTTP head
<6> 2026-02-02 09:17:56.053 +00:00 [INF] - Closing link Events for testdevice/IoTEdgeMetricsCollector
<6> 2026-02-02 09:17:56.062 +00:00 [INF] - Setting proxy inactive for testdevice/IoTEdgeMetricsCollector.
<6> 2026-02-02 09:17:56.063 +00:00 [INF] - Removing device connection for device testdevice/IoTEdgeMetricsCollector with removeCloudConnection flag 'True'.
<6> 2026-02-02 09:17:56.072 +00:00 [INF] - Closing link Events for testdevice/module1
<6> 2026-02-02 09:17:56.083 +00:00 [INF] - Closing link Events for testdevice/module2
<6> 2026-02-02 09:17:56.087 +00:00 [INF] - Closing link Events for testdevice/module3
<6> 2026-02-02 09:17:56.095 +00:00 [INF] - Closing link TwinSending for testdevice/module1
<6> 2026-02-02 09:17:56.098 +00:00 [INF] - Closing link TwinSending for testdevice/module2
<6> 2026-02-02 09:17:56.099 +00:00 [INF] - Closing link TwinReceiving for testdevice/module1
<6> 2026-02-02 09:17:56.099 +00:00 [INF] - Closing link ModuleMessages for testdevice/module1
<6> 2026-02-02 09:17:56.100 +00:00 [INF] - Setting proxy inactive for testdevice/module1.
<6> 2026-02-02 09:17:56.100 +00:00 [INF] - Removing device connection for device testdevice/module1 with removeCloudConnection flag 'True'.
<6> 2026-02-02 09:17:56.101 +00:00 [INF] - Closing receiver in cloud proxy 81d69f45-9824-44c9-8ee0-4cb7ba2969fe for testdevice/IoTEdgeMetricsCollector
<6> 2026-02-02 09:17:56.101 +00:00 [INF] - Closed cloud proxy 81d69f45-9824-44c9-8ee0-4cb7ba2969fe for testdevice/IoTEdgeMetricsCollector
<6> 2026-02-02 09:17:56.102 +00:00 [INF] - Device connection removed for device testdevice/IoTEdgeMetricsCollector
<6> 2026-02-02 09:17:56.102 +00:00 [INF] - Remove device connection for device testdevice/IoTEdgeMetricsCollector
<6> 2026-02-02 09:17:56.104 +00:00 [INF] - Closing link TwinReceiving for testdevice/module2
<6> 2026-02-02 09:17:56.107 +00:00 [INF] - Closing link MethodSending for testdevice/module2
<6> 2026-02-02 09:17:56.108 +00:00 [INF] - Closing link MethodReceiving for testdevice/module2
<6> 2026-02-02 09:17:56.108 +00:00 [INF] - Setting proxy inactive for testdevice/module2.
<6> 2026-02-02 09:17:56.109 +00:00 [INF] - Removing device connection for device testdevice/module2 with removeCloudConnection flag 'True'.
<6> 2026-02-02 09:17:56.112 +00:00 [INF] - Retrying cloud proxy operation SendMessageAsync for testdevice/module1.
Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Amqp resource is disconnected.
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSendingLink.SendAmqpMessageAsync(AmqpMessage amqpMessage, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTSendingLink.SendMessageAsync(Message message, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.SendMessageAsync(Message message, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.SendEventAsync(Message message, TimeSpan timeout)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass21_0.<<SendEventAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.SendEventAsync(Message message)
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 194
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 146
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 182
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 158
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 159
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 168
   at Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.RetryingCloudProxy.<>c__DisplayClass24_0.<<ExecuteOperation>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/cloud/RetryingCloudProxy.cs:line 63
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.RetryingCloudProxy.ExecuteOperation[T](Func`2 func, String operation) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/cloud/RetryingCloudProxy.cs:line 76
<6> 2026-02-02 09:17:56.203 +00:00 [INF] - Closing link TwinSending for testdevice/module3
<6> 2026-02-02 09:17:56.204 +00:00 [INF] - Attempting to connect to IoT Hub for client testdevice/module1 via AMQP over WebSockets...
<6> 2026-02-02 09:17:56.204 +00:00 [INF] - Closing link TwinReceiving for testdevice/module3
<6> 2026-02-02 09:17:56.204 +00:00 [INF] - Setting proxy inactive for testdevice/module3.
<6> 2026-02-02 09:17:56.206 +00:00 [INF] - Removing device connection for device testdevice/module3 with removeCloudConnection flag 'True'.
<6> 2026-02-02 09:17:56.206 +00:00 [INF] - Closing receiver in cloud proxy c81c89d9-1b88-4a03-a9ac-d341989b6bef for testdevice/module3
<6> 2026-02-02 09:17:56.206 +00:00 [INF] - Closed cloud proxy c81c89d9-1b88-4a03-a9ac-d341989b6bef for testdevice/module3
<6> 2026-02-02 09:17:56.207 +00:00 [INF] - Device connection removed for device testdevice/module3
<6> 2026-02-02 09:17:56.207 +00:00 [INF] - Remove device connection for device testdevice/module3
<6> 2026-02-02 09:17:56.210 +00:00 [INF] - Connection closed CorrelationId 6ac8a228-bbf7-4076-9234
<6> 2026-02-02 09:17:56.211 +00:00 [INF] - Connection closed CorrelationId 2530ece6-1896-4577-b8ea
<6> 2026-02-02 09:17:56.215 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "https"://"testdevice""""/$iothub/websocket""" - 101 null null 2591837481.1931ms
<6> 2026-02-02 09:17:56.216 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "https"://"testdevice""""/$iothub/websocket""" - 101 null null 2591836677.3995ms
<4> 2026-02-02 09:17:56.235 +00:00 [WRN] - Websockets close failed CorrelationId 2530ece6-1896-4577-b8ea-8bd49b32c94f
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'HttpRequestStream'.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.<ValidateState>g__ThrowObjectDisposedException|14_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Stream.ReadAtLeastAsyncCore(Memory`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.CloseAsyncPrivate(WebSocketCloseStatus closeStatus, String statusDescription, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.ServerWebSocketTransport.CloseInternalAsync(TimeSpan timeout) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/ServerWebSocketTransport.cs:line 244
<6> 2026-02-02 09:17:56.241 +00:00 [INF] - Closing receiver in cloud proxy 60b519ea-7f3c-4c65-89ca for testdevice/module2
<6> 2026-02-02 09:17:56.241 +00:00 [INF] - Closed cloud proxy 60b519ea-7f3c-4c65-89ca for testdevice/module2
<6> 2026-02-02 09:17:56.241 +00:00 [INF] - Device connection removed for device testdevice/module2
<6> 2026-02-02 09:17:56.241 +00:00 [INF] - Remove device connection for device testdevice/module2
<6> 2026-02-02 09:17:56.241 +00:00 [INF] - Closing receiver in cloud proxy d261f882-ec54-47f6-9fb3 for testdevice/module1
<6> 2026-02-02 09:17:56.241 +00:00 [INF] - Closed cloud proxy d261f882-ec54-47f6-9fb3-5720b695eb5b for testdevice/module1
<4> 2026-02-02 09:17:56.235 +00:00 [WRN] - Websockets close failed CorrelationId 6ac8a228-bbf7-4076-9234
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'HttpRequestStream'.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.<ValidateState>g__ThrowObjectDisposedException|14_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Stream.ReadAtLeastAsyncCore(Memory`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.CloseAsyncPrivate(WebSocketCloseStatus closeStatus, String statusDescription, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.ServerWebSocketTransport.CloseInternalAsync(TimeSpan timeout) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/ServerWebSocketTransport.cs:line 244
<6> 2026-02-02 09:17:56.242 +00:00 [INF] - Device connection removed for device testdevice/module1
<6> 2026-02-02 09:17:56.242 +00:00 [INF] - Remove device connection for device testdevice/module1
<6> 2026-02-02 09:17:56.248 +00:00 [INF] - Connection closed CorrelationId e349f210-29f7-430e
<6> 2026-02-02 09:17:56.250 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "https"://"testdevice""""/$iothub/websocket""" - 101 null null 2591838266.5222ms
<4> 2026-02-02 09:17:56.255 +00:00 [WRN] - Websockets close failed CorrelationId e349f210-29f7-430e-8c2b-da4f69538cde
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'HttpRequestStream'.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestPipeReader.<ValidateState>g__ThrowObjectDisposedException|14_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Stream.ReadAtLeastAsyncCore(Memory`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.WebSockets.ManagedWebSocket.CloseAsyncPrivate(WebSocketCloseStatus closeStatus, String statusDescription, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.ServerWebSocketTransport.CloseInternalAsync(TimeSpan timeout) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Amqp/ServerWebSocketTransport.cs:line 244
<6> 2026-02-02 09:17:56.273 +00:00 [INF] - Closed HTTP head
<6> 2026-02-02 09:17:56.855 +00:00 [INF] - Cloud connection for testdevice/module1 is True
<6> 2026-02-02 09:17:56.855 +00:00 [INF] - Connection status for testdevice/module1 changed to ConnectionEstablished
<6> 2026-02-02 09:17:56.855 +00:00 [INF] - Client testdevice/module1 connected to cloud, processing existing subscriptions.
<6> 2026-02-02 09:17:56.856 +00:00 [INF] - Created cloud proxy for client testdevice/module1 via AMQP over WebSockets, with client operation timeout 20 seconds.
<6> 2026-02-02 09:17:56.856 +00:00 [INF] - Initialized cloud proxy 83dbdacc-ea4c-4dbb-9f2c for testdevice/module1
<6> 2026-02-02 09:17:56.856 +00:00 [INF] - Created cloud connection for client testdevice/module1
<6> 2026-02-02 09:17:57.075 +00:00 [INF] - Updated reported properties for testdevice/$edgeHub
<6> 2026-02-02 09:18:02.386 +00:00 [INF] - Updated reported properties for testdevice/$edgeHub
<6> 2026-02-02 09:18:41.166 +00:00 [INF] - Stopped MQTT protocol head
<6> 2026-02-02 09:18:41.166 +00:00 [INF] - Protocol heads stopped.
<6> 2026-02-02 09:18:41.194 +00:00 [INF] - Shutdown complete.
<6> 2026-02-02 09:18:41.197 +00:00 [INF] - Termination requested, initiating shutdown.
<6> 2026-02-02 09:18:41.197 +00:00 [INF] - Waiting for cleanup to finish
<6> 2026-02-02 09:18:41.197 +00:00 [INF] - Done with cleanup. Shutting down.
2026-02-02 09:18:55  Starting Edge Hub
2026-02-02 09:18:55  Starting Edge Hub
<6> 2026-02-02 09:39:00.350 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 09:39:03.199 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:40:03.198 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:41:03.202 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:42:01.990 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 09:42:01.992 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 1.1886ms
<4> 2026-02-02 09:42:03.203 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:43:03.204 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:44:00.347 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 09:44:03.204 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:45:03.207 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:46:03.206 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:47:02.015 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 09:47:02.025 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 10.5143ms
<4> 2026-02-02 09:47:03.205 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:48:03.208 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:48:59.640 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint iothub
<6> 2026-02-02 09:48:59.902 +00:00 [INF] - Cleaned up 596 messages from queue for endpoint iothub and 596 messages from message store.
<6> 2026-02-02 09:48:59.902 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint testdevice/module1/machine_msg
<6> 2026-02-02 09:49:00.145 +00:00 [INF] - Cleaned up 717 messages from queue for endpoint testdevice/module1/machine_msg and 717 messages from message store.
<6> 2026-02-02 09:49:00.352 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 09:49:03.209 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:50:03.210 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:51:03.210 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:52:02.045 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 09:52:02.059 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 13.6758ms
<4> 2026-02-02 09:52:03.215 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:53:03.215 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:54:00.350 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 09:54:03.218 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:55:03.220 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:56:03.218 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:57:02.095 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 09:57:02.099 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 4.1774ms
<4> 2026-02-02 09:57:03.224 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 09:58:03.229 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 09:59:00.350 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 09:59:03.232 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:00:03.235 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:01:03.239 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 10:02:02.123 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 10:02:02.139 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 16.0334ms
<4> 2026-02-02 10:02:03.240 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:03:03.243 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 10:04:00.350 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 10:04:03.245 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:05:03.243 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:06:03.246 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 10:07:02.157 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 10:07:02.181 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 23.3139ms
<4> 2026-02-02 10:07:03.249 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:08:03.247 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 10:09:00.347 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 10:09:03.247 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:10:03.248 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:11:03.252 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 10:12:02.203 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 10:12:02.205 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 1.8521ms
<4> 2026-02-02 10:12:03.250 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:13:03.253 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 10:13:03.374 +00:00 [INF] - New token received on the Cbs link
<6> 2026-02-02 10:13:03.375 +00:00 [INF] - Token updated for testdevice/IoTEdgeMetricsCollector
<6> 2026-02-02 10:14:00.351 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2026-02-02 10:14:03.256 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:15:03.255 +00:00 [WRN] - Module testdevice/module1 is not connected
<4> 2026-02-02 10:16:03.259 +00:00 [WRN] - Module testdevice/module1 is not connected
<6> 2026-02-02 10:17:02.235 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - null null
<6> 2026-02-02 10:17:02.258 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "http"://"edgehub:9600""""/metrics""" - 200 null "text/plain; version=0.0.4; charset=utf-8" 22.8807ms
<4> 2026-02-02 10:17:03.258 +00:00 [WRN] - Module testdevice/module1 is not connected```

[malavvvakharia]

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

The IoT Edge device is set up for receiving messages
Messages are received for 30 days without any interruptions
After 30 days messages can't be received due to the error that happens after automatic server certificate renewal.
It starts the edgeHub module but custom module run it as ideal state and did not receive any messages.

I also found same bug: #6044

Here I am also not sure why this certificate will always auto renew after 30 days even certificate has longer validity.

[malavvvakharia]

I finally understand all the details about what’s happening, but I couldn’t find any relevant documentation on the issue.

In my case, since I didn’t specify anything related to the Edge CA in the config.toml file, it worked with the default configuration mentioned in the documentation: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-manage-device-certificates?tabs=ubuntu#quickstart-edge-ca, which is acceptable. However, I now have a problem with the EdgeHub server certificate, which is automatically generated by Microsoft and has a validity of only 30 days.

Since my device has been running for 30 days, it has initiated the process of renewing the EdgeHub server certificate, leading to a restart of the EdgeHub module. The issue arises because when the EdgeHub restarts and no other modules do, it loses connection to one of my custom modules and is unable to find the subscription or route for that module.

I am uncertain if I need to register the input again using moduleClientService.ConfigureInputMessageHandlerAsync via the SDK, but in one instance, the module was still not connected. To me, this seems like a bug, as I can’t find any documentation indicating that you need to register the input again.

@scb01 @arsing

[vipeller]

@malavvvakharia If I understand correctly, when edge hub restarts it does not have connectivity to iot hub (I am not sure that this due to some problem or that is how the setup is intended to work)

As you write correctly, in this case edge hub is not able to pull the routing information.

However, edge hub can be configured to save its configuration, so when it cannot reach iot hub during startup, it can load the config from the backup. (I need to check how to turn that on, though)

Your module clients don't need to do anything special - I think the problem is the missing routing info.

[malavvvakharia]

@vipeller , thanks for the update.

As you write correctly, in this case edge hub is not able to pull the routing information.

However, edge hub can be configured to save its configuration, so when it cannot reach iot hub during startup, it can load the config from the backup. (I need to check how to turn that on, though)

Could you please let me know how to configure the edgeHub to switch to the backup routing?

I noticed something interesting: when the Edge Hub renews the server certificate, it shuts down and restarts, which also sends a disconnect event. This, I believe, causes our retry mechanism to stop. Is this correct? Once the Edge Hub shuts down, it provides a disconnected status, and the retry policy does not work at this point. It only works when we receive a disconnect_retrying status. For your information, we are using a custom retry policy with the IRetryPolicy from Microsoft.Azure.Devices.Client.

Hack: Continuously sending messages will automatically reconnect the custom module to edgeHub.

I believe that the restart we perform using the command "iotedge restart edgeHub" works differently from the restart that occurs due to the renewal of the edgeHub server certificate.