Skip to content

AzureStorageOrchestrationService fails to store an event once OrchestrationStatus gets bigger #1185

Description

@dsempi

If the orchestration status is a more complex class, that serializes into a larger string, the following exception is thrown and orchestration does not proceed further:

DurableTask.Core[29]
TaskOrchestrationDispatcher-091002dd69a645bb89247b8f24b44b98-0: Unhandled exception with work item '8652685d247d44589559f13e3096234e': DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
 ---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge

Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}

Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8

   at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
   at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
   at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

Backing off for 10 seconds until 5 successful operations
DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
 ---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge

Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}

Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8

   at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
   at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
   at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

This limitation is not present for ServiceFabric and Emulator.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions