inconsistent trace-indicator formed for rare scenario of /v1/relay-server-replacement

[IswaryaaS]

While testing /v1/relay-server-replacement, found one weird scenario, where the trace-indictor was created as per expectation in all cases except the below case.

Case 1 creates same trace-indicator for RO and TAR clients whereas case 2 creates a correct one (unique trace-indicator for each)

Case 1:

{
                        "uuid": "ro-2-0-1-op-fc-is-009",
                        "name": [
                            {
                                "value-name": "ForwardingKind",
                                "value": "core-model-1-4:FORWARDING_KIND_TYPE_SUBSCRIPTION"
                            },
                            {
                                "value-name": "ForwardingName",
                                "value": "ServerReplacementBroadcast"
                            }
                        ],
                        "fc-port": [
                            {
                                "local-id": "000",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_MANAGEMENT",
                                "logical-termination-point": "ro-2-0-1-op-s-is-003"
                            },
                            {
                                "local-id": "001",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_MANAGEMENT",
                                "logical-termination-point": "ro-2-0-1-op-s-is-002"
                            },
                            {
                                "local-id": "100",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_INPUT",
                                "logical-termination-point": "ro-2-0-1-op-s-is-010"
                            },
                            {
                                "local-id": "200",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-ro-2-0-1-001"
                            },
                            {
                                "local-id": "201",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-tar-1-0-0-001"
                            },
                            {
                                "local-id": "202",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-eatl-1-0-0-001"
                            },
                            {
                                "local-id": "203",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-ol-1-0-0-001"
                            },
                            {
                                "local-id": "204",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-aa-1-0-0-001"
                            },
                            {
                                "local-id": "205",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-alt-1-0-0-001"
                            },
                            {
                                "local-id": "206",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-okm-1-0-0-001"
                            },
                            {
                                "local-id": "207",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-cc-1-0-0-001"
                            },
                            {
                                "local-id": "208",
                                "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
                                "logical-termination-point": "ro-2-0-1-op-c-im-dan-2-0-1-001"
                            }
                        ]
                    }

Sample log flow for case 1: where two instances having same trace-indicator 6.1

{
    "x-correlator": "510e5200-e11b-41d4-a716-516655641124",
    "trace-indicator": "6",
    "user": "User Name",
    "originator": "Resolver",
    "application-name": "RegistryOffice",
    "release-number": "2.0.1",
    "operation-name": "/v1/relay-server-replacement",
    "response-code": 204,
    "timestamp": "2023-06-13T14:31:40+02:00",
    "stringified-body": "{\"current-application-name\":\"ApplicationLayerTopology\",\"current-release-number\":\"1.0.0\",\"future-application-name\":\"ApplicationLayerTopology\",\"future-release-number\":\"2.0.1\",\"future-protocol\":\"HTTP\",\"future-address\":{\"ip-address\":{\"ipv-4-address\":\"127.0.0.1\"}},\"future-port\":3005}"
  },
  {
    "x-correlator": "510e5200-e11b-41d4-a716-516655641124",
    "trace-indicator": "6.1",
    "user": "User Name",
    "originator": "RegistryOffice",
    "application-name": "RegistryOffice",
    "release-number": "2.0.1",
    "operation-name": "/v1/update-client",
    "response-code": 204,
    "timestamp": "2023-06-13T14:31:40+02:00",
    "stringified-body": "{\"current-application-name\":\"ApplicationLayerTopology\",\"current-release-number\":\"1.0.0\",\"future-application-name\":\"ApplicationLayerTopology\",\"future-release-number\":\"2.0.1\",\"future-protocol\":\"HTTP\",\"future-address\":{\"ip-address\":{\"ipv-4-address\":\"127.0.0.1\"}},\"future-port\":3005}"
  },
  {
    "x-correlator": "510e5200-e11b-41d4-a716-516655641124",
    "trace-indicator": "6.1",
    "user": "User Name",
    "originator": "RegistryOffice",
    "application-name": "TypeApprovalRegister",
    "release-number": "2.0.1",
    "operation-name": "/v1/update-client",
    "response-code": 204,
    "timestamp": "2023-06-13T14:31:40+02:00",
    "stringified-body": "{\"current-application-name\":\"ApplicationLayerTopology\",\"current-release-number\":\"1.0.0\",\"future-application-name\":\"ApplicationLayerTopology\",\"future-release-number\":\"2.0.1\",\"future-protocol\":\"HTTP\",\"future-address\":{\"ip-address\":{\"ipv-4-address\":\"127.0.0.1\"}},\"future-port\":3005}"
  },
  {
    "x-correlator": "510e5200-e11b-41d4-a716-516655641124",
    "trace-indicator": "6.2",
    "user": "User Name",
    "originator": "RegistryOffice",
    "application-name": "ExecutionAndTraceLog",
    "release-number": "2.0.1",
    "operation-name": "/v1/update-client",
    "response-code": 204,
    "timestamp": "2023-06-13T14:31:40+02:00",
    "stringified-body": "{\"current-application-name\":\"ApplicationLayerTopology\",\"current-release-number\":\"1.0.0\",\"future-application-name\":\"ApplicationLayerTopology\",\"future-release-number\":\"2.0.1\",\"future-protocol\":\"HTTP\",\"future-address\":{\"ip-address\":{\"ipv-4-address\":\"127.0.0.1\"}},\"future-port\":3005}"
  },

Case 2:

{
    "uuid": "ro-2-0-1-op-fc-is-009",
    "name": [
        {
            "value-name": "ForwardingKind",
            "value": "core-model-1-4:FORWARDING_KIND_TYPE_SUBSCRIPTION"
        },
        {
            "value-name": "ForwardingName",
            "value": "ServerReplacementBroadcast"
        }
    ],
    "fc-port": [
        {
            "local-id": "000",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_MANAGEMENT",
            "logical-termination-point": "ro-2-0-1-op-s-is-003"
        },
        {
            "local-id": "001",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_MANAGEMENT",
            "logical-termination-point": "ro-2-0-1-op-s-is-002"
        },
        {
            "local-id": "100",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_INPUT",
            "logical-termination-point": "ro-2-0-1-op-s-is-010"
        },
        {
            "local-id": "201",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-tar-1-0-0-001"
        },
        {
            "local-id": "202",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-eatl-1-0-0-001"
        },
        {
            "local-id": "203",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-ol-1-0-0-001"
        },
        {
            "local-id": "204",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-aa-1-0-0-001"
        },
        {
            "local-id": "205",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-alt-1-0-0-001"
        },
        {
            "local-id": "206",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-okm-1-0-0-001"
        },
        {
            "local-id": "207",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-cc-1-0-0-001"
        },
        {
            "local-id": "208",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-da-2-0-1-001"
        },
        {
            "local-id": "209",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-dan-2-0-1-001"
        },
        {
            "local-id": "210",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-nr-2-0-1-001"
        },
        {
            "local-id": "211",
            "port-direction": "core-model-1-4:PORT_DIRECTION_TYPE_OUTPUT",
            "logical-termination-point": "ro-2-0-1-op-c-im-ro-2-0-3-001"
        }
    ]
}

Priority:Low;
AffectedFlow:None;

[DanaSunal]

Cause: ROs operation relay-server-replacement sends forwardings to all applications with 'update-client' operation. Since RO is first in the list of forwardings and we are not waiting for response, update-client on RO starts to execute. Update-client has also forwardings. Which means a second call to automateForwardingConstructAsync (within RO, within this one API call) which resets traceIndicatorIncrementer back to 1.