[Bug] Make sure `is_replaying` is `True` upon eviction, and only throw certain exception types out on eviction

[cretz]

Describe the bug

Today, anyone in a finally or except Exception can catch exceptions occurring during eviction and do side-effecting things like logging. We need to set is_replaying to True during eviction (or maybe Core does) and we need to make a new class WorkflowEvictingException(BaseException) that we use when we see stuff happening during delete.

[jpcarlino]

@cretz I recently upgraded our pipeline workers to use the Temporal SDK 1.6.0. Since then I see this error being printed in our logs: Ignoring add while deleting, which seems to come from this line. I see you did change it to this.

My question is: should I catch that exception and ignore it? If not, what should I do on my side to avoid that. Unfortunately it is flooding our logs right now. Thank you.

[cretz]

Yes, this issue is meant to fix that problem because logging should not occur during is_replaying. In the meantime you can ignore these exceptions, they are only on cache eviction.