Add sub metrics for conversation correctly finished - #191
Conversation
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| from eva.metrics.processor import is_agent_timeout_on_user_turn |
There was a problem hiding this comment.
i think we should move is_agent_timeout_on_user_turn to utils instead of importing from eva.metrics since the metrics already import utils or compute is_parent_failure in the metric itself
| - ``acknowledgement``: leads with a confirmation/ack ("Yes, that is correct.", "Ok thanks", | ||
| "[neutral] Okay.", "No.") AND is at most ``_ACKNOWLEDGEMENT_MAX_WORDS`` words — a long sentence | ||
| that merely starts "yeah …" is not counted. | ||
| - ``spelled_entity``: letter/digit spell-out of an ID/code/name ("E M P eight nine …", NATO, |
There was a problem hiding this comment.
i think the NATO alphabet check is a bit loose, eg "hotel" or "november". Like i think "I am waiting in the hotel lobby" or "I need a flight that leaves November 6th." would get flagged incorrectly
| signals = extract_conv_finish_signals(context) | ||
| classification = classify_conv_finish_failure(signals) | ||
| sub_metrics = build_conv_finish_sub_metrics(classification, self.name) | ||
| # Orthogonal input-characteristic flags (short / acknowledgement / spelled final turn). | ||
| sub_metrics.update(build_final_turn_flag_sub_metrics(signals.user_final_words, self.name)) |
There was a problem hiding this comment.
this section should likely have it's own try-catch, otherwise if something fails in here, we lose the other calculated info like details.reason or details.speaker
|
@katstankiewicz thank you for the feedback! I should have addressed all of your comments. |
2 participants
No description provided.