Add three types of validators#47
Conversation
Temerius
commented
Mar 17, 2025
Temerius
commented
- Add method call_with_validate
- Add base abstract class Validator
- Add JSON-, XML-, YAMLValidator classes
- Add tests for each validator
| for error in validator.get_errors(): | ||
| validation_errors.append({ | ||
| "iteration": iteration, | ||
| "error": validator.format_error(error) |
There was a problem hiding this comment.
we need to add here validator's id/name
| "iteration": iteration, | ||
| "error": validator.format_error(error) | ||
| }) | ||
| if validator.can_retry(): |
There was a problem hiding this comment.
I like that you made that logic isolated
| def __init__(self, validation_format: t.Dict): | ||
| super().__init__("json", validation_format) | ||
|
|
||
| def validate(self, response: AIResponse): |
There was a problem hiding this comment.
can you add in AIResponse method json_from_response - which is cached property, same for yaml, xml
There was a problem hiding this comment.
Do you mean that I should add a method similar to _get_format_from_response in AIResponse, so that I can simply access the property and get the content in JSON, XML, or YAML format from response? The method should also return all found JSON, XML, and YAML.
| [Tag("```json", "\n```", 0), Tag("```json", "```", 0)], | ||
| start_from=0 | ||
| ) | ||
| if content: |
There was a problem hiding this comment.
in that sense, let's get all jsons, and check each json;
add another method for json_from_response, which will return an array of jsons (dict or arrays)
Katsiarynka
left a comment
Katsiarynka
left a comment
There was a problem hiding this comment.
Ideally, we need to think:
- How the user will add to that specific prompt validators; It should be an easy way;
- Let's rename to
call_and_validate, it's an action which will happen; - Can you add a real test with JSON, Yaml, XML, we have an integrational tests;
- In that method - call_and_validate - we don't need to add validators; They need to be already defined to the prompts; We have defined glabal variable like PROMPTS, we can add PROMPT_VALIDATORS; - which will have all these formats;
- Let's delete strange schemas from required parameters;
- Validator should be a class, and easily be created, like for any not specialist in the programming,
- additional fields from the json should be passed, so we shouldn't fail if json has more fields, let's add it in the test;
- Add in readme a documentation for validators;
- AIResponse method json_from_response, jsons_from_response, ...xml, yaml;
- DRY (Don't Repeat Yourself)
- create additional ValidatorException
- Use functions, f.e. for iteration in range: _make_an_attempt()
|
|
||
| if content: | ||
| try: | ||
| yaml_data = yaml.safe_load(content) |
There was a problem hiding this comment.
DRY (dont repeat yourself). Let's create a class as Formatalidator, which will be a parent of each classes JSON, YAML, XML?
| json_format = { | ||
| "validator_type": "json", | ||
| "schema": { | ||
| "$schema": "http://json-schema.org/draft-07/schema#", |
There was a problem hiding this comment.
what is that schema for? If you have qn, then everyone else. Be ready to make sure that everything for you makes sense, feels just stupid repetition of the doc. We can obscure that schema parameter
|
|
||
| class Validator(ABC): | ||
| def __init__(self, prompt_id: str, validator_id: str, validator_type: str, shema: t.Dict, retry_count: int = 0, retry_rules: t.Optional[t.Dict]= None): | ||
| self.prompt_id = prompt_id |
There was a problem hiding this comment.
do we need to have here prompt_id?
I believe another variable should store for prompt all validators
2 participants
