-
Notifications
You must be signed in to change notification settings - Fork 17.5k
Restore argument-provision checks to __init__ in common.ai operators #70628
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
baha-bouali
wants to merge
13
commits into
apache:main
Choose a base branch
from
baha-bouali:fix/restore-common-ai-to-init-
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
71ce9e2
Restore __init__ validation for common AI operators
baha-bouali 9d03505
Merge branch 'main' into fix/restore-common-ai-to-init-
baha-bouali 47531bb
Guard rendered source_path against None; fix misleading comment
baha-bouali af4d38f
Guard rendered source_path against None; fix misleading comment
baha-bouali f6d4946
Merge branch 'main' into fix/restore-common-ai-to-init-
baha-bouali c52aabd
Add test for None source_path in DocumentLoaderOperator
baha-bouali 0c74e34
Merge branch 'main' into fix/restore-common-ai-to-init-
baha-bouali be01d18
Merge branch 'main' into fix/restore-common-ai-to-init-
baha-bouali 48a6274
Merge branch 'main' into fix/restore-common-ai-to-init-
baha-bouali a4ec133
static checks: delete trailing spaces in a comment
baha-bouali d04ba43
static checks: add line break before a function
baha-bouali dd6b729
Merge branch 'main' into fix/restore-common-ai-to-init-
baha-bouali 3f52378
Merge branch 'fix/restore-common-ai-to-init-' of https://github.com/b…
baha-bouali File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -136,6 +136,10 @@ def __init__( | |
| **kwargs: Any, | ||
| ) -> None: | ||
| super().__init__(**kwargs) | ||
| if source_path is not None and source_bytes is not None: | ||
| raise ValueError("Provide exactly one of 'source_path' or 'source_bytes', not both.") | ||
| if source_path is None and source_bytes is None: | ||
| raise ValueError("Provide exactly one of 'source_path' or 'source_bytes'.") | ||
| self.source_path = source_path | ||
| self.source_conn_id = source_conn_id | ||
| self.source_bytes = source_bytes | ||
|
|
@@ -148,13 +152,15 @@ def __init__( | |
| self.json_text_field = json_text_field | ||
|
|
||
| def execute(self, context: Context) -> list[dict[str, Any]]: | ||
| # source_path/file_type are template fields; validate after rendering, not in __init__. | ||
| if self.source_path is not None and self.source_bytes is not None: | ||
| raise ValueError("Provide exactly one of 'source_path' or 'source_bytes', not both.") | ||
| if self.source_path is None and self.source_bytes is None: | ||
| raise ValueError("Provide exactly one of 'source_path' or 'source_bytes'.") | ||
| # file_type and source_path can each be *supplied* (as non-None argument) yet still | ||
| # render to None. These aren't provision checks (that already happened in __init__); | ||
| # they guard the rendered value itself, since _parse_bytes/_resolve_files need a real | ||
| # value to work with. Checking this in __init__ would validate the unrendered template | ||
| # string instead of the value actually used here. | ||
| if self.source_bytes is not None and self.file_type is None: | ||
| raise ValueError("'file_type' is required when using 'source_bytes' (e.g. '.pdf').") | ||
| if self.source_bytes is None and self.source_path is None: | ||
| raise ValueError("Provide exactly one of 'source_path' or 'source_bytes'.") | ||
|
Comment on lines
+162
to
+163
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you please add a unit test for this case?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done! I added the unit test right in |
||
|
|
||
| if self.source_bytes is not None: | ||
| if TYPE_CHECKING: | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
file_typekeeps a rendered-value guard here, butsource_pathno longer has one, and theassert self.source_path is not Nonebelow was leaning on the check that just moved to__init__.A supplied
source_paththat renders toNone(the native-obj case #70505 is built around) now falls into theelsebranch, and_resolve_files(None)raisesTypeError: argument of type 'NoneType' is not iterableinstead of the previous clearValueError. Worth keeping aself.source_path is Noneguard here next to thefile_typeone.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is fixed now. I reworded the comment and added back the source_path guard, thanks for catching this! Let me know if something is still wrong