Remove a note about cloudpickle support being experimental#34199
Closed
tvalentyn wants to merge 2952 commits intoapache:cloudpicklefrom
Closed
Remove a note about cloudpickle support being experimental#34199tvalentyn wants to merge 2952 commits intoapache:cloudpicklefrom
tvalentyn wants to merge 2952 commits intoapache:cloudpicklefrom
Conversation
… aggregate a collection of string sequences (FQNs) with a limited size.
… mutable BoundedTrieData
…in fuzzy run tests
…l Dataflow java client support is ready
* Add python ml extras * Create clear bounds on transformers suite and add latest test * Add docs
Introduce a BoundedTrie metric which is used to efficiently store and aggregate a collection of string sequences (FQNs) with a limited size.
It is recommended to review this PR by commits.
BoundedTrie is a space-saving way to store many string sequences (like FQN/file paths). It acts like a tree with branches, holding sequences within a size limit. It can efficiently add, combine, and search and perform trimming of children when the size increases beyond defined max.
Let's say we want to store these sequences, with a size limit of 3:
"folder1/file1.txt"
"folder1/file2.txt"
"folder2/file3.txt"
Here's how the BoundedTrie might look:
root
- folder1
- file1.txt
- file2.txt
- folder2
- file3.txt
If we try to add "folder1/file4.txt", the trie might trim to "folder1", dropping all children to stay within the size limit.
This will be used to replace the StringSet metric for lineage tracking for very large lineage graphs to overcome the size limits.
* The Bag Partition is now configurable. Configuring the number of partitions in the Dask runner is very important to tune performance. This CL gives users control over this parameter. * Apply formatter. * Passing lint via the `run_pylint.sh` script. * Implementing review feedback. * Attempting to pass lint/fmt check. * Fixing isort issues by reading CI output. * More indentation. * rm blank like for isort.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.