HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled#5517
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Recon NSSummaries currently support
FSO,OBSbuckets, andLegacybuckets with theFSEnabledflag set totrue. WhenFSEnabledisfalse, Legacy buckets act like OBS buckets, requiring a similar approach for managing keys. We've adapted our OBS bucket handling methods for these cases, maximizing code reuse.Here's a breakdown of the changes :-
NSSummaryTaskWithLegacy.java: This includes methods to process keys in Legacy buckets, distinguishing between filesystem and object store layouts to generate NSSummaries accurately.BucketHandler.java: Depending on whether FileSystemPaths is enabled, we assign the appropriate handler for each bucket. For Legacy buckets with disabled FileSystemPaths, we employ OBS bucket handler, thanks to their identical key management approach.EntityHandler.java: This class is activated by theNSSummaryEndpointand determines which handler to assign based on the path provided by the user. For Legacy buckets with the FileSystemPaths option disabled, we directly employ the OBSHandler and its associated methods. We have introduced specific normalization methods for OBS keys. OBS keys follow the formatvolumeName/bucketName/keyName, where the keyName can include any characters, even multiple slashes, such as “////KeyName.” There's no need to normalize the keyName itself, but it's essential to ensure the path up to the bucket is correctly normalized. This is because volumeName and bucketName must not contain multiple slashes or spaces. Our standard normalization methods for the FSO layout normalize the entire path, including the keyName, which could alter the keyName and lead to invalid output. To address this, I've added a method in the OmUtils.java class that normalizes the path up to the bucket level.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9534
How was this patch tested?
Manual Testing and Unit Testing