CAMEL-14199 : camel-hdfs - Add maxMessagesPerPoll for Consumer#3356
Conversation
update to latest
merge latest
merge master
Update to latest
update to latest
get latest
update to latest
update to latest
update to latest
update to latest
# Conflicts: # components/camel-aws-cw/src/main/java/org/apache/camel/component/aws/cw/CwConfiguration.java # components/camel-aws-ddb/src/main/java/org/apache/camel/component/aws/ddb/DdbConfiguration.java # components/camel-aws-ddb/src/main/java/org/apache/camel/component/aws/ddbstream/DdbStreamConfiguration.java # components/camel-aws-s3/src/main/java/org/apache/camel/component/aws/s3/S3Configuration.java
# Conflicts: # components/camel-aws-ec2/src/main/java/org/apache/camel/component/aws/ec2/EC2Configuration.java # components/camel-aws-ecs/src/main/java/org/apache/camel/component/aws/ecs/ECSConfiguration.java # components/camel-aws-eks/src/main/java/org/apache/camel/component/aws/eks/EKSConfiguration.java # components/camel-aws-iam/src/main/java/org/apache/camel/component/aws/iam/IAMConfiguration.java # components/camel-aws-kinesis/src/main/java/org/apache/camel/component/aws/firehose/KinesisFirehoseConfiguration.java # components/camel-aws-kinesis/src/main/java/org/apache/camel/component/aws/kinesis/KinesisConfiguration.java # components/camel-aws-kms/src/main/java/org/apache/camel/component/aws/kms/KMSConfiguration.java # components/camel-aws-lambda/src/main/java/org/apache/camel/component/aws/lambda/LambdaConfiguration.java # components/camel-aws-mq/src/main/java/org/apache/camel/component/aws/mq/MQConfiguration.java # components/camel-aws-msk/src/main/java/org/apache/camel/component/aws/msk/MSKConfiguration.java # components/camel-aws-sdb/src/main/java/org/apache/camel/component/aws/sdb/SdbConfiguration.java # components/camel-aws-ses/src/main/java/org/apache/camel/component/aws/ses/SesConfiguration.java # components/camel-aws-sns/src/main/java/org/apache/camel/component/aws/sns/SnsConfiguration.java # components/camel-aws-sqs/src/main/java/org/apache/camel/component/aws/sqs/SqsConfiguration.java # components/camel-aws-translate/src/main/java/org/apache/camel/component/aws/translate/TranslateConfiguration.java
|
Is this depending on the other PR of yours #3354 ? |
|
Thanks. Is there not a way to limit directly on hadoop file system to tell it to not give all file status, but only X number? Also I forgot how it works about avoiding processing the same files again on next pool. So if you limit to 10, and there are a total of 65 files. Then pool #1 will process 10 files, then pool #2 will it then know that the previous 10 was processed, or would hadoop return 55 files now? |
These are independent, though touch the same areas of code. |
[MC] Not with the 2.x hadoop version; I didn't check with 3.x; This is just a file reference, so memory impact should be minimal. Also, the FileStatus[] is only filtered by pattern, then an extra check for owner is done (hadoop client has no other api to get only for owner). It may be so that out of the 100 matched files, only 10 are for that owner. So this is a more greedy fetch.
[MC] Files are renamed with the ".OPEN" extension then to the ".READ" extension; So it works as you would expect even without an idempotent or inprogress repo. |
|
ok thanks |
3 participants
CAMEL-14199 : camel-hdfs - Add maxMessagesPerPoll for Consumer