Skip to content

Modify AuditReplay workflow to output count and latency of operations#92

Merged
xkrogen merged 5 commits into
linkedin:masterfrom
csgregorian:count_and_time
Mar 14, 2019
Merged

Modify AuditReplay workflow to output count and latency of operations#92
xkrogen merged 5 commits into
linkedin:masterfrom
csgregorian:count_and_time

Conversation

@csgregorian

@csgregorian csgregorian commented Mar 14, 2019

Copy link
Copy Markdown
Contributor

Created a composite output type, CountTimeWritable, to track the count in addition to the cumulative time of operations. As well, adds another level of granularity through the operation name instead of just the type. This gives more insight into per-operation behaviour, and makes it possible to calculate the average latency per operation instead of just the total.

EDIT: also changes the separator to a , for better tabular representation

The output from a workflow changes as follows:

Before

hdfs,WRITE	590

After

hdfs,WRITE,CREATE,1,565
hdfs,WRITE,MKDIRS,7,14
hdfs,WRITE,RENAME,1,11

@xkrogen xkrogen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we change the TextOutputFormat.SEPERATOR config to be a , so that we end up with CSV output instead of half-CSV, half-tabulated?

Other than that, and my minor comments, LGTM

@csgregorian

Copy link
Copy Markdown
Contributor Author

Fixed up, edited PR to match.

@xkrogen xkrogen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few more minor comments

@csgregorian

Copy link
Copy Markdown
Contributor Author

Fixed up 👍

@xkrogen xkrogen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@xkrogen xkrogen merged commit 66d3e19 into linkedin:master Mar 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants