Skip to content

[SPARK-17671] Changed implementation of HistoryServer.getApplicationInfoList for lazy evaluation #15303

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wgtmac wants to merge 3 commits into from
Closed

[SPARK-17671] Changed implementation of HistoryServer.getApplicationInfoList for lazy evaluation #15303

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fbf1a65
changed implementation of HistoryServer.getApplicationInfoList for la…
wgtmac Sep 29, 2016
dc8146e
directly operator on iterator
wgtmac Sep 30, 2016
955154b
moved toInt logic in map
wgtmac Sep 30, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,25 +40,23 @@ private[v1] class ApplicationListResource(uiRoot: UIRoot) {
status
}
}

val numApps = Option(limit).map(_.toInt).getOrElse(Integer.MAX_VALUE)
val includeCompleted = adjStatus.contains(ApplicationStatus.COMPLETED)
val includeRunning = adjStatus.contains(ApplicationStatus.RUNNING)
val appList = allApps.filter { app =>
allApps.filter { app =>
val anyRunning = app.attempts.exists(!_.completed)
// if any attempt is still running, we consider the app to also still be running
val statusOk = (!anyRunning && includeCompleted) ||
(anyRunning && includeRunning)

// keep the app if *any* attempts fall in the right time window
val dateOk = app.attempts.exists { attempt =>
attempt.startTime.getTime >= minDate.timestamp &&
attempt.startTime.getTime <= maxDate.timestamp

@srowen srowen Sep 30, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be off topic but if your original concern was performance, then I'd note that there's no real need to evaluate dateOk by looking at all attempts again, if statusOk is false. Not sure it matters.

@wgtmac wgtmac Sep 30, 2016

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. Compared to the transformation from ApplicationHistoryInfo to ApplicationInfo, this one is negligible.

}
statusOk && dateOk
}
if (limit != null) {
appList.take(limit)
} else {
appList
}
}.take(numApps)
}
}

Expand Down