[SPARK-2393][SQL] Cost estimation optimization framework for Catalyst logical plans & sample usage.#1238

Closed

concretevitamin wants to merge 27 commits into

apache:masterfrom

concretevitamin:estimates

Commits on Jul 29, 2014

Prototype impl of estimations for Catalyst logical plans.
concretevitamin
committed
Typo.
concretevitamin
committed
Refactors.
concretevitamin
committed
Cleanups.
concretevitamin
committed
Add parquetAfter() properly in test.
concretevitamin
committed
s/Estimates/Statistics, s/cardinality/numTuples.
concretevitamin
committed
Move SQLConf to Catalyst & add default val for sizeInBytes.
concretevitamin
committed
Move SQLConf back. Assign default sizeInBytes to SparkLogicalPlan.
concretevitamin
committed
Apply estimation to planning ShuffleHashJoin & BroadcastNestedLoopJoin.
concretevitamin
committed
Fix optimization conditions & update scala docs to explain.
concretevitamin
committed
Add comment.
concretevitamin
committed
Make Statistics a case class.
concretevitamin
committed
Use multiplication instead of sum for default estimates.
concretevitamin
committed
Revert optimization for BroadcastNestedLoopJoin (this fixes tests).
concretevitamin
committed
Inject SQLContext into SparkLogicalPlan, removing SQLConf mixin from it.
concretevitamin
committed
{Cleanup, use synchronized in, enrich} StatisticsSuite.
concretevitamin
committed
Remove singleton SQLConf and move back `settings` to the trait.
concretevitamin
committed
Update docs to be more explicit.
concretevitamin
committed
Remove numTuples in Statistics for now.
concretevitamin
committed
Update scaladoc for a field and move it to @param section.
concretevitamin
committed
Get size info from metastore for MetastoreRelation.
concretevitamin
committed
Add a note on performance of statistics.
concretevitamin
committed
Avoid calling statistics on plans if auto join conversion is disabled.
concretevitamin
committed
Remove childrenStats.
concretevitamin
committed
Fix statistics for SparkLogicalPlan.
concretevitamin
committed
Use BigInt for stat; for logical leaves, by default throw an exception.
concretevitamin
committed
Address review comments; turn config name from string to field in SQLConf.
concretevitamin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-2393][SQL] Cost estimation optimization framework for Catalyst logical plans & sample usage.#1238

[SPARK-2393][SQL] Cost estimation optimization framework for Catalyst logical plans & sample usage.#1238
concretevitamin wants to merge 27 commits into
apache:masterfrom
concretevitamin:estimates

Commits on Jul 29, 2014

Prototype impl of estimations for Catalyst logical plans.

Typo.

Refactors.

Cleanups.

Add parquetAfter() properly in test.

s/Estimates/Statistics, s/cardinality/numTuples.

Move SQLConf to Catalyst & add default val for sizeInBytes.

Move SQLConf back. Assign default sizeInBytes to SparkLogicalPlan.

Apply estimation to planning ShuffleHashJoin & BroadcastNestedLoopJoin.

Fix optimization conditions & update scala docs to explain.

Add comment.

Make Statistics a case class.

Use multiplication instead of sum for default estimates.

Revert optimization for BroadcastNestedLoopJoin (this fixes tests).

Inject SQLContext into SparkLogicalPlan, removing SQLConf mixin from it.

{Cleanup, use synchronized in, enrich} StatisticsSuite.

Remove singleton SQLConf and move back `settings` to the trait.

Update docs to be more explicit.

Remove numTuples in Statistics for now.

Update scaladoc for a field and move it to @param section.

Get size info from metastore for MetastoreRelation.

Add a note on performance of statistics.

Avoid calling statistics on plans if auto join conversion is disabled.

Remove childrenStats.

Fix statistics for SparkLogicalPlan.

Use BigInt for stat; for logical leaves, by default throw an exception.

Address review comments; turn config name from string to field in SQLConf.