Skip to content

[SPARK-10540] [SQL] Ignore HadoopFsRelationTest's "test all data types" if it is too flaky#8705

Closed
yhuai wants to merge 1 commit into
apache:masterfrom
yhuai:SPARK-10540-ignore
Closed

[SPARK-10540] [SQL] Ignore HadoopFsRelationTest's "test all data types" if it is too flaky#8705
yhuai wants to merge 1 commit into
apache:masterfrom
yhuai:SPARK-10540-ignore

Conversation

@yhuai

@yhuai yhuai commented Sep 10, 2015

Copy link
Copy Markdown
Contributor

If hadoopFsRelationSuites's "test all data types" is too flaky we can disable it for now.

https://issues.apache.org/jira/browse/SPARK-10540

@SparkQA

SparkQA commented Sep 10, 2015

Copy link
Copy Markdown

Test build #42284 has finished for PR 8705 at commit 7c4791f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Sep 11, 2015
…s" if it is too flaky

If hadoopFsRelationSuites's "test all data types" is too flaky we can disable it for now.

https://issues.apache.org/jira/browse/SPARK-10540

Author: Yin Huai <yhuai@databricks.com>

Closes #8705 from yhuai/SPARK-10540-ignore.

(cherry picked from commit 6ce0886)
Signed-off-by: Yin Huai <yhuai@databricks.com>
@asfgit asfgit closed this in 6ce0886 Sep 11, 2015
@liancheng

Copy link
Copy Markdown
Contributor

Found two possible sources of flakiness:

  1. For double and float, NaN == NaN is always false

  2. checkAnswer doesn't handle Map[K, V] properly. For example:

    scala> Map(1 -> 2, 2 -> 1).toString
    res0: String = Map(1 -> 2, 2 -> 1)
    
    scala> Map(2 -> 1, 1 -> 2).toString
    res1: String = Map(2 -> 1, 1 -> 2)
    

    We can't rely on toString to compare Map instances.

Need to update checkAnswer to special case NaN and Map[K, V].

@liancheng

Copy link
Copy Markdown
Contributor

Filed https://issues.apache.org/jira/browse/SPARK-10591 to track this issue.

@yhuai

yhuai commented Sep 14, 2015

Copy link
Copy Markdown
Contributor Author

For the issue of NaN, looks like there is something wrong because we want to return true when compare two NaN.

ashangit pushed a commit to ashangit/spark that referenced this pull request Oct 19, 2016
…s" if it is too flaky

If hadoopFsRelationSuites's "test all data types" is too flaky we can disable it for now.

https://issues.apache.org/jira/browse/SPARK-10540

Author: Yin Huai <yhuai@databricks.com>

Closes apache#8705 from yhuai/SPARK-10540-ignore.

(cherry picked from commit 6ce0886)
Signed-off-by: Yin Huai <yhuai@databricks.com>
(cherry picked from commit 295281f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants