[SPARK-17346][SQL][Tests]Fix the flaky topic deletion in KafkaSourceStressSuite

[zsxwing]

What changes were proposed in this pull request?

A follow up Pr for SPARK-17346 to fix flaky org.apache.spark.sql.kafka010.KafkaSourceStressSuite.

Test log: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.4/1855/testReport/junit/org.apache.spark.sql.kafka010/KafkaSourceStressSuite/_It_is_not_a_test_/

Looks like deleting the Kafka internal topic __consumer_offsets is flaky. This PR just simply ignores internal topics.

How was this patch tested?

Existing tests.

[zsxwing]

/cc @tdas

[srowen]

Yes we shouldn't be deleting that topic

[SparkQA]

Test build #66466 has finished for PR 15384 at commit 5158d5d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

Are you sure we are ignoring all internal topics with that filter?
Because, it maybe better to not make assumption, and rather prefix all topics we create with something, and deleted only those.

[zsxwing]

Are you sure we are ignoring all internal topics with that filter?
Because, it maybe better to not make assumption, and rather prefix all topics we create with something, and deleted only those.

Good point. Just use stress to filter instead.

[SparkQA]

Test build #66471 has finished for PR 15384 at commit c7950f0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

LGTM. Merging it to master and 2.0

[tdas]

Correction. Merging to master only. Can you fix the 2.0 PR with this issue since that is still open?

[zsxwing]

Since deleting a normal topic may be timeout as well: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/1763/testReport/junit/org.apache.spark.sql.kafka010/KafkaSourceStressSuite/_It_is_not_a_test_/

I just removed the topic deletion codes. It's not necessary since we are going to shut down the kafka cluster.

[SparkQA]

Test build #66520 has finished for PR 15384 at commit 0addb26.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class KafkaSourceStressSuite extends KafkaSourceTest

[zsxwing]

@tdas since this one has not yet been merged, I just updated the PR to remove the topic cleanup.

[tdas]

Merging it to master, and branch 2.0