[SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* by maropu · Pull Request #28151 · apache/spark

maropu · 2020-04-08T08:00:12Z

What changes were proposed in this pull request?

This PR intends to clean up the SQL documents in doc/sql-ref*.
Main changes are as follows;

Fixes wrong syntaxes and capitalize sub-titles
Adds some DDL queries in Examples so that users can run examples there
Makes query output in Examples follows the Dataset.showString (right-aligned) format
Adds/Removes spaces, Indents, or blank lines to follow the format below;

---
license...
---

### Description

Writes what's the syntax is.

### Syntax

{% highlight sql %}
SELECT...
    WHERE... // 4 indents after the second line
    ...
{% endhighlight %}

### Parameters

<dl>
  <!-- 2 indents in this section -->
  <dt><code><em>Param Name</em></code></dt>
  <dd>
    Param Description
  </dd>
  ...
</dl>

### Examples

{% highlight sql %}
-- It is better that users are able to execute example queries here.
-- So, we prepare test data in the first section if possible.
CREATE TABLE t (key STRING, value DOUBLE);
INSERT INTO t VALUES
    ('a', 1.0), ('a', 2.0), ('b', 3.0), ('c', 4.0);

-- query output has 2 indents and it follows the `Dataset.showString`
-- format (right-aligned).
SELECT * FROM t;
  +---+-----+
  |key|value|
  +---+-----+
  |  a|  1.0|
  |  a|  2.0|
  |  b|  3.0|
  |  c|  4.0|
  +---+-----+

-- Query statements after the second line have 4 indents.
SELECT key, SUM(value)
    FROM t
    GROUP BY key;
  +---+----------+                                                                
  |key|sum(value)|
  +---+----------+
  |  c|       4.0|
  |  b|       3.0|
  |  a|       3.0|
  +---+----------+
...
{% endhighlight %}

### Related Statements

 * [XXX](xxx.html)
 * ...

Why are the changes needed?

The most changes of this PR are pretty minor, but I think the consistent formats/rules to write documents are important for long-term maintenance in our community

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

Manually checked.

maropu · 2020-04-08T08:07:18Z

@gatorsmile @huaxingao @kevinyu98 @srowen I've roughly checked the current whole SQL docs and fixed minor&trivial issues for consistency across the docs. How about this update?

SparkQA · 2020-04-08T08:13:00Z

Test build #120959 has finished for PR 28151 at commit 0c56949.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen

If it's just cleanup, fine. This is for 3.0?
I skimmed the changes and it looks indeed like just formatting, etc.

huaxingao · 2020-04-08T15:49:25Z

-  |Fred|50 |
-  |Dan |50 |
+  |Fred| 50|
+  | Da | 50|


huaxingao · 2020-04-08T16:00:27Z

+  |            col_name|           data_type|comment|
+  +--------------------+--------------------+-------+
+  |                name|              string|   null|
+  |          student_id|                 int|   null|


null -> NULL?
Shall we change all the null to NULL in tables?

Ur, null looks ok because Dataset.showString does so;

scala> Seq(Some(1), None).toDF("a").show() +----+ | a| +----+ | 1| |null| +----+

These are SQL examples. SQL uses NULL?

I'm not faimilar with that though, the output format looks implementation-specific. For example, PostgreSQL uses an empty cell for null by default. Anyway, I like the simple document rule like Query output in Examples should follow the Dataset.showString (right-aligned) format. Since we can easily replace it later, I leave it as it is in this PR. If necessary, please do follow-up later.

huaxingao · 2020-04-08T16:00:46Z

+  |name|rollno|age|
+  +----+------+---+
+  +----+------+---+
+  No rows selected 


nit: This line is a comment, right? put -- in front of it?

Ah, yes. Probably, we can remove this. Thanks!

huaxingao · 2020-04-08T16:10:03Z

@maropu Thanks a lot for doing this!

huaxingao · 2020-04-08T16:11:16Z

This is for 3.0. @srowen

maropu · 2020-04-08T23:49:05Z

Yea, this is just a clean-up for 3.0.

maropu · 2020-04-08T23:55:34Z

+
+#### Examples
+
+{% highlight sql %}


@huaxingao I moved the examples into the seb-sections of EXCEPT/INTERSECT/UNION for readability. Could you check this?

Looks good. Thanks!

maropu · 2020-04-08T23:57:55Z

@huaxingao @kevinyu98 If no problem, could you apply the same cleanup for your open PRs at your convenience?

SparkQA · 2020-04-09T00:03:14Z

Test build #120988 has finished for PR 28151 at commit a7c5e23.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-04-09T00:06:20Z

I will reformat my open PRs.

kevinyu98 · 2020-04-09T05:13:10Z

+                                   |-- name: string (nullable = true)
+                                   |-- grade: integer (nullable = true)
+                                                                                                            
+  +--------+---------+-----------+--------------------------------------------------------------+


does this output need to be right-aligned? also the following SHOW TABLE output.

Yea right, but the information cell itself assumes to be left-aligned, so I left it as it is. Probably, this is an exception case.

kevinyu98 · 2020-04-09T05:35:05Z


 ALTER TABLE StudentInfo ADD IF NOT EXISTS PARTITION (age=18);

 -- After adding a new partition to the table


Do we need to leave space after --? I saw some have one space, some didn't have.

yea, you're right. I'll remove that. thanks.

SparkQA · 2020-04-09T07:05:02Z

Test build #121008 has finished for PR 28151 at commit 5f6677a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-09T07:08:36Z

retest this please

SparkQA · 2020-04-09T07:26:03Z

Test build #121012 has finished for PR 28151 at commit 5f6677a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kevinyu98 · 2020-04-09T16:51:27Z

--Default Output
+{% highlight sql %}
+-- Default Output



do we need leave space here? also the following -- in this file.

SparkQA · 2020-04-10T04:40:17Z

Test build #121058 has finished for PR 28151 at commit 945b28c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-10T08:14:19Z

@srowen Could you check?

huaxingao · 2020-04-10T17:22:53Z

LGTM

SparkQA · 2020-04-13T00:59:51Z

Test build #121156 has finished for PR 28151 at commit 658fc77.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-13T01:44:12Z

Test build #121158 has finished for PR 28151 at commit 2022c42.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? This PR intends to clean up the SQL documents in `doc/sql-ref*`. Main changes are as follows; - Fixes wrong syntaxes and capitalize sub-titles - Adds some DDL queries in `Examples` so that users can run examples there - Makes query output in `Examples` follows the `Dataset.showString` (right-aligned) format - Adds/Removes spaces, Indents, or blank lines to follow the format below; ``` --- license... --- ### Description Writes what's the syntax is. ### Syntax {% highlight sql %} SELECT... WHERE... // 4 indents after the second line ... {% endhighlight %} ### Parameters <dl> <dt><code><em>Param Name</em></code></dt> <dd> Param Description </dd> ... </dl> ### Examples {% highlight sql %} -- It is better that users are able to execute example queries here. -- So, we prepare test data in the first section if possible. CREATE TABLE t (key STRING, value DOUBLE); INSERT INTO t VALUES ('a', 1.0), ('a', 2.0), ('b', 3.0), ('c', 4.0); -- query output has 2 indents and it follows the `Dataset.showString` -- format (right-aligned). SELECT * FROM t; +---+-----+ |key|value| +---+-----+ | a| 1.0| | a| 2.0| | b| 3.0| | c| 4.0| +---+-----+ -- Query statements after the second line have 4 indents. SELECT key, SUM(value) FROM t GROUP BY key; +---+----------+ |key|sum(value)| +---+----------+ | c| 4.0| | b| 3.0| | a| 3.0| +---+----------+ ... {% endhighlight %} ### Related Statements * [XXX](xxx.html) * ... ``` ### Why are the changes needed? The most changes of this PR are pretty minor, but I think the consistent formats/rules to write documents are important for long-term maintenance in our community ### Does this PR introduce any user-facing change? Yes. ### How was this patch tested? Manually checked. Closes #28151 from maropu/MakeRightAligned. Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com> (cherry picked from commit 179289f) Signed-off-by: Sean Owen <srowen@gmail.com>

srowen · 2020-04-13T04:40:52Z

Merged to master/3.0

### What changes were proposed in this pull request? This PR intends to clean up the SQL documents in `doc/sql-ref*`. Main changes are as follows; - Fixes wrong syntaxes and capitalize sub-titles - Adds some DDL queries in `Examples` so that users can run examples there - Makes query output in `Examples` follows the `Dataset.showString` (right-aligned) format - Adds/Removes spaces, Indents, or blank lines to follow the format below; ``` --- license... --- ### Description Writes what's the syntax is. ### Syntax {% highlight sql %} SELECT... WHERE... // 4 indents after the second line ... {% endhighlight %} ### Parameters <dl> <dt><code><em>Param Name</em></code></dt> <dd> Param Description </dd> ... </dl> ### Examples {% highlight sql %} -- It is better that users are able to execute example queries here. -- So, we prepare test data in the first section if possible. CREATE TABLE t (key STRING, value DOUBLE); INSERT INTO t VALUES ('a', 1.0), ('a', 2.0), ('b', 3.0), ('c', 4.0); -- query output has 2 indents and it follows the `Dataset.showString` -- format (right-aligned). SELECT * FROM t; +---+-----+ |key|value| +---+-----+ | a| 1.0| | a| 2.0| | b| 3.0| | c| 4.0| +---+-----+ -- Query statements after the second line have 4 indents. SELECT key, SUM(value) FROM t GROUP BY key; +---+----------+ |key|sum(value)| +---+----------+ | c| 4.0| | b| 3.0| | a| 3.0| +---+----------+ ... {% endhighlight %} ### Related Statements * [XXX](xxx.html) * ... ``` ### Why are the changes needed? The most changes of this PR are pretty minor, but I think the consistent formats/rules to write documents are important for long-term maintenance in our community ### Does this PR introduce any user-facing change? Yes. ### How was this patch tested? Manually checked. Closes apache#28151 from maropu/MakeRightAligned. Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>

… for MsSqlServerDialect ## What changes were proposed in this pull request? This PR aims to correct mappings in `MsSqlServerDialect`. `ShortType` is mapped to `SMALLINT` and `FloatType` is mapped to `REAL` per [JBDC mapping]( https://docs.microsoft.com/en-us/sql/connect/jdbc/using-basic-data-types?view=sql-server-2017) respectively. ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables and spark data frame being created with unintended types. The issue was observed when validating against SQLServer. Refer [JBDC mapping]( https://docs.microsoft.com/en-us/sql/connect/jdbc/using-basic-data-types?view=sql-server-2017 ) for guidance on mappings between SQLServer, JDBC and Java. Note that java "Short" type should be mapped to JDBC "SMALLINT" and java Float should be mapped to JDBC "REAL". Some example issue that can happen because of wrong mappings - Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT.Thus a larger table that expected. - Read results in a dataframe with type INTEGER as opposed to ShortType - ShortType has a problem in both the the write and read path - FloatTypes only have an issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'. Refer apache#28151 which contained this fix as one part of a larger PR. Following PR apache#28151 discussion it was decided to file seperate PRs for each of the fixes. ## How was this patch tested? UnitTest added in JDBCSuite.scala and these were tested. Integration test updated and passed in MsSqlServerDialect.scala E2E test done with SQLServer Closes apache#25146 from shivsood/float_short_type_fix. Authored-by: shivsood <shivsood@microsoft.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

srowen reviewed Apr 8, 2020

View reviewed changes

huaxingao reviewed Apr 8, 2020

View reviewed changes

maropu force-pushed the MakeRightAligned branch from 0c56949 to 1abe5b8 Compare April 8, 2020 23:46

maropu commented Apr 8, 2020

View reviewed changes

maropu mentioned this pull request Apr 9, 2020

[SPARK-31349][SQL][DOCS] Document built-in aggregate functions in SQL Reference #28120

Closed

kevinyu98 reviewed Apr 9, 2020

View reviewed changes

maropu force-pushed the MakeRightAligned branch from 5f6677a to 945b28c Compare April 10, 2020 04:23

maropu added 2 commits April 13, 2020 09:40

Fix

42ccdcb

Fix

658fc77

maropu force-pushed the MakeRightAligned branch from 945b28c to 658fc77 Compare April 13, 2020 00:41

Fix

2022c42

probot-autolabeler Bot added the DOCS label Apr 13, 2020

srowen closed this in 179289f Apr 13, 2020

maropu mentioned this pull request Apr 20, 2020

[SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs #28104

Closed


		ALTER TABLE StudentInfo ADD IF NOT EXISTS PARTITION (age=18);

		-- After adding a new partition to the table

Uh oh!

Conversation

maropu commented Apr 8, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

maropu commented Apr 8, 2020

Uh oh!

SparkQA commented Apr 8, 2020

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huaxingao commented Apr 8, 2020

Uh oh!

huaxingao commented Apr 8, 2020

Uh oh!

maropu commented Apr 8, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maropu commented Apr 8, 2020

Uh oh!

SparkQA commented Apr 9, 2020

Uh oh!

huaxingao commented Apr 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 9, 2020

Uh oh!

maropu commented Apr 9, 2020

Uh oh!

SparkQA commented Apr 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 10, 2020

Uh oh!

maropu commented Apr 10, 2020

Uh oh!

huaxingao commented Apr 10, 2020

Uh oh!

SparkQA commented Apr 13, 2020

Uh oh!

SparkQA commented Apr 13, 2020

Uh oh!

srowen commented Apr 13, 2020

Uh oh!

Reviewers

Assignees

Labels