Skip to content

ignore case of with header row in sql when creating external table#1237

Merged
alamb merged 3 commits into
apache:masterfrom
dylankyc:case-insensitive-with-header-row-token
Nov 7, 2021
Merged

ignore case of with header row in sql when creating external table#1237
alamb merged 3 commits into
apache:masterfrom
dylankyc:case-insensitive-with-header-row-token

Conversation

@dylankyc

@dylankyc dylankyc commented Nov 4, 2021

Copy link
Copy Markdown
Contributor

Resolves #1242

Currently, creating an external table using datafusion-cli with this sql:

CREATE EXTERNAL TABLE t(c1 int) STORED AS CSV with header row LOCATION 'foo.csv'

will fail when with header row tokens are in lower case. This pr is a fix.

@github-actions github-actions Bot added datafusion sql SQL Planner labels Nov 4, 2021
Comment thread datafusion/src/sql/parser.rs Outdated

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thank you @lichuan6

@alamb

alamb commented Nov 4, 2021

Copy link
Copy Markdown
Contributor

@lichuan6 there appears to be some failures on the CI checks

@houqp houqp added the bug Something isn't working label Nov 5, 2021
@dylankyc

dylankyc commented Nov 5, 2021

Copy link
Copy Markdown
Contributor Author

@lichuan6 there appears to be some failures on the CI checks

OK, thanks @alamb for pointing it out. It seems using self.consume_token(&Token::make_keyword("WITH")) will only recognize upper case keyword. What's the best way to do case-insensitive token parsing?

Currently, after changing signature from

fn consume_token(&mut self, expected: &str) -> bool

to

fn consume_token(&mut self, expected: &Token) -> bool

to do case-insensitive parsing, I created a new Token::Word enum with upper case of next token. Don't know if this is the best practice.

@dylankyc dylankyc requested a review from alamb November 5, 2021 05:36
@dylankyc

dylankyc commented Nov 5, 2021

Copy link
Copy Markdown
Contributor Author

@Dandandan Could you please approve and make the tests run?

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thank you @lichuan6

Any thoughts @Dandandan ?

@dylankyc

dylankyc commented Nov 6, 2021

Copy link
Copy Markdown
Contributor Author

Hi, @Dandandan Could you please review this PR?

@alamb

alamb commented Nov 6, 2021

Copy link
Copy Markdown
Contributor

If @Dandandan doesn't have anything to add I'll plan to merge this PR tomorrow

@alamb alamb merged commit a8029e5 into apache:master Nov 7, 2021
@alamb

alamb commented Nov 7, 2021

Copy link
Copy Markdown
Contributor

Thanks again @lichuan6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CREATE EXTERNAL TABLE ... STORED AS CSV with header row fails when with header row is lowercase

4 participants