Skip to content

[ZEPPELIN-2245] separate precode into JDBCInterpreter #2121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tinkoff-dwh wants to merge 3 commits into from
Closed

[ZEPPELIN-2245] separate precode into JDBCInterpreter #2121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f896ea8
[ZEPPELIN-2245] separate precode into JDBCInterpreter
tinkoff-dwh Mar 10, 2017
a136a0e
[ZEPPELIN-2245] documentation for usage of precode
tinkoff-dwh Mar 12, 2017
970c064
[ZEPPELIN-2245] editing documentation
tinkoff-dwh Mar 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 74 additions & 4 deletions docs/interpreter/jdbc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,11 @@ The JDBC interpreter properties are defined by default like below.
<td>gpadmin</td>
<td>The JDBC user name</td>
</tr>
<tr>
<td>default.precode</td>
<td></td>
<td>Some SQL which executes while opening connection</td>
</tr>
Copy link
Copy Markdown
Member

@felixcheung felixcheung Mar 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add an example of using this?
perhaps in ## Binding JDBC interpter to notebook? Or another place you find

Copy link
Copy Markdown
Contributor Author

@tinkoff-dwh tinkoff-dwh Mar 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added example

</table>

If you want to connect other databases such as `Mysql`, `Redshift` and `Hive`, you need to edit the property values.
Expand Down Expand Up @@ -167,10 +172,6 @@ There are more JDBC interpreter properties you can specify like below.
<td>default.jceks.credentialKey</td>
<td>jceks credential key</td>
</tr>
<tr>
<td>zeppelin.jdbc.precode</td>
<td>Some SQL which executes while opening connection</td>
</tr>
</table>

You can also add more properties by using this [method](http://docs.oracle.com/javase/7/docs/api/java/sql/DriverManager.html#getConnection%28java.lang.String,%20java.util.Properties%29).
Expand Down Expand Up @@ -221,6 +222,75 @@ SELECT name, country, performer
FROM demo.performers
WHERE name='{{"{{performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia"}}}}'
```
### Usage *precode*
You can set *precode* for each data source. Code runs once while opening the connection.

##### Properties
An example settings of interpreter for the two data sources, each of which has its *precode* parameter.

<table class="table-configuration">
<tr>
<th>Property Name</th>
<th>Value</th>
</tr>
<tr>
<td>default.driver</td>
Copy link
Copy Markdown
Member

@felixcheung felixcheung Mar 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, should it have/use default. here?

Copy link
Copy Markdown
Member

@felixcheung felixcheung Mar 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just omit them in this example?

Copy link
Copy Markdown
Contributor Author

@tinkoff-dwh tinkoff-dwh Mar 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, here is set two data sources, each one with a precode

<td>org.postgresql.Driver</td>
</tr>
<tr>
<td>default.password</td>
<td>1</td>
</tr>
<tr>
<td>default.url</td>
<td>jdbc:postgresql://localhost:5432/</td>
</tr>
<tr>
<td>default.user</td>
<td>postgres</td>
</tr>
<tr>
<td>default.precode</td>
<td>set search_path='test_path'</td>
</tr>
<tr>
<td>mysql.driver</td>
<td>com.mysql.jdbc.Driver</td>
</tr>
<tr>
<td>mysql.password</td>
<td>1</td>
</tr>
<tr>
<td>mysql.url</td>
<td>jdbc:mysql://localhost:3306/</td>
</tr>
<tr>
<td>mysql.user</td>
<td>root</td>
</tr>
<tr>
<td>mysql.precode</td>
<td>set @v=12</td>
</tr>
</table>

##### Usage
Test of execution *precode* for each data source.

```sql
%jdbc
show search_path
```
Returns value of `search_path` which is set in the *default.precode*.


```sql
%jdbc(mysql)
select @v
```
Returns value of `v` which is set in the *mysql.precode*.


## Examples
Here are some examples you can refer to. Including the below connectors, you can connect every databases as long as it can be configured with it's JDBC driver.
Expand Down
10 changes: 6 additions & 4 deletions jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,10 @@ public class JDBCInterpreter extends Interpreter {
static final String URL_KEY = "url";
static final String USER_KEY = "user";
static final String PASSWORD_KEY = "password";
static final String PRECODE_KEY = "precode";
static final String JDBC_JCEKS_FILE = "jceks.file";
static final String JDBC_JCEKS_CREDENTIAL_KEY = "jceks.credentialKey";
static final String ZEPPELIN_JDBC_PRECODE_KEY = "zeppelin.jdbc.precode";
static final String PRECODE_KEY_TEMPLATE = "%s.precode";
static final String DOT = ".";

private static final char WHITESPACE = ' ';
Expand All @@ -118,6 +119,7 @@ public class JDBCInterpreter extends Interpreter {
static final String DEFAULT_URL = DEFAULT_KEY + DOT + URL_KEY;
static final String DEFAULT_USER = DEFAULT_KEY + DOT + USER_KEY;
static final String DEFAULT_PASSWORD = DEFAULT_KEY + DOT + PASSWORD_KEY;
static final String DEFAULT_PRECODE = DEFAULT_KEY + DOT + PRECODE_KEY;

static final String EMPTY_COLUMN_VALUE = "";

Expand Down Expand Up @@ -342,7 +344,7 @@ private Connection getConnectionFromPool(String url, String user, String propert
if (!getJDBCConfiguration(user).isConnectionInDBDriverPool(propertyKey)) {
createConnectionPool(url, user, propertyKey, properties);
try (Connection connection = DriverManager.getConnection(jdbcDriver)) {
executePrecode(connection);
executePrecode(connection, propertyKey);
}
}
return DriverManager.getConnection(jdbcDriver);
Expand Down Expand Up @@ -548,8 +550,8 @@ protected ArrayList<String> splitSqlQueries(String sql) {
return queries;
}

private void executePrecode(Connection connection) throws SQLException {
String precode = getProperty(ZEPPELIN_JDBC_PRECODE_KEY);
private void executePrecode(Connection connection, String propertyKey) throws SQLException {
String precode = getProperty(String.format(PRECODE_KEY_TEMPLATE, propertyKey));
if (StringUtils.isNotBlank(precode)) {
precode = StringUtils.trim(precode);
logger.info("Run SQL precode '{}'", precode);
Expand Down
12 changes: 6 additions & 6 deletions jdbc/src/main/resources/interpreter-setting.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@
"defaultValue": "org.postgresql.Driver",
"description": "JDBC Driver Name"
},
"default.precode": {
"envName": null,
"propertyName": "zeppelin.jdbc.precode",
"defaultValue": "",
"description": "SQL which executes while opening connection"
},
"common.max_count": {
"envName": null,
"propertyName": "common.max_count",
Expand Down Expand Up @@ -63,12 +69,6 @@
"propertyName": "zeppelin.jdbc.principal",
"defaultValue": "",
"description": "Kerberos principal"
},
"zeppelin.jdbc.precode": {
"envName": null,
"propertyName": "zeppelin.jdbc.precode",
"defaultValue": "",
"description": "SQL which executes while opening connection"
}
},
"editor": {
Expand Down
30 changes: 25 additions & 5 deletions jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
import static org.apache.zeppelin.jdbc.JDBCInterpreter.DEFAULT_PASSWORD;
import static org.apache.zeppelin.jdbc.JDBCInterpreter.DEFAULT_USER;
import static org.apache.zeppelin.jdbc.JDBCInterpreter.DEFAULT_URL;
import static org.apache.zeppelin.jdbc.JDBCInterpreter.DEFAULT_PRECODE;
import static org.apache.zeppelin.jdbc.JDBCInterpreter.PRECODE_KEY_TEMPLATE;
import static org.apache.zeppelin.jdbc.JDBCInterpreter.COMMON_MAX_LINE;
import static org.junit.Assert.*;

Expand All @@ -44,8 +46,6 @@

import com.mockrunner.jdbc.BasicJDBCTestCaseAdapter;

import static org.apache.zeppelin.jdbc.JDBCInterpreter.ZEPPELIN_JDBC_PRECODE_KEY;

/**
* JDBC interpreter unit tests
*/
Expand Down Expand Up @@ -397,7 +397,7 @@ public void testPrecode() throws SQLException, IOException {
properties.setProperty("default.url", getJdbcConnection());
properties.setProperty("default.user", "");
properties.setProperty("default.password", "");
properties.setProperty(ZEPPELIN_JDBC_PRECODE_KEY, "SET @testVariable=1");
properties.setProperty(DEFAULT_PRECODE, "SET @testVariable=1");
JDBCInterpreter jdbcInterpreter = new JDBCInterpreter(properties);
jdbcInterpreter.open();

Expand All @@ -417,7 +417,7 @@ public void testIncorrectPrecode() throws SQLException, IOException {
properties.setProperty("default.url", getJdbcConnection());
properties.setProperty("default.user", "");
properties.setProperty("default.password", "");
properties.setProperty(ZEPPELIN_JDBC_PRECODE_KEY, "incorrect command");
properties.setProperty(DEFAULT_PRECODE, "incorrect command");
JDBCInterpreter jdbcInterpreter = new JDBCInterpreter(properties);
jdbcInterpreter.open();

Expand All @@ -428,4 +428,24 @@ public void testIncorrectPrecode() throws SQLException, IOException {
assertEquals(InterpreterResult.Code.ERROR, interpreterResult.code());
assertEquals(InterpreterResult.Type.TEXT, interpreterResult.message().get(0).getType());
}
}

@Test
public void testPrecodeWithAnotherPrefix() throws SQLException, IOException {
Properties properties = new Properties();
properties.setProperty("anotherPrefix.driver", "org.h2.Driver");
properties.setProperty("anotherPrefix.url", getJdbcConnection());
properties.setProperty("anotherPrefix.user", "");
properties.setProperty("anotherPrefix.password", "");
properties.setProperty(String.format(PRECODE_KEY_TEMPLATE, "anotherPrefix"), "SET @testVariable=2");
JDBCInterpreter jdbcInterpreter = new JDBCInterpreter(properties);
jdbcInterpreter.open();

String sqlQuery = "(anotherPrefix) select @testVariable";

InterpreterResult interpreterResult = jdbcInterpreter.interpret(sqlQuery, interpreterContext);

assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code());
assertEquals(InterpreterResult.Type.TABLE, interpreterResult.message().get(0).getType());
assertEquals("@TESTVARIABLE\n2\n", interpreterResult.message().get(0).getData());
}
}