diff --git a/README.md b/README.md index dffc9b21..49a324b6 100644 --- a/README.md +++ b/README.md @@ -179,10 +179,10 @@ Download the self-contained fat JAR for your Elasticsearch version: | Elasticsearch Version | Artifact | |-----------------------|----------------------------------------| -| ES 6.x | `softclient4es6-jdbc-driver-0.1.4.jar` | -| ES 7.x | `softclient4es7-jdbc-driver-0.1.4.jar` | -| ES 8.x | `softclient4es8-jdbc-driver-0.1.4.jar` | -| ES 9.x | `softclient4es9-jdbc-driver-0.1.4.jar` | +| ES 6.x | `softclient4es6-jdbc-driver-0.2.0.jar` | +| ES 7.x | `softclient4es7-jdbc-driver-0.2.0.jar` | +| ES 8.x | `softclient4es8-jdbc-driver-0.2.0.jar` | +| ES 9.x | `softclient4es9-jdbc-driver-0.2.0.jar` | ```text JDBC URL: jdbc:elastic://localhost:9200 @@ -197,20 +197,20 @@ Driver class: app.softnetwork.elastic.jdbc.ElasticDriver app.softnetwork.elastic softclient4es8-jdbc-driver - 0.1.4 + 0.2.0 ``` **Gradle:** ```groovy -implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.1.4' +implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.0' ``` **sbt:** ```scala -libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.1.4" +libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.0" ``` The JDBC driver JARs are Scala-version-independent (no `_2.12` or `_2.13` suffix) and include all required dependencies. @@ -298,9 +298,9 @@ resolvers += "Softnetwork" at "https://softnetwork.jfrog.io/artifactory/releases // Choose your Elasticsearch version libraryDependencies += "app.softnetwork.elastic" %% "softclient4es8-java-client" % "0.20.0" // Add the community extensions for materialized views (optional) -libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-community-extensions" % "0.1.4" +libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-community-extensions" % "0.2.0" // Add the JDBC driver if you want to use it from Scala (optional) -libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-jdbc-driver" % "0.1.4" +libraryDependencies += "app.softnetwork.elastic" %% "softclient4es-jdbc-driver" % "0.2.0" ``` ```scala diff --git a/documentation/client/README.md b/documentation/client/README.md index 13f09c16..e737d80a 100644 --- a/documentation/client/README.md +++ b/documentation/client/README.md @@ -21,8 +21,9 @@ Welcome to the Client Engine Documentation. Navigate through the sections below: - [Template Management](templates.md) - [SQL Gateway Usage](gateway.md) - [REPL Client](repl.md) -- [Arrow Flight SQL](arrow_flight_sql.md) +- [JDBC Driver](jdbc.md) - [ADBC Driver](adbc_driver.md) +- [Arrow Flight SQL](arrow_flight_sql.md) - [BI Tool Integration](bi_tools.md) — DBeaver, Superset, Grafana, Tableau, Power BI, Metabase (full guides on the website) - [Telemetry & Privacy](telemetry.md) - [Download Analytics](download_analytics.md) diff --git a/documentation/client/adbc_driver.md b/documentation/client/adbc_driver.md index 13f81376..2d5ced09 100644 --- a/documentation/client/adbc_driver.md +++ b/documentation/client/adbc_driver.md @@ -21,12 +21,12 @@ In-process columnar access to Elasticsearch — direct Arrow format, no separate Download the self-contained fat JAR for your Elasticsearch version: -| Elasticsearch | Artifact | -|----------------|----------------------------------------| -| ES 6.x | `softclient4es6-adbc-driver-0.1.5.jar` | -| ES 7.x | `softclient4es7-adbc-driver-0.1.5.jar` | -| ES 8.x | `softclient4es8-adbc-driver-0.1.5.jar` | -| ES 9.x | `softclient4es9-adbc-driver-0.1.5.jar` | +| Elasticsearch | Artifact | +|----------------|-----------------------------------------------------| +| ES 6.x | `softclient4es6-adbc-driver-0.2.0.jar` | +| ES 7.x | `softclient4es7-adbc-driver-0.2.0.jar` | +| ES 8.x | `softclient4es8-adbc-driver-0.2.0.jar` | +| ES 9.x | `softclient4es9-adbc-driver-0.2.0.jar` | ### Maven / Gradle / sbt @@ -36,20 +36,20 @@ Download the self-contained fat JAR for your Elasticsearch version: app.softnetwork.elastic softclient4es8-adbc-driver - 0.1.5 + 0.2.0 ``` **Gradle:** ```groovy -implementation 'app.softnetwork.elastic:softclient4es8-adbc-driver:0.1.5' +implementation 'app.softnetwork.elastic:softclient4es8-adbc-driver:0.2.0' ``` **sbt:** ```scala -libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-adbc-driver" % "0.1.5" +libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-adbc-driver" % "0.2.0" ``` --- @@ -98,6 +98,21 @@ adbc:elastic://:?bearer= --- +## Your first JOIN + +Elasticsearch can't JOIN across indices — SoftClient4ES does. **Free in Community: up to 2 cross-index JOINs per query** (a 3-table JOIN). ADBC routes the JOIN through the same embedded DuckDB engine as the JDBC driver, so the SQL is identical — only the connection setup differs: + +```sql +SELECT e.name, e.salary, d.dept_name +FROM jdbc_join_emp e +JOIN jdbc_join_dept d ON e.dept_id = d.dept_id; +-- 5 rows (the orphan employee with dept_id = 99 is dropped by the INNER JOIN) +``` + +Set this as the statement's SQL (`stmt.setSqlQuery(...)`) and read the Arrow stream as shown in [Usage](#usage). For the full JOIN matrix, see the Cross-Index JOIN walkthrough (`../sql/joins.md`). + +--- + ## Configuration ```hocon @@ -131,9 +146,43 @@ elastic.credentials { --- +## Version compatibility + +| Driver | Scala | ES versions | Clients | Process model | +|--------|-------|-------------|---------|---------------| +| ADBC | cross-built Scala 2.12 + 2.13 | ES 6.x / 7.x / 8.x / 9.x | Java/JVM only (polyglot → Flight SQL) | In-process | + +The fat JARs are **Scala-version-independent for consumers** — they bundle their own Scala runtime, so you almost never need to think about the Scala axis. For non-JVM languages (Python, Go, DuckDB, C++), use the [Arrow Flight SQL server](arrow_flight_sql.md) with the Arrow project's standard `adbc_driver_flightsql` client. + +--- + +## Licensing & self-selection + +All client drivers (JDBC, ADBC, and the REPL) plus the Arrow Flight SQL sidecar are **free in Community**, including up to **2 cross-index JOINs per query** (a 3-table JOIN). A 4-table JOIN (a 3rd cross-index JOIN in one query) is rejected by the planner with a message ending `… Upgrade to Pro … See: https://portal.softclient4es.com/pricing`. + +Multi-cluster **federation** — joining across *separate* ES clusters — is **Pro+** (`maxClusters` 1 / 5 / ∞). This quickstart covers the **single-cluster** shape, which is free. + +--- + +## What does NOT work yet + +Subqueries (`IN (SELECT …)`, `EXISTS`, scalar, derived tables) and CTEs (`WITH`) are not supported in the current release — they arrive in a later release. Write the JOIN explicitly instead. See the Known Limitations & Roadmap (`../sql/known_limitations.md`) for the full list. + +--- + +## Going further + +- [JDBC Driver](jdbc.md) — the row-based in-process alternative for any JDBC tool. +- [Arrow Flight SQL](arrow_flight_sql.md) — the gRPC server for polyglot and networked clients. +- Cross-Index JOIN walkthrough (`../sql/joins.md`) — the full JOIN matrix (rows 1/2/3) with worked examples. +- Multi-cluster federation operator guide (`federation_operator_guide.md`) — the Pro+ path: JOIN across separate ES clusters. +- Known Limitations & Roadmap (`../sql/known_limitations.md`) — what works in the current release vs what's coming. + +--- + ## Telemetry -The ADBC driver sends one anonymous usage ping per day (no IP, no SQL, no PII). Opt out with `softclient4es.telemetry.enabled = false` in your HOCON config. See [Telemetry & Privacy](telemetry.md) for details. +The ADBC driver sends one anonymous usage ping per day (no IP, no SQL, no PII). Opt out with `softclient4es.telemetry.enabled = false` in your HOCON config, the `SOFTCLIENT4ES_TELEMETRY_ENABLED=false` environment variable, or `-Dsoftclient4es.telemetry.enabled=false`. ADBC has no connection-string opt-out — the `adbc:elastic://` URI carries no `telemetry` option. See [Telemetry & Privacy](telemetry.md) for details. --- diff --git a/documentation/client/arrow_flight_sql.md b/documentation/client/arrow_flight_sql.md index 489176af..2ea2545f 100644 --- a/documentation/client/arrow_flight_sql.md +++ b/documentation/client/arrow_flight_sql.md @@ -4,6 +4,15 @@ Zero-copy columnar access to Elasticsearch over gRPC — for DuckDB, Python, Apa --- +## Sidecar vs federation + +There are two Flight SQL servers, and these quickstarts cover only the first: + +- **Single-cluster sidecar** (port `32010`) — one server in front of **one** ES cluster. Cross-index JOINs run in the sidecar's embedded DuckDB engine. **This is what this page covers, and it is free in Community.** +- **Multi-cluster federation server** — a coordinator that fans out across **multiple** ES clusters to JOIN data living in different clusters. This is the **Pro+** path; see the federation operator guide (`federation_operator_guide.md`). + +--- + ## Features - **gRPC Protocol** — High-performance columnar data access over HTTP/2 @@ -40,15 +49,15 @@ Available images per ES version: ### Fat JAR ```bash -java -jar softclient4es8-arrow-flight-sql-.jar +java -jar softclient4es8-arrow-flight-sql-0.2.0.jar ``` | Elasticsearch | Artifact | |---------------|----------| -| ES 6.x | `softclient4es6-arrow-flight-sql-.jar` | -| ES 7.x | `softclient4es7-arrow-flight-sql-.jar` | -| ES 8.x | `softclient4es8-arrow-flight-sql-.jar` | -| ES 9.x | `softclient4es9-arrow-flight-sql-.jar` | +| ES 6.x | `softclient4es6-arrow-flight-sql-0.2.0.jar` | +| ES 7.x | `softclient4es7-arrow-flight-sql-0.2.0.jar` | +| ES 8.x | `softclient4es8-arrow-flight-sql-0.2.0.jar` | +| ES 9.x | `softclient4es9-arrow-flight-sql-0.2.0.jar` | --- @@ -71,6 +80,24 @@ duckdb.sql(""" """) ``` +## Your first JOIN + +Elasticsearch can't JOIN across indices — the Flight SQL sidecar does. **Free in Community: up to 2 cross-index JOINs per query** (a 3-table JOIN). Both indices live in the **one** ES cluster the sidecar fronts; the JOIN runs in the sidecar's embedded DuckDB engine: + +```python +cursor.execute(""" + SELECT e.name, e.salary, d.dept_name + FROM jdbc_join_emp e + JOIN jdbc_join_dept d ON e.dept_id = d.dept_id +""") +table = cursor.fetch_arrow_table() +# 5 rows (the orphan employee with dept_id = 99 is dropped by the INNER JOIN) +``` + +To JOIN across *separate* ES clusters, deploy the federation server instead — see [Going further](#going-further). + +--- + ## Apache Superset Connect Superset via the Arrow Flight SQL dialect. Then query Elasticsearch directly from any dashboard: @@ -138,9 +165,48 @@ elastic.credentials { --- +## In-process alternative: the ADBC driver + +If you need columnar Arrow access **without** running a separate server, the in-process **ADBC driver** delivers the same SQL (including cross-index JOIN) inside your JVM — see the [ADBC Driver](adbc_driver.md) page. ADBC is Java/JVM in-process; polyglot clients (Python, Go, DuckDB, C++) connect to *this* Flight SQL server. + +--- + +## Version compatibility + +| Driver | Scala | ES versions | Clients | Process model | +|--------|-------|-------------|---------|---------------| +| Arrow Flight SQL | cross-built Scala 2.12 + 2.13 (server) | ES 6.x / 7.x / 8.x / 9.x | Any ADBC/Flight SQL client — Python, Go, DuckDB, C++, Grafana, Superset | Separate server (gRPC) | + +The fat JARs are **Scala-version-independent for consumers** — you almost never need to think about the Scala axis. + +--- + +## Licensing & self-selection + +All client drivers (JDBC, ADBC, and the REPL) plus the Arrow Flight SQL sidecar are **free in Community**, including up to **2 cross-index JOINs per query** (a 3-table JOIN). A 4-table JOIN (a 3rd cross-index JOIN in one query) is rejected by the planner with a message ending `… Upgrade to Pro … See: https://portal.softclient4es.com/pricing`. + +The single-cluster sidecar on this page is the free shape. Multi-cluster **federation** — joining across *separate* ES clusters — is **Pro+** (`maxClusters` 1 / 5 / ∞). + +--- + +## What does NOT work yet + +Subqueries (`IN (SELECT …)`, `EXISTS`, scalar, derived tables) and CTEs (`WITH`) are not supported in the current release — they arrive in a later release. Write the JOIN explicitly instead. See the Known Limitations & Roadmap (`../sql/known_limitations.md`) for the full list. + +--- + +## Going further + +- [ADBC Driver](adbc_driver.md) — the in-process columnar alternative (no separate server). +- Cross-Index JOIN walkthrough (`../sql/joins.md`) — the full JOIN matrix (rows 1/2/3) with worked examples. +- Multi-cluster federation operator guide (`federation_operator_guide.md`) — the Pro+ path: JOIN across separate ES clusters. +- Known Limitations & Roadmap (`../sql/known_limitations.md`) — what works in the current release vs what's coming. + +--- + ## Telemetry -The Arrow Flight SQL sidecar sends one anonymous usage ping per day (no IP, no SQL, no PII). Opt out with `softclient4es.telemetry.enabled = false` in your server HOCON config. See [Telemetry & Privacy](telemetry.md) for details. +The Arrow Flight SQL sidecar sends one anonymous usage ping per day (no IP, no SQL, no PII). Opt out with `softclient4es.telemetry.enabled = false` in the server HOCON config, the `SOFTCLIENT4ES_TELEMETRY_ENABLED=false` environment variable, or `-Dsoftclient4es.telemetry.enabled=false`. The sidecar has no connection-string opt-out — the `grpc://` URI carries no `telemetry` option. See [Telemetry & Privacy](telemetry.md) for details. --- diff --git a/documentation/client/jdbc.md b/documentation/client/jdbc.md new file mode 100644 index 00000000..9fb2f138 --- /dev/null +++ b/documentation/client/jdbc.md @@ -0,0 +1,260 @@ +# JDBC Driver + +SoftClient4ES provides a JDBC Type 4 driver that lets you connect any JDBC-compatible tool to Elasticsearch — DBeaver, IntelliJ DataGrip, Apache Superset, Tableau, and custom Java/Scala applications. + +--- + +## Connection Details + +| Property | Value | +|----------|-------| +| **JDBC URL** | `jdbc:elastic://localhost:9200` | +| **Driver class** | `app.softnetwork.elastic.jdbc.ElasticDriver` | +| **Group ID** | `app.softnetwork.elastic` | + +--- + +## Driver JARs + +Download the self-contained fat JAR for your Elasticsearch version. The JARs are Scala-version-independent and include all required dependencies. + +| Elasticsearch | Artifact | +|---------------|----------| +| ES 6.x | `softclient4es6-jdbc-driver-0.2.0.jar` | +| ES 7.x | `softclient4es7-jdbc-driver-0.2.0.jar` | +| ES 8.x | `softclient4es8-jdbc-driver-0.2.0.jar` | +| ES 9.x | `softclient4es9-jdbc-driver-0.2.0.jar` | + +### Build Tool Integration + +**Maven:** + +```xml + + app.softnetwork.elastic + softclient4es8-jdbc-driver + 0.2.0 + +``` + +**Gradle:** + +```groovy +implementation 'app.softnetwork.elastic:softclient4es8-jdbc-driver:0.2.0' +``` + +**sbt:** + +```scala +libraryDependencies += "app.softnetwork.elastic" % "softclient4es8-jdbc-driver" % "0.2.0" +``` + +--- + +## JDBC URL Format + +``` +jdbc:elastic://host:port[?param=value&...] +``` + +### Authentication Parameters + +| Parameter | Description | +|-----------|-------------| +| `user` | Username for basic authentication | +| `password` | Password for basic authentication | +| `api-key` | Elasticsearch API key | +| `bearer` | OAuth/JWT bearer token | +| `scheme` | Connection scheme (`http` or `https`, default: `http`) | + +### Examples + +``` +# Basic connection +jdbc:elastic://localhost:9200 + +# With authentication +jdbc:elastic://es.example.com:9200?user=elastic&password=changeme + +# HTTPS with API key +jdbc:elastic://es.example.com:9243?scheme=https&api-key=your-api-key +``` + +--- + +## Your first query + +A plain `SELECT` is the fastest way to confirm the connection works: + +```sql +SELECT * FROM my_index LIMIT 10; +``` + +```java +try (Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT * FROM my_index LIMIT 10")) { + while (rs.next()) { + System.out.println(rs.getString(1)); + } +} +``` + +--- + +## Your first JOIN + +Elasticsearch can't JOIN across indices — SoftClient4ES does. **Free in Community: up to 2 cross-index JOINs per query** (a 3-table JOIN). The JOIN runs in an embedded DuckDB engine; your existing single ES cluster needs no extra infrastructure. + +A cross-index INNER JOIN over two ES indices — `jdbc_join_emp` (employees) and `jdbc_join_dept` (departments): + +```sql +SELECT e.name, e.salary, d.dept_name +FROM jdbc_join_emp e +JOIN jdbc_join_dept d ON e.dept_id = d.dept_id; +-- 5 rows (the orphan employee with dept_id = 99 is dropped by the INNER JOIN) +``` + +Denormalize the joined result into a brand-new index with `CREATE TABLE … AS SELECT` (CTAS): + +```sql +CREATE TABLE jdbc_row1_ctas_join_target AS +SELECT e.name, e.salary, d.dept_name +FROM jdbc_join_emp e +JOIN jdbc_join_dept d ON e.dept_id = d.dept_id; +``` + +Upsert the joined rows into an existing index idempotently with `INSERT … ON CONFLICT (col) DO UPDATE`: + +```sql +INSERT INTO jdbc_row1_insert_join_upsert_target +SELECT e.emp_id, e.name, e.salary, d.dept_name +FROM jdbc_join_emp e +JOIN jdbc_join_dept d ON e.dept_id = d.dept_id +ON CONFLICT (emp_id) DO UPDATE; +``` + +> **Note:** `CREATE TABLE … AS … ON CONFLICT` and `INSERT … ON CONFLICT … DO NOTHING` are rejected at the JDBC boundary — use `INSERT … ON CONFLICT … DO UPDATE` for idempotent upserts. + +Bind parameters through a JOIN with a `PreparedStatement`: + +```java +PreparedStatement ps = conn.prepareStatement( + "SELECT e.name, d.dept_name " + + "FROM jdbc_join_emp e " + + "JOIN jdbc_join_dept d ON e.dept_id = d.dept_id " + + "WHERE e.salary > ?"); +ps.setDouble(1, 3500.0); // lower threshold → more rows +ResultSet rs1 = ps.executeQuery(); /* … */ rs1.close(); +ps.setDouble(1, 6500.0); // higher threshold → fewer rows (Carol = 8000 survives) +ResultSet rs2 = ps.executeQuery(); /* … */ rs2.close(); +``` + +For the full JOIN matrix, see the Cross-Index JOIN walkthrough (`../sql/joins.md`). + +--- + +## Java Example + +```java +import java.sql.*; + +String url = "jdbc:elastic://localhost:9200"; +try (Connection conn = DriverManager.getConnection(url)) { + try (Statement stmt = conn.createStatement()) { + // DDL + stmt.execute("CREATE TABLE IF NOT EXISTS demo (id INT, name VARCHAR, PRIMARY KEY (id))"); + + // DML + stmt.execute("INSERT INTO demo (id, name) VALUES (1, 'Alice'), (2, 'Bob')"); + + // DQL + try (ResultSet rs = stmt.executeQuery("SELECT * FROM demo ORDER BY id")) { + while (rs.next()) { + System.out.printf("id=%d, name=%s%n", + rs.getInt("id"), rs.getString("name")); + } + } + + // Cleanup + stmt.execute("DROP TABLE IF EXISTS demo"); + } +} +``` + +--- + +## Supported SQL + +The JDBC driver supports the full SQL Gateway syntax: + +- **DDL** — CREATE/ALTER/DROP TABLE, pipelines, watchers, enrich policies +- **DML** — INSERT, UPDATE, DELETE, COPY INTO +- **DQL** — SELECT with WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, UNION ALL, JOIN UNNEST, window functions +- **SHOW/DESCRIBE** — Tables, pipelines, watchers, enrich policies + +--- + +## BI Tool Setup + +### DBeaver + +1. Open **Database > New Database Connection** +2. Choose **Driver Manager > New** +3. Set Driver Name: `SoftClient4ES` +4. Add the fat JAR file +5. Set Driver Class: `app.softnetwork.elastic.jdbc.ElasticDriver` +6. Set URL Template: `jdbc:elastic://{host}:{port}` +7. Create a new connection using this driver + +### IntelliJ DataGrip + +1. Open **Database > + > Driver** +2. Add the fat JAR +3. Set Driver Class: `app.softnetwork.elastic.jdbc.ElasticDriver` +4. Create a new data source with URL `jdbc:elastic://localhost:9200` + +--- + +## Version compatibility + +| Driver | Scala | ES versions | Clients | Process model | +|--------|-------|-------------|---------|---------------| +| JDBC | published for Scala 2.13 | ES 6.x / 7.x / 8.x / 9.x | Java/JVM + any JDBC tool (DBeaver, DataGrip, Superset) | In-process | + +The fat JARs are **Scala-version-independent for consumers** — they bundle their own Scala runtime, so a Java application (or a Scala 2.12 project) can use the driver without matching Scala versions. You almost never need to think about the Scala axis. + +--- + +## Licensing & self-selection + +All client drivers (JDBC, ADBC, and the REPL) plus the Arrow Flight SQL sidecar are **free in Community**, including up to **2 cross-index JOINs per query** (a 3-table JOIN). A 4-table JOIN (a 3rd cross-index JOIN in one query) is rejected by the planner with a message ending `… Upgrade to Pro … See: https://portal.softclient4es.com/pricing`. + +Multi-cluster **federation** — joining across *separate* ES clusters — is **Pro+** (`maxClusters` 1 / 5 / ∞). These quickstarts cover the **single-cluster** shape, which is free. + +--- + +## What does NOT work yet + +Subqueries (`IN (SELECT …)`, `EXISTS`, scalar, derived tables) and CTEs (`WITH`) are not supported in the current release — they arrive in a later release. Write the JOIN explicitly instead. See the Known Limitations & Roadmap (`../sql/known_limitations.md`) for the full list. + +--- + +## Going further + +- [ADBC Driver](adbc_driver.md) — the Arrow-native in-process columnar alternative. +- [Arrow Flight SQL](arrow_flight_sql.md) — the gRPC server for polyglot and networked clients. +- Cross-Index JOIN walkthrough (`../sql/joins.md`) — the full JOIN matrix (rows 1/2/3) with worked examples. +- Multi-cluster federation operator guide (`federation_operator_guide.md`) — the Pro+ path: JOIN across separate ES clusters. +- Known Limitations & Roadmap (`../sql/known_limitations.md`) — what works in the current release vs what's coming. + +--- + +## Telemetry + +The JDBC driver sends one anonymous usage ping per day (no IP, no SQL, no PII). Opt out either on the JDBC URL — `jdbc:elastic://localhost:9200?telemetry=false` — or with `softclient4es.telemetry.enabled = false` on your application classpath. See [Telemetry & Privacy](telemetry.md) for the full field list and per-surface details. + +--- + +## License + +The JDBC driver is licensed under the **Elastic License 2.0** — free to use, not open source.