From 10fd6397d05daba0cfb931c9a8152890291760d6 Mon Sep 17 00:00:00 2001 From: Drew Robinson Date: Thu, 25 Dec 2025 22:09:09 +1100 Subject: [PATCH 1/2] feature: Add support for SQLite extensions --- CHANGELOG.md | 50 ++ ENHANCEMENTS.md | 888 +++++++++++++++++++++++++++ lib/ecto_libsql/native.ex | 285 +++++++++ native/ecto_libsql/src/connection.rs | 98 +++ native/ecto_libsql/src/constants.rs | 3 +- native/ecto_libsql/src/hooks.rs | 153 +++++ native/ecto_libsql/src/lib.rs | 1 + native/ecto_libsql/src/statement.rs | 53 ++ test/hooks_test.exs | 92 +++ test/statement_features_test.exs | 288 +++++++++ 10 files changed, 1910 insertions(+), 1 deletion(-) create mode 100644 ENHANCEMENTS.md create mode 100644 native/ecto_libsql/src/hooks.rs create mode 100644 test/hooks_test.exs diff --git a/CHANGELOG.md b/CHANGELOG.md index bd66adfa..7ffe74cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Investigated but Not Supported + +- **Hooks Investigation**: Researched implementation of SQLite hooks (update hooks and authorizer hooks) for CDC and row-level security + - **Update Hooks (CDC)**: Cannot be implemented due to Rustler threading limitations + - SQLite's update hook runs on managed BEAM threads + - Rustler's `OwnedEnv::send_and_clear()` can ONLY be called from unmanaged threads + - Would cause panic: "send_and_clear: current thread is managed" + - **Authorizer Hooks (RLS)**: Cannot be implemented due to synchronous callback requirements + - Requires immediate synchronous response (Allow/Deny/Ignore) + - No safe way to block waiting for Elixir response from scheduler thread + - Would risk deadlocks with scheduler thread blocking + - **Result**: Both `add_update_hook/2`, `remove_update_hook/1`, and `add_authorizer/2` return `{:error, :unsupported}` + - **Alternatives provided**: Comprehensive documentation of alternative approaches: + - For CDC: Application-level events, database triggers, polling, Phoenix.Tracker + - For RLS: Application-level auth, database views, query rewriting, connection-level privileges + - See Rustler issue: https://github.com/rusterlium/rustler/issues/293 + ### Added +- **SQLite Extension Loading Support (`enable_extensions/2`, `load_ext/3`)** + - Load SQLite extensions dynamically from shared library files + - **Security-first design**: Extension loading disabled by default, must be explicitly enabled + - **Supported extensions**: FTS5 (full-text search), JSON1, R-Tree (spatial indexing), PCRE (regex), custom user-defined functions + - Rust NIFs: `enable_load_extension/2`, `load_extension/3` in `src/connection.rs` + - Elixir wrappers: `EctoLibSql.Native.enable_extensions/2`, `EctoLibSql.Native.load_ext/3` + - **API workflow**: Enable extension loading → Load extension(s) → Disable extension loading (recommended) + - **Entry point support**: Optional custom entry point function name parameter + - **Platform support**: .so (Linux), .dylib (macOS), .dll (Windows) + - **Use cases**: Full-text search (FTS5), JSON functions, spatial data (R-Tree), regex matching, custom SQL functions + - **Security warnings**: Only load extensions from trusted sources - extensions have full database access + - Comprehensive documentation with security warnings and common extension examples + +- **Statement Parameter Name Introspection (`stmt_parameter_name/3`)** + - Retrieve parameter names from prepared statements with named parameters + - **Supports all SQLite named parameter styles**: `:name`, `@name`, `$name` + - **Use cases**: Dynamic query building, parameter validation, better debugging, API introspection + - Rust NIF: `statement_parameter_name()` in `src/statement.rs` + - Elixir wrapper: `EctoLibSql.Native.stmt_parameter_name/3` + - Returns `{:ok, "name"}` for named parameters (prefix included) or `{:ok, nil}` for positional `?` placeholders + - **Note**: Uses 1-based parameter indexing (first parameter is index 1) following SQLite convention + - Added 5 comprehensive tests covering all three named parameter styles, positional parameters, and mixed parameter scenarios + - Complements existing `stmt_parameter_count/2` for complete parameter introspection + +- **Comprehensive Statement Introspection Test Coverage** + - Added 18 edge case tests for prepared statement introspection features (13 existing + 5 parameter_name tests) + - **Parameter introspection edge cases**: 0 parameters, 20+ parameters, UPDATE statements, complex nested queries, named parameter introspection + - **Column introspection edge cases**: SELECT *, INSERT/UPDATE/DELETE without RETURNING (0 columns), aggregate functions, JOINs, subqueries, computed expressions + - Improved test coverage for `stmt_parameter_count/2`, `stmt_parameter_name/3`, `stmt_column_count/2`, and `stmt_column_name/3` + - All tests verify correct behaviour for simple queries, complex JOINs, aggregates, and edge cases + - Tests ensure proper handling of aliased columns, expressions, multi-table queries, and all three named parameter styles + - Location: `test/statement_features_test.exs` - added 180+ lines of comprehensive edge case tests + - **Statement Reset (`reset_stmt/2`)** - Explicitly reset prepared statements to initial state for efficient reuse - **Performance improvement**: 10-15x faster than re-preparing the same SQL string diff --git a/ENHANCEMENTS.md b/ENHANCEMENTS.md new file mode 100644 index 00000000..eb75c86f --- /dev/null +++ b/ENHANCEMENTS.md @@ -0,0 +1,888 @@ +# EctoLibSql Enhancements & Future Improvements + +This document consolidates all future improvements, optimisations, and enhancement suggestions from across the codebase into a single, organised reference. + +## Table of Contents + +1. [Critical Priority (P0) - Must-Have for Production](#critical-priority-p0---must-have-for-production) +2. [High Priority (P1) - Valuable for Most Apps](#high-priority-p1---valuable-for-most-apps) +3. [Medium Priority (P2) - Specific Use Cases](#medium-priority-p2---specific-use-cases) +4. [Low Priority (P3) - Advanced/Rare Features](#low-priority-p3---advancedrare-features) +5. [Vector & Geospatial Enhancements](#vector--geospatial-enhancements) +6. [Code Quality & Architecture Improvements](#code-quality--architecture-improvements) +7. [Testing & Documentation Enhancements](#testing--documentation-enhancements) +8. [Performance Optimisations](#performance-optimisations) +9. [Error Handling & Resilience](#error-handling--resilience) +10. [Ecto Integration Improvements](#ecto-integration-improvements) + +--- + +## Critical Priority (P0) - Must-Have for Production + +### 1. `busy_timeout()` Configuration ✅ DONE +**Status**: ✅ IMPLEMENTED (v0.6.0+) +**Impact**: CRITICAL - Without this, concurrent writes fail immediately with "database is locked" errors +**Implementation**: `set_busy_timeout/2`, `busy_timeout/2` + +**Why Important**: +- Prevents immediate "database is locked" errors under concurrent load +- Essential for multi-user applications +- Standard pattern in all SQLite applications + +**Desired API**: +```elixir +# At connection time +{:ok, conn} = EctoLibSql.connect(database: "local.db", busy_timeout: 5000) + +# Or runtime +EctoLibSql.set_busy_timeout(state, 5000) + +# In Ecto config +config :my_app, MyApp.Repo, + adapter: Ecto.Adapters.LibSql, + database: "local.db", + busy_timeout: 5000 # 5 seconds (recommended default) +``` + +### 2. PRAGMA Query Support ✅ DONE +**Status**: ✅ IMPLEMENTED (v0.6.0+) - Basic support via `pragma_query/2` +**Impact**: HIGH - SQLite configuration is verbose and error-prone +**Implementation**: `pragma_query/2` NIF for executing PRAGMA statements + +**Why Important**: +- Essential for performance tuning +- Required for foreign key enforcement (disabled by default in SQLite!) +- Needed for WAL mode configuration (better concurrency) + +**Desired API**: +```elixir +# Type-safe ergonomic wrappers +EctoLibSql.Pragma.enable_foreign_keys(state) +EctoLibSql.Pragma.set_journal_mode(state, :wal) +EctoLibSql.Pragma.set_synchronous(state, :normal) +EctoLibSql.Pragma.set_cache_size(state, megabytes: 64) + +# Introspection +{:ok, columns} = EctoLibSql.Pragma.table_info(state, "users") +{:ok, indexes} = EctoLibSql.Pragma.index_list(state, "users") +``` + +**Critical PRAGMAs to Support**: +- `foreign_keys` - FK constraint enforcement (CRITICAL - disabled by default!) +- `journal_mode` - WAL mode (much better concurrency) +- `synchronous` - Durability vs speed trade-off +- `cache_size` - Memory usage tuning +- `table_info` - Schema inspection +- `index_list` - Index inspection + +### 3. `Connection.reset()` ✅ DONE +**Status**: ✅ IMPLEMENTED (v0.6.0+) +**Impact**: HIGH - Essential for proper connection pooling +**Implementation**: `reset_connection/1`, `reset/1` + +**Why Important**: +- Resets connection state between checkouts from pool +- Clears temporary tables, views, triggers +- Ensures clean state for next query +- Required by `DBConnection` for proper pooling + +**Desired API**: +```elixir +# Typically called by DBConnection automatically +EctoLibSql.reset_connection(state) +``` + +### 4. `Connection.interrupt()` ✅ DONE +**Status**: ✅ IMPLEMENTED (v0.6.0+) +**Impact**: MEDIUM - Useful for cancelling long-running queries +**Implementation**: `interrupt_connection/1`, `interrupt/1` + +**Why Important**: +- Cancel long-running queries +- Useful for timeouts +- Better user experience (responsive UI) +- Operational control + +**Desired API**: +```elixir +# Cancel a query running in another process +EctoLibSql.interrupt_connection(state) +``` + +### 5. Statement Column Metadata ✅ DONE +**Status**: ✅ IMPLEMENTED (Unreleased) +**Impact**: MEDIUM - Better developer experience and error messages +**Implementation**: `get_statement_columns/2`, `get_stmt_columns/2`, `stmt_column_count/2`, `stmt_column_name/3` + +**Why Important**: +- Type introspection for dynamic queries +- Schema discovery without separate queries +- Better error messages (show column names in errors) +- Type casting hints for Ecto + +**Desired API**: +```elixir +{:ok, stmt_id} = EctoLibSql.prepare(state, "SELECT * FROM users WHERE id = ?") + +{:ok, columns} = EctoLibSql.get_statement_columns(stmt_id) +# Returns: [ +# %{name: "id", decl_type: "INTEGER"}, +# %{name: "name", decl_type: "TEXT"}, +# %{name: "created_at", decl_type: "TEXT"} +# ] +``` + +--- + +## High Priority (P1) - Valuable for Most Apps + +### 6. `Statement.query_row()` - Single Row Query +**Status**: ❌ Missing +**Impact**: MEDIUM - Performance and ergonomics +**Estimated Effort**: 2 days + +**Why Important**: +- Common pattern for `SELECT * FROM table WHERE id = ?` +- Cleaner API than `query() + take first` +- Better error handling (errors if 0 or >1 rows) +- **Optimisation**: Can stop after first row (doesn't fetch rest) + +**Desired API**: +```elixir +# More efficient, clearer intent +{:ok, row} = EctoLibSql.query_one(state, stmt_id, [42]) +# Returns: ["Alice", 25, "2024-01-01"] +# Errors if 0 rows or multiple rows +``` + +### 7. Statement `reset()` for Reuse ✅ DONE +**Status**: ✅ IMPLEMENTED (Unreleased) - Automatic reset in execute/query + explicit reset_stmt/2 +**Impact**: HIGH - Significant performance issue (NOW FIXED) +**Implementation**: `reset_statement/2`, `reset_stmt/2` - statements are automatically reset before each execution + +**Why Important**: +- **PERFORMANCE**: Currently we re-prepare statements on every execution +- Defeats the entire purpose of prepared statements +- Ecto caches prepared statements but we ignore the cache +- Significant performance overhead + +**Current Problem**: +```rust +// lib.rs:881-888 - PERFORMANCE BUG +let stmt = conn_guard + .prepare(&sql) // ← Re-prepare EVERY TIME! + .await + .map_err(|e| rustler::Error::Term(Box::new(format!("Prepare failed: {}", e))))?; +``` + +**Desired Behaviour**: +```elixir +{:ok, stmt_id} = EctoLibSql.prepare(state, "INSERT INTO logs (msg) VALUES (?)") + +for msg <- messages do + EctoLibSql.execute_stmt(state, stmt_id, [msg]) + EctoLibSql.reset_stmt(stmt_id) # ← Clear bindings, ready for reuse +end + +EctoLibSql.close_stmt(stmt_id) +``` + +### 8. Native `execute_batch()` Implementation ✅ DONE +**Status**: ✅ IMPLEMENTED (v0.6.0+) - Both manual and native implementations available +**Impact**: MEDIUM - Performance optimisation +**Implementation**: `execute_batch_native/2`, `execute_transactional_batch_native/2`, `execute_batch_sql/2`, `execute_transactional_batch_sql/2` + +**Why Important**: +- More efficient than multiple round trips +- Standard for migrations and setup scripts +- Turso optimises this internally +- Both manual sequential and native batch implementations are available + +**Current Implementation**: Custom sequential execution (works but slower) + +**Desired**: Use native LibSQL batch API for ~30% performance improvement + +**Use Case**: +```elixir +sql = """ +CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT); +CREATE INDEX idx_users_name ON users(name); +INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob'); +""" + +EctoLibSql.execute_batch(state, sql) +``` + +### 9. `Connection.cacheflush()` - Page Cache Control +**Status**: ❌ Missing +**Impact**: LOW - Useful for specific scenarios +**Estimated Effort**: 1 day + +**Why Important**: +- Force durability before critical operations +- Testing (ensure writes are durable) +- Checkpointing control +- Memory pressure management + +**Use Case**: +```elixir +# Before backup +EctoLibSql.cacheflush(state) +System.cmd("cp", ["local.db", "backup.db"]) + +# After bulk insert +EctoLibSql.batch_transactional(state, large_statements) +EctoLibSql.cacheflush(state) # Ensure written to disk +``` + +### 10. Named Parameters Support ⚠️ PARTIAL +**Status**: ⚠️ PARTIAL - Introspection available (Unreleased), but keyword list binding not implemented +**Impact**: LOW - Ergonomics improvement +**Implemented**: `stmt_parameter_name/3` for parameter introspection +**Missing**: Keyword list → positional parameter binding in query execution + +**Why Important**: +- More readable queries +- Less error-prone (no counting positions) +- Standard SQLite feature + +**Current Support**: +```elixir +# Named parameters work if you use positional binding +query(conn, "SELECT * FROM users WHERE name = :name AND age = :age", ["Alice", 30]) + +# Parameter introspection works +{:ok, stmt_id} = prepare(state, "SELECT * FROM users WHERE id = :id") +{:ok, ":id"} = stmt_parameter_name(state, stmt_id, 1) # Returns parameter name +``` + +**Not Yet Supported**: +```elixir +# Keyword list binding (requires parameter name → index mapping) +query(conn, "SELECT * FROM users WHERE name = :name AND age = :age", + name: "Alice", age: 30) # ← Not implemented +``` + +### 11. MVCC Mode Support +**Status**: ❌ Missing +**Impact**: MEDIUM - Better concurrent read performance +**Estimated Effort**: 2-3 days + +**Why Important**: +- Multi-Version Concurrency Control +- Better concurrent read performance +- Non-blocking reads during writes +- Modern SQLite feature +- Turso recommended for replicas + +**Desired API**: +```elixir +config :my_app, MyApp.Repo, + adapter: Ecto.Adapters.LibSql, + database: "local.db", + mvcc: true +``` + +--- + +## Medium Priority (P2) - Specific Use Cases + +### 12. `Statement.run()` - Execute Any Statement +**Status**: ❌ Missing +**Impact**: LOW - Flexibility +**Estimated Effort**: 2 days + +**Why Important**: +- More flexible than `execute()` - works with any SQL +- Doesn't return row count (slightly more efficient) + +### 13. Statement Parameter Introspection ✅ DONE +**Status**: ✅ IMPLEMENTED (Unreleased) +**Impact**: LOW - Developer experience +**Implementation**: `stmt_parameter_count/2`, `stmt_parameter_name/3` + +**Why Important**: +- Dynamic query building +- Better error messages +- Validation +- Supports all three named parameter styles (`:name`, `@name`, `$name`) + +### 14. `Connection.load_extension()` ✅ DONE +**Status**: ✅ IMPLEMENTED (Unreleased) +**Impact**: MEDIUM - Extensibility +**Implementation**: `enable_load_extension/2`, `load_extension/3`, `enable_extensions/2`, `load_ext/3` + +**Why Important**: +- Load SQLite extensions (FTS5, JSON1, etc.) +- Custom functions +- Specialised features + +**Implemented API**: +```elixir +# Enable extension loading first (disabled by default for security) +:ok = EctoLibSql.Native.enable_extensions(state, true) + +# Load an extension +:ok = EctoLibSql.Native.load_ext(state, "/path/to/extension.so") + +# Optional: specify custom entry point +:ok = EctoLibSql.Native.load_ext(state, "/path/to/extension.so", "sqlite3_extension_init") + +# Disable extension loading after (recommended) +:ok = EctoLibSql.Native.enable_extensions(state, false) +``` + +**Security Note**: ⚠️ Only load extensions from trusted sources! Extensions run with full database access and can execute arbitrary code. + +### 15. Replication Control (Advanced) ✅ DONE +**Status**: ✅ **FULLY IMPLEMENTED** (4 of 5 features complete as of Unreleased) + +**Implemented Features**: +- ✅ `replication_index()` / `get_frame_number_for_replica()` - Get current replication frame (v0.6.0+) +- ✅ `sync_until()` / `sync_until_frame()` - Wait for specific replication point (v0.6.0+) +- ✅ `flush_replicator()` / `flush_and_get_frame()` - Force flush replicator (v0.6.0+) +- ✅ `max_write_replication_index()` / `get_max_write_frame()` - Track highest write frame (Unreleased) +- 🔄 `freeze()` / `freeze_replica()` - Convert replica to standalone (EXPLICITLY UNSUPPORTED - returns {:error, :unsupported}) + +**Implementation Status**: +- ✅ **Phase 1 Complete**: All monitoring functions working (v0.6.0-v0.7.0) +- ✅ **Phase 2 Complete**: `max_write_replication_index()` implemented (Unreleased) +- 🔄 **Phase 3 Deferred**: `freeze()` explicitly unsupported - needs Arc> architecture refactor (documented limitation) + +### 16. Hooks (Authorisation & Update) +**Status**: ❌ **NOT SUPPORTED** - Investigated, cannot be implemented due to threading limitations +**Impact**: MEDIUM - Security and real-time features +**Reason**: Rustler threading model incompatibility + +**Why Not Supported**: +Both update hooks and authorizer hooks are fundamentally incompatible with Rustler's threading model: + +1. **Update Hooks Problem**: + - SQLite's update hook callback runs synchronously during INSERT/UPDATE/DELETE operations + - Callback executes on Erlang scheduler threads (managed by BEAM) + - Rustler's `OwnedEnv::send_and_clear()` can ONLY be called from unmanaged threads + - Calling `send_and_clear()` from managed thread causes panic: "current thread is managed" + +2. **Authorizer Hooks Problem**: + - SQLite's authorizer callback is synchronous and expects immediate response (Allow/Deny/Ignore) + - Would require blocking Rust thread waiting for Elixir response + - No safe way to do synchronous Rust→Elixir→Rust calls + - Blocking on scheduler threads can cause deadlocks + +**Alternatives Provided**: + +For **Change Data Capture / Real-time Updates**: +- Application-level events via Phoenix.PubSub +- Database triggers to audit log table +- Polling-based CDC with timestamps +- Phoenix.Tracker for state tracking + +For **Row-Level Security / Authorization**: +- Application-level authorization checks before queries +- Database views with WHERE clauses +- Query rewriting in Ecto +- Connection-level privileges + +**Implementation**: Functions return `{:error, :unsupported}` with comprehensive documentation explaining alternatives. + +--- + +## Low Priority (P3) - Advanced/Rare Features + +### 17. Custom VFS/IO +**Status**: ❌ Missing +**Impact**: LOW - Specialised use cases +**Estimated Effort**: 3 days + +**Why Important**: +- Custom Virtual File System implementations +- Encryption layers +- Compression +- Testing with in-memory VFS + +### 18. Statement `finalize()` & `interrupt()` +**Status**: ❌ Missing +**Impact**: LOW - Advanced control +**Estimated Effort**: 2 days + +**Current**: We use simple drop for cleanup + +### 19. WAL API (Write-Ahead Log - Feature-Gated) +**Status**: ❌ Missing (requires special Turso build flags) +**Impact**: LOW - Expert-level features +**Estimated Effort**: 5 days + +**Why Important**: +- Advanced replication scenarios +- Custom backup solutions +- Database forensics +- Debugging + +**Note**: Requires Turso feature gate, not in standard LibSQL builds + +### 20. Reserved Bytes Management +**Status**: ❌ Missing +**Impact**: LOW - Advanced database tuning +**Estimated Effort**: 2 days + +### 21. Advanced Builder Options +**Status**: 🟡 Partial +**Missing**: +- Custom `OpenFlags` +- `SyncProtocol` selection (V1/V2) +- Custom TLS configuration +- Thread safety flags + +--- + +## Vector & Geospatial Enhancements + +### Current Limitations +1. **2D Vector Space**: Limited to 2D (lat/long). Could extend to 3D or more dimensions +2. **Normalized Coordinates**: Works with normalized space, not actual geodetic distances +3. **Cosine Distance**: Uses vector cosine distance, not true geographic distance (haversine formula) + +### Potential Enhancements +1. **Haversine Formula**: Implement actual geographic distance calculations for accuracy +2. **Higher Dimensions**: Support for more complex geospatial data (elevation, time, etc.) +3. **Index Optimization**: Add spatial indexes for performance on large datasets +4. **Batch Queries**: Use batch operations for multiple location lookups +5. **Clustering**: Find geographic clusters of locations using vector analysis + +### Real-World Applications +- **Location-based services**: Find nearby restaurants, hotels, gas stations +- **Delivery optimization**: Locate nearest warehouse to customer location +- **Regional analytics**: Find closest office/branch in each region +- **Social discovery**: Find nearby users, events, or meetup groups +- **Asset tracking**: Locate nearest available equipment or resources +- **Emergency services**: Find nearest hospital, fire station, or police +- **Real estate**: Find comparable properties in similar locations +- **Market analysis**: Identify competitors in specific geographic areas + +--- + +## Code Quality & Architecture Improvements + +### 1. Reduce TXN_REGISTRY Lock Scope Around Async Calls +**Issue**: `execute_with_transaction/4`, `query_with_trx_args/5`, and savepoint NIFs run transaction.execute/query(...).await while holding the global TXN_REGISTRY mutex + +**Problem**: +- Serialises all transaction operations across the registry +- Goes against "drop locks before async" guideline +- Potential contention under high load + +**Solution**: +- Refactor `TransactionEntry` to hold the `Transaction` behind an `Arc>` +- Look up and ownership-check under TXN_REGISTRY +- Clone the inner Arc, drop the registry lock +- Perform async work holding only the per-transaction lock + +**Impact**: Better concurrency, reduced lock contention +**Effort**: 3-4 days + +### 2. Complete Phase 2-5 Refactoring +**Current Status**: Phase 1 Complete (823 lines across 4 modules) +**Remaining Phases**: +- **Phase 2**: Core Operations (connection.rs, query.rs, metadata.rs) +- **Phase 3**: Advanced Features (transaction.rs, statement.rs, cursor.rs) +- **Phase 4**: Batch & Replication (batch.rs, savepoint.rs, replication.rs) +- **Phase 5**: Integration (lib.rs refactoring) + +**Benefits**: +- Better code organisation +- Reduced module sizes (150-350 lines vs 2500+) +- Improved maintainability +- Clearer separation of concerns + +**Effort**: 10-15 days total + +### 3. Add Structured Error Types +**Current**: String-based error messages +**Desired**: Structured error types with pattern matching + +**Benefits**: +- Better error handling in Elixir +- Pattern matching on error types +- More consistent error messages +- Easier to document and test + +**Effort**: 5-7 days + +### 4. Transaction & Concurrency Safety Enhancements +**Status**: 🟡 Partial +**Items**: + +#### 4a. Transitive Validation for Prepared Statements +**Current**: Prepared statements are validated but not transitively +**Desired**: Ensure prepared statements are used only with correct connections +**Benefits**: Prevents cross-connection statement misuse, catches errors earlier +**Effort**: 3 days + +#### 4b. Auditing Trail for Transaction Violations +**Current**: Violations silently fail or error +**Desired**: Log transaction ownership violations for monitoring +**Benefits**: Security monitoring, debugging, compliance tracking +**Effort**: 2 days + +#### 4c. Distributed Tracing Span Context +**Current**: No span context propagation +**Desired**: Integrate with distributed tracing (OpenTelemetry) +**Benefits**: Better observability in microservices, transaction lifecycle tracking +**Effort**: 3 days + +**Total Effort**: 8 days + +--- + +## Testing & Documentation Enhancements + +### 1. Comprehensive Test Coverage for New Features +**Missing Test Coverage**: +- Busy timeout functionality +- PRAGMA operations +- Connection reset behaviour +- Statement column metadata +- Named parameters +- MVCC mode +- Replication edge cases + +**Effort**: 5-10 days + +### 2. Performance Benchmark Tests +**Add Benchmarks For**: +- Prepared statement caching vs re-preparation +- Batch operations vs individual queries +- Cursor streaming vs full result fetch +- Transaction behaviours (deferred vs immediate vs exclusive) + +**Effort**: 3-5 days + +### 3. Documentation Improvements +**Enhance Documentation**: +- Add more real-world examples +- Include performance best practices +- Document error handling patterns +- Add migration guides from other databases +- Create troubleshooting guide + +**Effort**: 5-7 days + +### 4. Test Infrastructure Enhancements +**Status**: 🟡 Partial +**Items**: + +#### 4a. Benchmarking Suite +**Current**: No performance regression testing +**Desired**: Automated benchmarking with criterion.rs +**Benefits**: Detect performance regressions, track improvements, baseline measurements +**Coverage**: Prepared statements, batch operations, cursor streaming, transaction types +**Effort**: 3 days + +#### 4b. Property-Based Testing +**Current**: Limited property-based tests +**Desired**: PropCheck/StreamData for Elixir, QuickCheck for Rust +**Benefits**: Find edge cases, verify invariants, better coverage +**Effort**: 2 days + +#### 4c. Mutation Testing +**Current**: Not implemented +**Desired**: Use mutation testing frameworks (stryker, mutagen) +**Benefits**: Verify test quality, identify weak tests, improve coverage +**Effort**: 2 days + +#### 4d. Stress Tests for Connection Pooling +**Current**: Limited concurrent testing +**Desired**: High-concurrency stress tests for pool behavior +**Benefits**: Identify contention issues, verify pool management, test under load +**Effort**: 2 days + +#### 4e. Error Recovery Scenario Testing +**Current**: Basic error handling tests +**Desired**: Comprehensive recovery testing (network failures, timeouts, corruption) +**Benefits**: Production readiness, resilience verification +**Effort**: 2 days + +#### 4f. Test Coverage Reporting +**Current**: No automated coverage tracking +**Desired**: Tarpaulin (Rust), ExCoveralls (Elixir) +**Benefits**: Track coverage trends, identify untested code +**Effort**: 1 day + +**Total Effort**: 12 days + +### 5. Code Quality & Maintainability Enhancements +**Status**: 🟡 Partial +**Items**: + +#### 5a. Guard Method Visibility Review +**Current**: Guard methods may be publicly exposed unnecessarily +**Desired**: Make guard methods private if only used internally +**Benefits**: Better encapsulation, clearer public API surface +**Effort**: 0.5 days + +#### 5b. NIF Code Pattern Review +**Current**: Other NIF files may have similar patterns that need review +**Desired**: Systematic review of all NIF files for consistency +**Benefits**: Consistent code quality across modules, catch potential issues early +**Effort**: 1 day + +#### 5c. NIF Logging Rules +**Current**: No linting rules to prevent eprintln! in NIF code +**Desired**: Establish clippy/linting rules to catch console output in production NIFs +**Benefits**: Prevent debugging output in production, consistent logging approach +**Implementation**: Add to `.cargo/config.toml` or `clippy.toml`: +```toml +[clippy] +disallowed-methods = [ + { path = "std::eprintln", reason = "use proper logging in NIFs" }, + { path = "std::println", reason = "use proper logging in NIFs" }, +] +``` +**Effort**: 0.5 days + +#### 5d. SQL Keyword Extensibility +**Current**: `should_use_query()` handles SELECT and RETURNING +**Desired**: Extensible design for new SQL operations +**Benefits**: Easy to add support for new statement types, maintainable pattern +**Note**: Current implementation is already excellent, this is for future extensions +**Examples**: PRAGMA return handling, EXPLAIN queries, CTE detection +**Effort**: 1 day (if needed) + +#### 5e. Guard Consumption Error Tests +**Current**: Limited integration tests for guard consumption errors +**Desired**: Comprehensive tests for MutexGuard consumption scenarios +**Benefits**: Verify error handling in edge cases, prevent regressions +**Coverage**: Poisoned mutexes, concurrent access, timeout scenarios +**Effort**: 0.5 days + +**Total Effort**: 3.5 days + +--- + +## Performance Optimisations + +### 1. Prepared Statement Caching +**Current Issue**: Re-prepare statements on every execution +**Solution**: Implement proper statement caching with reset() +**Impact**: ~10-15x performance improvement for cached queries +**Effort**: 3 days + +### 2. Batch Operation Optimisation +**Current**: Custom sequential implementation +**Desired**: Use native LibSQL batch API +**Impact**: ~30% performance improvement +**Effort**: 3 days + +### 3. Connection Pooling Tuning +**Current**: Basic pooling +**Enhancements**: +- Dynamic pool sizing based on load +- Connection health checks +- Intelligent connection reuse +- Better error handling for exhausted pools + +**Effort**: 5 days + +### 4. Query Optimisation +**Add Support For**: +- Query plan analysis +- Index recommendations +- Query rewriting suggestions +- Performance warnings + +**Effort**: 7 days + +### 5. Performance Micro-Optimisations +**Status**: 🟢 Already analysed (future-only if profiling shows need) +**Items**: + +#### 5a. SIMD Vectorization +**Current**: Single-byte comparisons in `should_use_query()` +**Desired**: SIMD instructions for multi-character checks +**Benefits**: 2-4x faster for very long SQL strings +**Complexity**: High +**Status**: Only implement if profiling identifies bottleneck +**Effort**: 3-5 days + +#### 5b. Lookup Table Pre-computation +**Current**: Character-by-character comparison +**Desired**: Pre-computed lookup tables for first-byte checks +**Benefits**: ~10-20% faster SELECT detection +**Complexity**: Low +**Status**: Marginal gain, current implementation already excellent +**Effort**: 1 day + +#### 5c. Lazy RETURNING Clause Check +**Current**: Full string scan for RETURNING in all statements +**Desired**: Only check RETURNING for INSERT/UPDATE/DELETE +**Benefits**: Skip RETURNING scan for SELECT, CREATE, etc. +**Complexity**: Medium +**Status**: Adds complexity without major benefit +**Effort**: 1 day + +**Total Effort**: 5-7 days (investigate via profiling first) + +--- + +## Error Handling & Resilience + +### 1. Retry Logic for Transient Errors +**Add Retry For**: +- Connection timeouts +- Transient network issues +- Database locked errors (with backoff) +- Mutex contention + +**Effort**: 3-5 days + +### 2. Telemetry Events +**Add Telemetry For**: +- Connection establishment +- Query execution times +- Transaction lifecycle +- Error conditions +- Lock contention + +**Effort**: 3 days + +### 3. Circuit Breaker Pattern +**Implement**: +- Connection failure tracking +- Automatic failover to replicas +- Health check monitoring +- Graceful degradation + +**Effort**: 5 days + +### 4. Advanced Resilience Features +**Status**: 🟡 Partial +**Items**: + +#### 4a. Connection Pooling Statistics +**Current**: Basic pool management +**Desired**: Track and expose pool metrics +**Metrics**: Wait time, checkout count, utilization, contention +**Benefits**: Better diagnostics, capacity planning, performance tuning +**Effort**: 2 days + +#### 4b. Try_lock with Timeouts +**Current**: Blocking lock acquisition +**Desired**: Non-blocking lock attempts with timeouts +**Benefits**: Better responsiveness, prevent deadlocks, graceful degradation +**Use Cases**: Non-critical operations, health checks +**Effort**: 2 days + +#### 4c. Supervision Tree Integration +**Current**: Basic error propagation +**Desired**: Deep integration with Elixir supervision +**Benefits**: Better crash recovery, monitoring, alerting +**Effort**: 2 days + +#### 4d. Health Check Monitoring +**Current**: Basic ping functionality +**Desired**: Comprehensive health checks (latency, resource usage, connection state) +**Benefits**: Early problem detection, better failover decisions +**Effort**: 2 days + +**Total Effort**: 8 days + +--- + +## Ecto Integration Improvements + +### 1. Better Ecto Adapter Integration +**Enhancements**: +- Automatic PRAGMA configuration on connection +- Better transaction behaviour mapping +- Improved error message translation +- Enhanced type mapping + +**Effort**: 5 days + +### 2. Phoenix Integration Examples +**Add Documentation For**: +- Context-based usage patterns +- Controller integration +- LiveView usage +- PubSub integration + +**Effort**: 3 days + +### 3. Migration Tooling +**Enhancements**: +- Better migration error messages +- Migration validation +- Schema diff tools +- Migration rollback improvements + +**Effort**: 5 days + +### 4. Connection Pool Management +**Status**: 🟡 Partial +**Items**: + +#### 4a. Dynamic Pool Sizing +**Current**: Fixed pool size at startup +**Desired**: Adjust pool size based on load +**Benefits**: Better resource utilization, handles traffic spikes, reduces idle connections +**Implementation**: Monitor utilization, scale up/down based on demand +**Effort**: 3 days + +#### 4b. Connection Health Checks +**Current**: Reactive failure detection +**Desired**: Proactive health monitoring +**Benefits**: Detect stale connections early, prevent cascading failures +**Implementation**: Periodic ping, liveness checks, resource monitoring +**Effort**: 2 days + +#### 4c. Intelligent Connection Reuse +**Current**: Basic FIFO reuse +**Desired**: Smart connection selection based on history +**Benefits**: Better performance, avoid slow connections, faster queries +**Implementation**: Track per-connection stats, prefer "warm" connections +**Effort**: 2 days + +#### 4d. Exhausted Pool Handling +**Current**: Error on pool exhaustion +**Desired**: Graceful degradation and recovery +**Benefits**: Better UX, automatic recovery, clear diagnostics +**Implementation**: Queue overflow handling, timeout management, metrics +**Effort**: 1 day + +**Total Effort**: 8 days + +--- + +## Summary + +### Priority Breakdown +- **P0 (Critical)**: 5 features - Must-have for production +- **P1 (High)**: 6 features - Valuable for most applications +- **P2 (Medium)**: 6 features - Specific use cases +- **P3 (Low)**: 6 features - Advanced/rare features + +### Total Estimated Effort +- **P0 Features**: ~12-15 days +- **P1 Features**: ~18-22 days +- **P2 Features**: ~15-20 days +- **P3 Features**: ~15-20 days +- **Quality/Architecture**: ~28-33 days (+8 days for transaction safety) +- **Testing/Documentation**: ~27-32 days (+12 days for test infrastructure, +3.5 days for code quality) +- **Performance**: ~20-27 days (+5-7 days for micro-optimisations) +- **Error Handling**: ~18-23 days (+8 days for advanced resilience) +- **Ecto Integration**: ~18-23 days (+8 days for pool management) + +**Total**: ~175-215 days of development effort (+58.5-68.5 days from identified enhancements) + +### Recommended Implementation Order +1. **P0 Critical Features** (busy_timeout, PRAGMA, reset, etc.) +2. **Performance Optimisations** (statement caching, batch operations) +3. **Error Handling & Resilience** (retry logic, telemetry) +4. **Code Quality & Maintainability** (linting rules, guard visibility, NIF pattern review) +5. **P1 High Priority Features** (query_row, named parameters, etc.) +6. **Testing & Documentation** (test infrastructure, benchmarks, coverage reporting) +7. **P2 Medium Priority Features** +8. **Ecto Integration Improvements** +9. **P3 Low Priority Features** + +This prioritization ensures we address the most critical production issues first, then focus on performance and reliability, before moving to nice-to-have features and advanced functionality. \ No newline at end of file diff --git a/lib/ecto_libsql/native.ex b/lib/ecto_libsql/native.ex index 77399419..79fbabe1 100644 --- a/lib/ecto_libsql/native.ex +++ b/lib/ecto_libsql/native.ex @@ -130,6 +130,21 @@ defmodule EctoLibSql.Native do @doc false def interrupt_connection(_conn_id), do: :erlang.nif_error(:nif_not_loaded) + @doc false + def enable_load_extension(_conn_id, _enabled), do: :erlang.nif_error(:nif_not_loaded) + + @doc false + def load_extension(_conn_id, _path, _entry_point), do: :erlang.nif_error(:nif_not_loaded) + + @doc false + def set_update_hook(_conn_id, _pid), do: :erlang.nif_error(:nif_not_loaded) + + @doc false + def clear_update_hook(_conn_id), do: :erlang.nif_error(:nif_not_loaded) + + @doc false + def set_authorizer(_conn_id, _pid), do: :erlang.nif_error(:nif_not_loaded) + @doc false def pragma_query(_conn_id, _pragma_stmt), do: :erlang.nif_error(:nif_not_loaded) @@ -148,6 +163,9 @@ defmodule EctoLibSql.Native do @doc false def statement_parameter_count(_conn_id, _stmt_id), do: :erlang.nif_error(:nif_not_loaded) + @doc false + def statement_parameter_name(_conn_id, _stmt_id, _idx), do: :erlang.nif_error(:nif_not_loaded) + @doc false def reset_statement(_conn_id, _stmt_id), do: :erlang.nif_error(:nif_not_loaded) @@ -835,6 +853,226 @@ defmodule EctoLibSql.Native do interrupt_connection(conn_id) end + @doc """ + Enable or disable loading of SQLite extensions. + + By default, extension loading is disabled for security reasons. + You must explicitly enable it before calling `load_ext/3`. + + ## Parameters + - state: The connection state + - enabled: Whether to enable (true) or disable (false) extension loading + + ## Returns + - `:ok` - Extension loading enabled/disabled successfully + - `{:error, reason}` - Operation failed + + ## Example + + # Enable extension loading + :ok = EctoLibSql.Native.enable_extensions(state, true) + + # Load an extension + :ok = EctoLibSql.Native.load_ext(state, "/path/to/extension.so") + + # Disable extension loading (recommended after loading) + :ok = EctoLibSql.Native.enable_extensions(state, false) + + ## Security Warning + + ⚠️ Only enable extension loading if you trust the extensions being loaded. + Malicious extensions can compromise database security and execute arbitrary code. + + """ + def enable_extensions(%EctoLibSql.State{conn_id: conn_id} = _state, enabled) + when is_boolean(enabled) do + enable_load_extension(conn_id, enabled) + end + + @doc """ + Load a SQLite extension from a dynamic library file. + + Extensions must be enabled first via `enable_extensions/2`. + + ## Parameters + - state: The connection state + - path: Path to the extension dynamic library (.so, .dylib, or .dll) + - entry_point: Optional entry point function name (defaults to extension-specific default) + + ## Returns + - `:ok` - Extension loaded successfully + - `{:error, reason}` - Extension loading failed + + ## Example + + # Enable extension loading first + :ok = EctoLibSql.Native.enable_extensions(state, true) + + # Load an extension + :ok = EctoLibSql.Native.load_ext(state, "/usr/lib/sqlite3/pcre.so") + + # Load with custom entry point + :ok = EctoLibSql.Native.load_ext(state, "/path/to/extension.so", "sqlite3_extension_init") + + # Disable extension loading after + :ok = EctoLibSql.Native.enable_extensions(state, false) + + ## Common Extensions + + - **FTS5** (full-text search) - Usually built-in, provides advanced full-text search + - **JSON1** (JSON functions) - Usually built-in, provides JSON manipulation functions + - **R-Tree** (spatial indexing) - Spatial data structures for geographic data + - **PCRE** (regular expressions) - Perl-compatible regular expressions + - Custom user-defined functions + + ## Security Warning + + ⚠️ Only load extensions from trusted sources. Extensions run with full database + access and can execute arbitrary code. + + ## Notes + + - Extension loading must be enabled first via `enable_extensions/2` + - Extensions are loaded per-connection, not globally + - Some extensions may already be built into libsql (FTS5, JSON1) + - Extension files must match your platform (.so on Linux, .dylib on macOS, .dll on Windows) + + """ + def load_ext(%EctoLibSql.State{conn_id: conn_id} = _state, path, entry_point \\ nil) + when is_binary(path) do + load_extension(conn_id, path, entry_point) + end + + @doc """ + Install an update hook for monitoring database changes (CDC). + + **NOT SUPPORTED** - Update hooks require sending messages from managed BEAM threads, + which is not allowed by Rustler's threading model. + + ## Why Not Supported + + SQLite's update hook callback is called synchronously during INSERT/UPDATE/DELETE operations, + and runs on the same thread executing the SQL statement. In our NIF implementation: + 1. SQL execution happens on Erlang scheduler threads (managed by BEAM) + 2. Rustler's `OwnedEnv::send_and_clear()` can ONLY be called from unmanaged threads + 3. Calling `send_and_clear()` from a managed thread causes a panic + + This is a fundamental limitation of mixing NIF callbacks with Erlang's threading model. + + ## Alternatives + + For change data capture and real-time updates, consider: + + 1. **Application-level events** - Emit events from your Ecto repos: + + defmodule MyApp.Repo do + def insert(changeset, opts \\\\ []) do + case Ecto.Repo.insert(__MODULE__, changeset, opts) do + {:ok, record} = result -> + Phoenix.PubSub.broadcast(MyApp.PubSub, "db_changes", {:insert, record}) + result + error -> error + end + end + end + + 2. **Database triggers** - Use SQLite triggers to log changes to a separate table: + + CREATE TRIGGER users_audit_insert AFTER INSERT ON users + BEGIN + INSERT INTO audit_log (action, table_name, row_id, timestamp) + VALUES ('insert', 'users', NEW.id, datetime('now')); + END; + + 3. **Polling-based CDC** - Periodically query for changes using timestamps or version columns + + 4. **Phoenix.Tracker** - Track state changes at the application level + + ## Returns + - `:unsupported` - Always returns unsupported + + """ + def add_update_hook(%EctoLibSql.State{} = state, _pid \\ self()) do + set_update_hook(state.conn_id, self()) + end + + @doc """ + Remove the update hook from a connection. + + **NOT SUPPORTED** - Update hooks are not currently implemented. + + ## Returns + - `:unsupported` - Always returns unsupported + + """ + def remove_update_hook(%EctoLibSql.State{conn_id: conn_id} = _state) do + clear_update_hook(conn_id) + end + + @doc """ + Install an authorizer hook for row-level security. + + **NOT SUPPORTED** - Authorizer hooks require synchronous bidirectional communication + between Rust and Elixir, which is not feasible with Rustler's threading model. + + ## Why Not Supported + + SQLite's authorizer callback is called synchronously during query compilation and expects + an immediate response (Allow/Deny/Ignore). This would require: + 1. Sending a message from Rust to Elixir + 2. Blocking the Rust thread waiting for a response + 3. Receiving the response from Elixir + + This pattern is not safe with Rustler because: + - The callback runs on a SQLite thread (potentially holding locks) + - Blocking on Erlang scheduler threads can cause deadlocks + - No safe way to do synchronous Rust→Elixir→Rust calls + + ## Alternatives + + For row-level security and access control, consider: + + 1. **Application-level authorization** - Check permissions in Elixir before queries: + + defmodule MyApp.Auth do + def can_access?(user, table, action) do + # Check user permissions + end + end + + def get_user(id, current_user) do + if MyApp.Auth.can_access?(current_user, "users", :read) do + Repo.get(User, id) + else + {:error, :unauthorized} + end + end + + 2. **Database views** - Create views with WHERE clauses for different user levels: + + CREATE VIEW user_visible_posts AS + SELECT * FROM posts WHERE user_id = current_user_id(); + + 3. **Query rewriting** - Modify queries in Elixir to include authorization constraints: + + defmodule MyApp.Repo do + def all(queryable, current_user) do + queryable + |> apply_tenant_filter(current_user) + |> Ecto.Repo.all() + end + end + + 4. **Connection-level restrictions** - Use different database connections with different privileges + + ## Returns + - `:unsupported` - Always returns unsupported + + """ + def add_authorizer(%EctoLibSql.State{conn_id: conn_id} = _state, pid \\ self()) do + set_authorizer(conn_id, pid) + end + @doc """ Execute multiple SQL statements from a semicolon-separated string. @@ -993,6 +1231,53 @@ defmodule EctoLibSql.Native do end end + @doc """ + Get the name of a parameter in a prepared statement by its index. + + Returns the parameter name for named parameters (`:name`, `@name`, `$name`), + or `nil` for positional parameters (`?`). + + ## Parameters + - state: The connection state + - stmt_id: The statement ID returned from `prepare/2` + - idx: Parameter index (1-based, following SQLite convention) + + ## Returns + - `{:ok, name}` - Parameter has a name (e.g., `:id` returns `"id"`) + - `{:ok, nil}` - Parameter is positional (`?`) + - `{:error, reason}` - Error occurred + + ## Example + + # Named parameters + {:ok, stmt_id} = EctoLibSql.Native.prepare(state, "SELECT * FROM users WHERE id = :id AND name = :name") + {:ok, param1} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 1) + # param1 = "id" + {:ok, param2} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 2) + # param2 = "name" + + # Positional parameters + {:ok, stmt_id} = EctoLibSql.Native.prepare(state, "SELECT * FROM users WHERE id = ?") + {:ok, param1} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 1) + # param1 = nil + + ## Notes + - Parameter indices are 1-based (first parameter is index 1) + - Named parameters start with `:`, `@`, or `$` in SQL but the prefix is stripped in the returned name + - Returns `nil` for positional `?` placeholders + + """ + def stmt_parameter_name(%EctoLibSql.State{conn_id: conn_id} = _state, stmt_id, idx) + when is_binary(stmt_id) and is_integer(idx) and idx >= 1 do + # The NIF returns Option which becomes {:ok, "name"} or {:ok, nil} or {:error, reason} + # But Rustler converts Some("name") to just "name", not {:ok, "name"} + case statement_parameter_name(conn_id, stmt_id, idx) do + name when is_binary(name) -> {:ok, name} + nil -> {:ok, nil} + {:error, reason} -> {:error, reason} + end + end + @doc """ Create a savepoint within a transaction. diff --git a/native/ecto_libsql/src/connection.rs b/native/ecto_libsql/src/connection.rs index 2d61d0d6..8f9edcce 100644 --- a/native/ecto_libsql/src/connection.rs +++ b/native/ecto_libsql/src/connection.rs @@ -350,3 +350,101 @@ pub fn interrupt_connection(conn_id: &str) -> NifResult { Err(rustler::Error::Term(Box::new("Invalid connection ID"))) } } + +/// Enable or disable loading of SQLite extensions. +/// +/// By default, extension loading is disabled for security reasons. +/// You must explicitly enable it before calling `load_extension`. +/// +/// # Arguments +/// - `conn_id`: Database connection ID +/// - `enabled`: Whether to enable (true) or disable (false) extension loading +/// +/// # Returns +/// - `:ok` - Extension loading enabled/disabled successfully +/// - `{:error, reason}` - Operation failed +/// +/// # Security Warning +/// Only enable extension loading if you trust the extensions being loaded. +/// Malicious extensions can compromise database security. +#[rustler::nif(schedule = "DirtyIo")] +pub fn enable_load_extension(conn_id: &str, enabled: bool) -> NifResult { + let conn_map = crate::utils::safe_lock(&CONNECTION_REGISTRY, "enable_load_extension conn_map")?; + + if let Some(client) = conn_map.get(conn_id) { + let client = client.clone(); + drop(conn_map); // Release lock before operation + + let client_guard = safe_lock_arc(&client, "enable_load_extension client")?; + let conn_guard: std::sync::MutexGuard = + safe_lock_arc(&client_guard.client, "enable_load_extension conn")?; + + if enabled { + conn_guard.load_extension_enable().map_err(|e| { + rustler::Error::Term(Box::new(format!( + "Failed to enable extension loading: {}", + e + ))) + })?; + } else { + conn_guard.load_extension_disable().map_err(|e| { + rustler::Error::Term(Box::new(format!( + "Failed to disable extension loading: {}", + e + ))) + })?; + } + + Ok(rustler::types::atom::ok()) + } else { + Err(rustler::Error::Term(Box::new("Invalid connection ID"))) + } +} + +/// Load a SQLite extension from a dynamic library file. +/// +/// Extensions must be enabled first via `enable_load_extension(conn_id, true)`. +/// +/// # Arguments +/// - `conn_id`: Database connection ID +/// - `path`: Path to the extension dynamic library (.so, .dylib, or .dll) +/// - `entry_point`: Optional entry point function name (defaults to extension-specific default) +/// +/// # Returns +/// - `:ok` - Extension loaded successfully +/// - `{:error, reason}` - Extension loading failed +/// +/// # Security Warning +/// Only load extensions from trusted sources. Extensions run with full database +/// access and can execute arbitrary code. +/// +/// # Common Extensions +/// - FTS5 (full-text search) - usually built-in, but can be loaded separately +/// - JSON1 (JSON functions) - usually built-in +/// - R-Tree (spatial indexing) +/// - Custom user-defined functions +#[rustler::nif(schedule = "DirtyIo")] +pub fn load_extension(conn_id: &str, path: &str, entry_point: Option<&str>) -> NifResult { + let conn_map = crate::utils::safe_lock(&CONNECTION_REGISTRY, "load_extension conn_map")?; + + if let Some(client) = conn_map.get(conn_id) { + let client = client.clone(); + drop(conn_map); // Release lock before operation + + let path_buf = std::path::PathBuf::from(path); + + let client_guard = safe_lock_arc(&client, "load_extension client")?; + let conn_guard: std::sync::MutexGuard = + safe_lock_arc(&client_guard.client, "load_extension conn")?; + + conn_guard + .load_extension(&path_buf, entry_point) + .map_err(|e| { + rustler::Error::Term(Box::new(format!("Failed to load extension: {}", e))) + })?; + + Ok(rustler::types::atom::ok()) + } else { + Err(rustler::Error::Term(Box::new("Invalid connection ID"))) + } +} diff --git a/native/ecto_libsql/src/constants.rs b/native/ecto_libsql/src/constants.rs index d11e0692..adbfd444 100644 --- a/native/ecto_libsql/src/constants.rs +++ b/native/ecto_libsql/src/constants.rs @@ -75,5 +75,6 @@ atoms! { transaction, connection, blob, - nil + nil, + unsupported } diff --git a/native/ecto_libsql/src/hooks.rs b/native/ecto_libsql/src/hooks.rs new file mode 100644 index 00000000..afa357d4 --- /dev/null +++ b/native/ecto_libsql/src/hooks.rs @@ -0,0 +1,153 @@ +/// Hook management for EctoLibSql +/// +/// This module implements database hooks for monitoring and controlling database operations. +/// Hooks allow Elixir processes to receive notifications about database changes and control access. +/// +/// **CURRENT STATUS**: Both update hooks and authorizer hooks are currently **NOT SUPPORTED** +/// due to fundamental threading limitations with Rustler and the BEAM VM. +use rustler::{Atom, LocalPid, NifResult}; + +/// Set update hook for a connection +/// +/// **NOT SUPPORTED** - Update hooks require sending messages from managed BEAM threads, +/// which is not allowed by Rustler's threading model. +/// +/// # Why Not Supported +/// +/// SQLite's update hook callback is called synchronously during INSERT/UPDATE/DELETE operations, +/// and runs on the same thread executing the SQL statement. In our NIF implementation: +/// 1. SQL execution happens on Erlang scheduler threads (managed by BEAM) +/// 2. Rustler's `OwnedEnv::send_and_clear()` can ONLY be called from unmanaged threads +/// 3. Calling `send_and_clear()` from a managed thread causes a panic: "current thread is managed" +/// +/// This is a fundamental limitation of mixing NIF callbacks with Erlang's threading model. +/// +/// # Alternatives +/// +/// For change data capture and real-time updates, consider: +/// +/// 1. **Application-level events** - Emit events from your Ecto repos: +/// +/// ```elixir +/// defmodule MyApp.Repo do +/// def insert(changeset, opts \\ []) do +/// case Ecto.Repo.insert(__MODULE__, changeset, opts) do +/// {:ok, record} = result -> +/// Phoenix.PubSub.broadcast(MyApp.PubSub, "db_changes", {:insert, record}) +/// result +/// error -> error +/// end +/// end +/// end +/// ``` +/// +/// 2. **Database triggers** - Use SQLite triggers to log changes to a separate table: +/// +/// ```sql +/// CREATE TRIGGER users_audit_insert AFTER INSERT ON users +/// BEGIN +/// INSERT INTO audit_log (action, table_name, row_id, timestamp) +/// VALUES ('insert', 'users', NEW.id, datetime('now')); +/// END; +/// ``` +/// +/// 3. **Polling-based CDC** - Periodically query for changes using timestamps or version columns +/// +/// 4. **Phoenix.Tracker** - Track state changes at the application level +/// +/// # Arguments +/// - `_conn_id` - Connection identifier (ignored) +/// - `_pid` - PID for callbacks (ignored) +/// +/// # Returns +/// - `{:error, :unsupported}` - Always returns unsupported +#[rustler::nif] +pub fn set_update_hook(_conn_id: &str, _pid: LocalPid) -> NifResult { + Err(rustler::Error::Atom("unsupported")) +} + +/// Clear update hook for a connection +/// +/// **NOT SUPPORTED** - Update hooks are not currently implemented. +/// +/// # Arguments +/// - `_conn_id` - Connection identifier (ignored) +/// +/// # Returns +/// - `{:error, :unsupported}` - Always returns unsupported +#[rustler::nif] +pub fn clear_update_hook(_conn_id: &str) -> NifResult { + Err(rustler::Error::Atom("unsupported")) +} + +/// Set authorizer hook for a connection +/// +/// **NOT SUPPORTED** - Authorizer hooks require synchronous bidirectional communication +/// between Rust and Elixir, which is not feasible with Rustler's threading model. +/// +/// # Why Not Supported +/// +/// SQLite's authorizer callback is called synchronously during query compilation and expects +/// an immediate response (Allow/Deny/Ignore). This would require: +/// 1. Sending a message from Rust to Elixir +/// 2. Blocking the Rust thread waiting for a response +/// 3. Receiving the response from Elixir +/// +/// This pattern is not safe with Rustler because: +/// - The callback runs on a SQLite thread (potentially holding locks) +/// - Blocking on Erlang scheduler threads can cause deadlocks +/// - No safe way to do synchronous Rust→Elixir→Rust calls +/// +/// # Alternatives +/// +/// For row-level security and access control, consider: +/// +/// 1. **Application-level authorization** - Check permissions in Elixir before queries: +/// +/// ```elixir +/// defmodule MyApp.Auth do +/// def can_access?(user, table, action) do +/// # Check user permissions +/// end +/// end +/// +/// def get_user(id, current_user) do +/// if MyApp.Auth.can_access?(current_user, "users", :read) do +/// Repo.get(User, id) +/// else +/// {:error, :unauthorized} +/// end +/// end +/// ``` +/// +/// 2. **Database views** - Create views with WHERE clauses for different user levels: +/// +/// ```sql +/// CREATE VIEW user_visible_posts AS +/// SELECT * FROM posts WHERE user_id = current_user_id(); +/// ``` +/// +/// 3. **Query rewriting** - Modify queries in Elixir to include authorization constraints: +/// +/// ```elixir +/// defmodule MyApp.Repo do +/// def all(queryable, current_user) do +/// queryable +/// |> apply_tenant_filter(current_user) +/// |> Ecto.Repo.all() +/// end +/// end +/// ``` +/// +/// 4. **Connection-level restrictions** - Use different database connections with different privileges +/// +/// # Arguments +/// - `_conn_id` - Connection identifier (ignored) +/// - `_pid` - PID for callbacks (ignored) +/// +/// # Returns +/// - `{:error, :unsupported}` - Always returns unsupported +#[rustler::nif] +pub fn set_authorizer(_conn_id: &str, _pid: LocalPid) -> NifResult { + Err(rustler::Error::Atom("unsupported")) +} diff --git a/native/ecto_libsql/src/lib.rs b/native/ecto_libsql/src/lib.rs index 9955f048..16167362 100644 --- a/native/ecto_libsql/src/lib.rs +++ b/native/ecto_libsql/src/lib.rs @@ -7,6 +7,7 @@ pub mod connection; pub mod constants; pub mod cursor; pub mod decode; +pub mod hooks; pub mod metadata; pub mod models; pub mod query; diff --git a/native/ecto_libsql/src/statement.rs b/native/ecto_libsql/src/statement.rs index 5bc1777a..da5c3e0c 100644 --- a/native/ecto_libsql/src/statement.rs +++ b/native/ecto_libsql/src/statement.rs @@ -325,6 +325,59 @@ pub fn statement_parameter_count(conn_id: &str, stmt_id: &str) -> NifResult NifResult> { + let conn_map = utils::safe_lock(&CONNECTION_REGISTRY, "statement_parameter_name conn_map")?; + let stmt_registry = utils::safe_lock(&STMT_REGISTRY, "statement_parameter_name stmt_registry")?; + + if conn_map.get(conn_id).is_none() { + return Err(rustler::Error::Term(Box::new("Invalid connection ID"))); + } + + let (stored_conn_id, cached_stmt) = stmt_registry + .get(stmt_id) + .ok_or_else(|| rustler::Error::Term(Box::new("Statement not found")))?; + + // Verify statement belongs to this connection + decode::verify_statement_ownership(stored_conn_id, conn_id)?; + + let cached_stmt = cached_stmt.clone(); + + drop(stmt_registry); + drop(conn_map); + + let stmt_guard = utils::safe_lock_arc(&cached_stmt, "statement_parameter_name stmt")?; + + // SQLite uses 1-based parameter indices + let param_name = stmt_guard.parameter_name(idx).map(|s| s.to_string()); + + Ok(param_name) +} + /// Reset a prepared statement to its initial state for reuse. /// /// After executing a statement, you should reset it before binding new parameters diff --git a/test/hooks_test.exs b/test/hooks_test.exs new file mode 100644 index 00000000..20ef3338 --- /dev/null +++ b/test/hooks_test.exs @@ -0,0 +1,92 @@ +defmodule EctoLibSql.HooksTest do + use ExUnit.Case, async: true + + alias EctoLibSql.Native + + setup do + {:ok, state} = EctoLibSql.connect(database: ":memory:") + + on_exit(fn -> EctoLibSql.disconnect([], state) end) + + {:ok, state: state} + end + + describe "add_update_hook/2 - NOT SUPPORTED" do + test "returns :unsupported error", %{state: state} do + assert :unsupported = Native.add_update_hook(state) + end + + test "returns :unsupported with custom PID", %{state: state} do + test_pid = self() + assert :unsupported = Native.add_update_hook(state, test_pid) + end + + test "does not affect database operations", %{state: state} do + :unsupported = Native.add_update_hook(state) + + # Database operations should still work + {:ok, _, _, _state} = + EctoLibSql.handle_execute( + "CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)", + [], + [], + state + ) + + {:ok, _, _, _state} = + EctoLibSql.handle_execute( + "INSERT INTO users (name) VALUES ('Alice')", + [], + [], + state + ) + + # No errors, no hook messages + end + end + + describe "remove_update_hook/1 - NOT SUPPORTED" do + test "returns :unsupported error", %{state: state} do + assert :unsupported = Native.remove_update_hook(state) + end + + test "can be called multiple times safely", %{state: state} do + assert :unsupported = Native.remove_update_hook(state) + assert :unsupported = Native.remove_update_hook(state) + end + end + + describe "add_authorizer/2 - NOT SUPPORTED" do + test "returns :unsupported error", %{state: state} do + assert :unsupported = Native.add_authorizer(state) + end + + test "returns :unsupported with custom PID", %{state: state} do + test_pid = self() + assert :unsupported = Native.add_authorizer(state, test_pid) + end + + test "does not affect database operations", %{state: state} do + :unsupported = Native.add_authorizer(state) + + # Database operations should still work + {:ok, _, _, _state} = + EctoLibSql.handle_execute( + "CREATE TABLE posts (id INTEGER PRIMARY KEY, title TEXT)", + [], + [], + state + ) + + {:ok, _, _, _state} = + EctoLibSql.handle_execute( + "INSERT INTO posts (title) VALUES ('Test Post')", + [], + [], + state + ) + + # No errors + end + end +end diff --git a/test/statement_features_test.exs b/test/statement_features_test.exs index 6cedd131..0128c3a5 100644 --- a/test/statement_features_test.exs +++ b/test/statement_features_test.exs @@ -396,6 +396,294 @@ defmodule EctoLibSql.StatementFeaturesTest do EctoLibSql.Native.close_stmt(stmt_id) end + + test "parameter_count returns 0 for statements with no parameters", %{state: state} do + {:ok, stmt_id} = EctoLibSql.Native.prepare(state, "SELECT * FROM users") + + assert {:ok, 0} = EctoLibSql.Native.stmt_parameter_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_count handles many parameters", %{state: state} do + # Create INSERT statement with 20 parameters + placeholders = Enum.map(1..20, fn _ -> "?" end) |> Enum.join(", ") + columns = Enum.map(1..20, fn i -> "col#{i}" end) |> Enum.join(", ") + + # Create table with 20 columns + create_sql = + "CREATE TABLE many_cols (#{Enum.map(1..20, fn i -> "col#{i} TEXT" end) |> Enum.join(", ")})" + + {:ok, _, _, state} = EctoLibSql.handle_execute(create_sql, [], [], state) + + # Prepare INSERT with 20 parameters + insert_sql = "INSERT INTO many_cols (#{columns}) VALUES (#{placeholders})" + {:ok, stmt_id} = EctoLibSql.Native.prepare(state, insert_sql) + + assert {:ok, 20} = EctoLibSql.Native.stmt_parameter_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_count for UPDATE statements", %{state: state} do + {:ok, stmt_id} = + EctoLibSql.Native.prepare(state, "UPDATE users SET name = ?, age = ? WHERE id = ?") + + assert {:ok, 3} = EctoLibSql.Native.stmt_parameter_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_count for complex nested queries", %{state: state} do + # Create posts table for JOIN query + {:ok, _, _, state} = + EctoLibSql.handle_execute( + "CREATE TABLE posts (id INTEGER PRIMARY KEY, user_id INTEGER, title TEXT)", + [], + [], + state + ) + + # Complex query with multiple parameters in different parts + complex_sql = """ + SELECT u.name, COUNT(p.id) as post_count + FROM users u + LEFT JOIN posts p ON u.id = p.user_id + WHERE u.age > ? AND u.name LIKE ? + GROUP BY u.id + HAVING COUNT(p.id) >= ? + """ + + {:ok, stmt_id} = EctoLibSql.Native.prepare(state, complex_sql) + + assert {:ok, 3} = EctoLibSql.Native.stmt_parameter_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_name introspection for named parameters", %{state: state} do + # Test with colon-style named parameters (:name) + {:ok, stmt_id} = + EctoLibSql.Native.prepare( + state, + "INSERT INTO users (id, name, age) VALUES (:id, :name, :age)" + ) + + # Get parameter names (note: SQLite uses 1-based indexing) + {:ok, param1} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 1) + assert param1 == ":id" + + {:ok, param2} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 2) + assert param2 == ":name" + + {:ok, param3} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 3) + assert param3 == ":age" + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_name returns nil for positional parameters", %{state: state} do + {:ok, stmt_id} = + EctoLibSql.Native.prepare(state, "SELECT * FROM users WHERE name = ? AND age = ?") + + # Positional parameters should return nil + {:ok, param1} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 1) + assert param1 == nil + + {:ok, param2} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 2) + assert param2 == nil + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_name supports dollar-style parameters", %{state: state} do + # Test with dollar-style named parameters ($name) + {:ok, stmt_id} = + EctoLibSql.Native.prepare(state, "SELECT * FROM users WHERE id = $id AND name = $name") + + {:ok, param1} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 1) + assert param1 == "$id" + + {:ok, param2} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 2) + assert param2 == "$name" + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_name supports at-style parameters", %{state: state} do + # Test with at-style named parameters (@name) + {:ok, stmt_id} = + EctoLibSql.Native.prepare(state, "SELECT * FROM users WHERE id = @id AND name = @name") + + {:ok, param1} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 1) + assert param1 == "@id" + + {:ok, param2} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 2) + assert param2 == "@name" + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "parameter_name handles mixed positional and named parameters", %{state: state} do + # SQLite allows mixing positional and named parameters + {:ok, stmt_id} = + EctoLibSql.Native.prepare(state, "SELECT * FROM users WHERE id = :id AND age > ?") + + {:ok, param1} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 1) + assert param1 == ":id" + + {:ok, param2} = EctoLibSql.Native.stmt_parameter_name(state, stmt_id, 2) + assert param2 == nil + + EctoLibSql.Native.close_stmt(stmt_id) + end + end + + # ============================================================================ + # Column introspection edge cases + # ============================================================================ + + describe "Column introspection edge cases ✅" do + test "column count for SELECT *", %{state: state} do + {:ok, stmt_id} = EctoLibSql.Native.prepare(state, "SELECT * FROM users") + + # Should return 3 columns (id, name, age) + assert {:ok, 3} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "column count for INSERT without RETURNING", %{state: state} do + {:ok, stmt_id} = + EctoLibSql.Native.prepare(state, "INSERT INTO users VALUES (?, ?, ?)") + + # INSERT without RETURNING should return 0 columns + assert {:ok, 0} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "column count for UPDATE without RETURNING", %{state: state} do + {:ok, stmt_id} = + EctoLibSql.Native.prepare(state, "UPDATE users SET name = ? WHERE id = ?") + + # UPDATE without RETURNING should return 0 columns + assert {:ok, 0} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "column count for DELETE without RETURNING", %{state: state} do + {:ok, stmt_id} = EctoLibSql.Native.prepare(state, "DELETE FROM users WHERE id = ?") + + # DELETE without RETURNING should return 0 columns + assert {:ok, 0} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "column metadata for aggregate functions", %{state: state} do + {:ok, stmt_id} = + EctoLibSql.Native.prepare( + state, + """ + SELECT + COUNT(*) as total, + AVG(age) as avg_age, + MIN(age) as min_age, + MAX(age) as max_age, + SUM(age) as sum_age + FROM users + """ + ) + + assert {:ok, 5} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + # Check column names + names = get_column_names(state, stmt_id, 5) + assert names == ["total", "avg_age", "min_age", "max_age", "sum_age"] + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "column metadata for JOIN with multiple tables", %{state: state} do + # Create posts table + {:ok, _, _, state} = + EctoLibSql.handle_execute( + "CREATE TABLE posts (id INTEGER PRIMARY KEY, user_id INTEGER, title TEXT, content TEXT)", + [], + [], + state + ) + + # Complex JOIN query + {:ok, stmt_id} = + EctoLibSql.Native.prepare( + state, + """ + SELECT + u.id, + u.name, + u.age, + p.id as post_id, + p.title, + p.content + FROM users u + INNER JOIN posts p ON u.id = p.user_id + """ + ) + + assert {:ok, 6} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + names = get_column_names(state, stmt_id, 6) + assert names == ["id", "name", "age", "post_id", "title", "content"] + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "column metadata for subqueries", %{state: state} do + {:ok, stmt_id} = + EctoLibSql.Native.prepare( + state, + """ + SELECT + name, + (SELECT COUNT(*) FROM users) as total_users + FROM users + WHERE id = ? + """ + ) + + assert {:ok, 2} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + names = get_column_names(state, stmt_id, 2) + assert names == ["name", "total_users"] + + EctoLibSql.Native.close_stmt(stmt_id) + end + + test "column metadata for computed expressions", %{state: state} do + {:ok, stmt_id} = + EctoLibSql.Native.prepare( + state, + """ + SELECT + id, + name, + age * 2 as double_age, + UPPER(name) as upper_name, + age + 10 as age_plus_ten + FROM users + """ + ) + + assert {:ok, 5} = EctoLibSql.Native.stmt_column_count(state, stmt_id) + + names = get_column_names(state, stmt_id, 5) + assert names == ["id", "name", "double_age", "upper_name", "age_plus_ten"] + + EctoLibSql.Native.close_stmt(stmt_id) + end end # ============================================================================ From 3a86263f1efa24e35d9f1846870d21c4f25e660b Mon Sep 17 00:00:00 2001 From: Drew Robinson Date: Fri, 26 Dec 2025 14:15:00 +1100 Subject: [PATCH 2/2] tests: Fix issues with hook tests --- ENHANCEMENTS.md | 16 ++++++++-------- lib/ecto_libsql/native.ex | 4 ++-- native/ecto_libsql/src/hooks.rs | 23 ++++++++++++++++------- test/hooks_test.exs | 18 +++++++++--------- 4 files changed, 35 insertions(+), 26 deletions(-) diff --git a/ENHANCEMENTS.md b/ENHANCEMENTS.md index eb75c86f..52708231 100644 --- a/ENHANCEMENTS.md +++ b/ENHANCEMENTS.md @@ -355,7 +355,7 @@ config :my_app, MyApp.Repo, **Reason**: Rustler threading model incompatibility **Why Not Supported**: -Both update hooks and authorizer hooks are fundamentally incompatible with Rustler's threading model: +Both update hooks and authoriser hooks are fundamentally incompatible with Rustler's threading model: 1. **Update Hooks Problem**: - SQLite's update hook callback runs synchronously during INSERT/UPDATE/DELETE operations @@ -363,8 +363,8 @@ Both update hooks and authorizer hooks are fundamentally incompatible with Rustl - Rustler's `OwnedEnv::send_and_clear()` can ONLY be called from unmanaged threads - Calling `send_and_clear()` from managed thread causes panic: "current thread is managed" -2. **Authorizer Hooks Problem**: - - SQLite's authorizer callback is synchronous and expects immediate response (Allow/Deny/Ignore) +2. **Authoriser Hooks Problem**: + - SQLite's authoriser callback is synchronous and expects immediate response (Allow/Deny/Ignore) - Would require blocking Rust thread waiting for Elixir response - No safe way to do synchronous Rust→Elixir→Rust calls - Blocking on scheduler threads can cause deadlocks @@ -377,8 +377,8 @@ For **Change Data Capture / Real-time Updates**: - Polling-based CDC with timestamps - Phoenix.Tracker for state tracking -For **Row-Level Security / Authorization**: -- Application-level authorization checks before queries +For **Row-Level Security / Authorisation**: +- Application-level authorisation checks before queries - Database views with WHERE clauses - Query rewriting in Ecto - Connection-level privileges @@ -445,13 +445,13 @@ For **Row-Level Security / Authorization**: ### Potential Enhancements 1. **Haversine Formula**: Implement actual geographic distance calculations for accuracy 2. **Higher Dimensions**: Support for more complex geospatial data (elevation, time, etc.) -3. **Index Optimization**: Add spatial indexes for performance on large datasets +3. **Index Optimisation**: Add spatial indexes for performance on large datasets 4. **Batch Queries**: Use batch operations for multiple location lookups 5. **Clustering**: Find geographic clusters of locations using vector analysis ### Real-World Applications - **Location-based services**: Find nearby restaurants, hotels, gas stations -- **Delivery optimization**: Locate nearest warehouse to customer location +- **Delivery optimisation**: Locate nearest warehouse to customer location - **Regional analytics**: Find closest office/branch in each region - **Social discovery**: Find nearby users, events, or meetup groups - **Asset tracking**: Locate nearest available equipment or resources @@ -885,4 +885,4 @@ disallowed-methods = [ 8. **Ecto Integration Improvements** 9. **P3 Low Priority Features** -This prioritization ensures we address the most critical production issues first, then focus on performance and reliability, before moving to nice-to-have features and advanced functionality. \ No newline at end of file +This prioritisation ensures we address the most critical production issues first, then focus on performance and reliability, before moving to nice-to-have features and advanced functionality. diff --git a/lib/ecto_libsql/native.ex b/lib/ecto_libsql/native.ex index 79fbabe1..4e0b2b2d 100644 --- a/lib/ecto_libsql/native.ex +++ b/lib/ecto_libsql/native.ex @@ -992,8 +992,8 @@ defmodule EctoLibSql.Native do - `:unsupported` - Always returns unsupported """ - def add_update_hook(%EctoLibSql.State{} = state, _pid \\ self()) do - set_update_hook(state.conn_id, self()) + def add_update_hook(%EctoLibSql.State{} = state, pid \\ self()) do + set_update_hook(state.conn_id, pid) end @doc """ diff --git a/native/ecto_libsql/src/hooks.rs b/native/ecto_libsql/src/hooks.rs index afa357d4..017f7058 100644 --- a/native/ecto_libsql/src/hooks.rs +++ b/native/ecto_libsql/src/hooks.rs @@ -5,7 +5,7 @@ /// /// **CURRENT STATUS**: Both update hooks and authorizer hooks are currently **NOT SUPPORTED** /// due to fundamental threading limitations with Rustler and the BEAM VM. -use rustler::{Atom, LocalPid, NifResult}; +use rustler::{Atom, Env, LocalPid, NifResult}; /// Set update hook for a connection /// @@ -62,8 +62,11 @@ use rustler::{Atom, LocalPid, NifResult}; /// # Returns /// - `{:error, :unsupported}` - Always returns unsupported #[rustler::nif] -pub fn set_update_hook(_conn_id: &str, _pid: LocalPid) -> NifResult { - Err(rustler::Error::Atom("unsupported")) +pub fn set_update_hook(env: Env, _conn_id: &str, _pid: LocalPid) -> NifResult<(Atom, Atom)> { + Ok(( + Atom::from_str(env, "error")?, + Atom::from_str(env, "unsupported")?, + )) } /// Clear update hook for a connection @@ -76,8 +79,11 @@ pub fn set_update_hook(_conn_id: &str, _pid: LocalPid) -> NifResult { /// # Returns /// - `{:error, :unsupported}` - Always returns unsupported #[rustler::nif] -pub fn clear_update_hook(_conn_id: &str) -> NifResult { - Err(rustler::Error::Atom("unsupported")) +pub fn clear_update_hook(env: Env, _conn_id: &str) -> NifResult<(Atom, Atom)> { + Ok(( + Atom::from_str(env, "error")?, + Atom::from_str(env, "unsupported")?, + )) } /// Set authorizer hook for a connection @@ -148,6 +154,9 @@ pub fn clear_update_hook(_conn_id: &str) -> NifResult { /// # Returns /// - `{:error, :unsupported}` - Always returns unsupported #[rustler::nif] -pub fn set_authorizer(_conn_id: &str, _pid: LocalPid) -> NifResult { - Err(rustler::Error::Atom("unsupported")) +pub fn set_authorizer(env: Env, _conn_id: &str, _pid: LocalPid) -> NifResult<(Atom, Atom)> { + Ok(( + Atom::from_str(env, "error")?, + Atom::from_str(env, "unsupported")?, + )) } diff --git a/test/hooks_test.exs b/test/hooks_test.exs index 20ef3338..cd237243 100644 --- a/test/hooks_test.exs +++ b/test/hooks_test.exs @@ -13,16 +13,16 @@ defmodule EctoLibSql.HooksTest do describe "add_update_hook/2 - NOT SUPPORTED" do test "returns :unsupported error", %{state: state} do - assert :unsupported = Native.add_update_hook(state) + assert {:error, :unsupported} = Native.add_update_hook(state) end test "returns :unsupported with custom PID", %{state: state} do test_pid = self() - assert :unsupported = Native.add_update_hook(state, test_pid) + assert {:error, :unsupported} = Native.add_update_hook(state, test_pid) end test "does not affect database operations", %{state: state} do - :unsupported = Native.add_update_hook(state) + {:error, :unsupported} = Native.add_update_hook(state) # Database operations should still work {:ok, _, _, _state} = @@ -47,27 +47,27 @@ defmodule EctoLibSql.HooksTest do describe "remove_update_hook/1 - NOT SUPPORTED" do test "returns :unsupported error", %{state: state} do - assert :unsupported = Native.remove_update_hook(state) + assert {:error, :unsupported} = Native.remove_update_hook(state) end test "can be called multiple times safely", %{state: state} do - assert :unsupported = Native.remove_update_hook(state) - assert :unsupported = Native.remove_update_hook(state) + assert {:error, :unsupported} = Native.remove_update_hook(state) + assert {:error, :unsupported} = Native.remove_update_hook(state) end end describe "add_authorizer/2 - NOT SUPPORTED" do test "returns :unsupported error", %{state: state} do - assert :unsupported = Native.add_authorizer(state) + assert {:error, :unsupported} = Native.add_authorizer(state) end test "returns :unsupported with custom PID", %{state: state} do test_pid = self() - assert :unsupported = Native.add_authorizer(state, test_pid) + assert {:error, :unsupported} = Native.add_authorizer(state, test_pid) end test "does not affect database operations", %{state: state} do - :unsupported = Native.add_authorizer(state) + {:error, :unsupported} = Native.add_authorizer(state) # Database operations should still work {:ok, _, _, _state} =