Translate parameterized collection Contains to a native array parameter (#39)

[alex-clickhouse]

Closes #39.

What

A Contains over a captured collection (ids.Contains(x.Id)) now binds the whole collection as a single native Array(T) parameter — has({ids:Array(Int64)}, column) — instead of EF Core's default one-scalar-parameter-per-element IN (p1, …, pN) expansion.

This removes the parameter-count / query-size ceilings that large IN lists hit, and keeps the query text (and plan-cache key) independent of the collection size. ClickHouse is OLAP and doesn't reuse plans by parameterization, so there's no downside to a bound array vs. inlined constants.

WHERE `x`.`Id` IN ({ids1:Int64}, {ids2:Int64}, … {idsN:Int64})   -- before
WHERE has({ids:Array(Int64)}, `x`.`Id`)                          -- after (1 parameter)

How

Implemented as an override of SqlNullabilityProcessor.VisitIn in ClickHouseSqlNullabilityProcessor. Every route (predicate Contains, Any(id => id == x)) converges on an InExpression carrying a collection ValuesParameter; that InExpression is rewritten to has(arrayParam, item) with the array element store type aligned to the tested column's mapping.

The rewrite is applied for an unmarked collection (the provider default) or an explicit EF.Parameter(...). It deliberately falls back to the standard per-element expansion (IN (…)) for:

• EF.MultipleParameters(...) (keeps one-parameter-per-element) and EF.Constant(...) (inlines literals) — the standard per-query knobs;
• a nullable tested column — has(arr, NULL) returns a concrete 0, so NOT-Contains would treat NULL rows differently from x NOT IN (…); deferring to the base path keeps null semantics unchanged;
• element types that need a value conversion (e.g. a CLR enum → Enum8) — the driver can't serialize the un-converted collection;
• element CLR types the driver doesn't serialize natively inside an array — temporal types (DateTime, DateTimeOffset, DateOnly, TimeOnly, TimeSpan); the array-parameter path covers integers, floating point, decimal, bool, string, and Guid;
• collections used as a queryable source (joins, Where(…).Contains(…)) — these keep the existing SELECT … UNION ALL … rewrite. The model-wide UseParameterizedCollectionMode knob is intentionally not consulted, since it also governs that path (which doesn't support ParameterTranslationMode.Parameter).

Design notes / limitations

• The collection is bound as a single parameter value, so extremely large lists (hundreds of thousands of elements) can hit the server's http_max_field_value_size (default 128 KiB). This is far better than the parameter-count ceiling and is server-configurable.
• EF.Parameter(ids) used with a queryable operator (join, .Where().Contains()) still fails to translate — a pre-existing gap that predates this change (it requires overriding TranslatePrimitiveCollection with an arrayJoin-style table translation; out of scope here).
• A future enhancement could implement TranslatePrimitiveCollection (via arrayJoin) to make Parameter mode a first-class native-array citizen everywhere (joins included), which would let the provider default ParameterizedCollectionMode to Parameter.

Testing

• New ParameterizedContainsTests (17 tests): Int32/Int64/String/Guid/List round-trips, element-store-type alignment, empty collection, large (12k) single-parameter, EF.Constant/EF.MultipleParameters/inline behavior, join-still-uses-UNION-ALL, negated Contains, and the decimal/bool/DateTime/enum/nullable-column fallback matrix.
• Two existing tests (misnamed "inline"/"local" but actually captured collections) updated to assert the new array-parameter behavior.
• Full unit/integration suite (644) and functional/Northwind suite (322) green.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...uery/Internal/ClickHouseSqlNullabilityProcessor.cs	96.77%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

This PR enhances the ClickHouse EF Core provider’s translation of Contains over captured/parameterized primitive collections by binding the collection as a single native Array(T) parameter and rewriting the predicate to has(arrayParam, column), avoiding large IN (p1, …, pN) expansions and parameter-count/query-size limits. It also bumps ClickHouse.Driver to 1.3.0 and updates docs/tests accordingly.

Changes:

• Rewrite InExpression with a collection ValuesParameter into has({p:Array(T)}, item) in ClickHouseSqlNullabilityProcessor, with guarded fallbacks for cases where semantics/serialization could differ.
• Add a new integration test suite (ParameterizedContainsTests) and update existing tests to assert the new SQL shape.
• Update documentation (README/CHANGELOG/RELEASENOTES) and bump ClickHouse.Driver to 1.3.0 across provider and test projects.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/EFCore.ClickHouse.Tests/ParameterizedContainsTests.cs	New integration coverage for array-parameter Contains translation across types and fallback cases.
test/EFCore.ClickHouse.Tests/ExtendedTypeMappingTests.cs	Updates an existing test to assert the new captured-collection array-parameter behavior.
test/EFCore.ClickHouse.Tests/EFCore.ClickHouse.Tests.csproj	Bumps ClickHouse.Driver dependency to 1.3.0 for the test project.
test/EFCore.ClickHouse.Tests/ClickHouseQuerySqlGeneratorTests.cs	Updates SQL-shape assertion to expect array-parameter Contains translation for captured collections.
test/EFCore.ClickHouse.FunctionalTests/EFCore.ClickHouse.FunctionalTests.csproj	Bumps ClickHouse.Driver dependency to 1.3.0 for functional tests.
src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlNullabilityProcessor.cs	Implements the IN {collectionParameter} → has(arrayParam, item) rewrite with type-mapping alignment and guarded fallbacks.
src/EFCore.ClickHouse/EFCore.ClickHouse.csproj	Bumps provider dependency on ClickHouse.Driver to 1.3.0.
RELEASENOTES.md	Documents the new Contains behavior and driver bump for the next release.
README.md	Adds user-facing documentation for the new captured-collection Contains translation and opt-outs.
CHANGELOG.md	Records the feature and dependency bump with detailed behavior notes and limitations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[DanielBunting]

I'd probably look to bolt onto IEnumerable<T> rather than T[] as it currently breaks paths that worked prior.

You could achieve this by extending ClickHouseArrayTypeMapping specifically for array parameters (dropping null values and rolling out any lazy enumerables). I've created the changes that may get you a little closer here: https://github.com/DanielBunting/ClickHouse.EntityFrameworkCore/tree/pr-49-review

Note: I think your driver bump may have fixed the HashSet/Lazy cases, but it would still be worth checking with null values in the collection as this could be a silent breakage - an int?[] [1, null, 3] will resolve to [1, 0, 3] and this feels off.

[DanielBunting]

[Query] Is it worth adding support for HashSet/IEnumerable here - quite a common use case is to use the HashSet to remove dupes? Right now it throws with this test:

  [Fact]
  public async Task HashSet_Collection_Matches()
  {
      await using var ctx = new ParamContainsDbContext(_fixture.ConnectionString);
      var ids = new HashSet<long> { 1L, 3L };

      // Driver 1.1.0 cannot serialize HashSet<T> as an array parameter value:
      // ArgumentOutOfRangeException "Unknown type: System.Collections.Generic.HashSet`1[System.Int64]".
      var query = ctx.Entities.Where(e => ids.Contains(e.Id));

      var rows = await query.OrderBy(e => e.Id).Select(e => e.Id).ToListAsync();
      Assert.Equal([1L, 3L], rows);
  }

[DanielBunting]

Lazy IEnumerable also throws here, but again - down to if IEnumerable is supported.

[DanielBunting]

[Query] Nulls inside the collection aren't guarded, and can't be in the processor. Should we strip the nulls at binding time?