Translate parameterized collection Contains to a native array parameter (#39)

CHANGELOG.md

@@ -1,5 +1,11 @@
-v0.3.1 (Unreleased)
+v0.4.0 (Unreleased)
 ---
+### Advanced queries
+* **Parameterized collection `Contains` binds a single native `Array(T)` parameter.** A captured collection used with `Contains` (`ids.Contains(x.Id)`) now translates to `has({ids:Array(T)}, column)` — one bound array parameter — instead of EF Core's default one-scalar-parameter-per-element `IN (…)` expansion. This removes the parameter-count / query-size ceilings that large `IN` lists hit and keeps the query text (and plan-cache key) independent of the collection size. The array's element store type is aligned to the tested column (`Array(Int64)`, `Array(String)`, `Array(UUID)`, …). Per-query overrides are honored via the standard markers: `EF.MultipleParameters(…)` keeps the one-parameter-per-element expansion and `EF.Constant(…)` inlines the values as literals. Verified natively-serialized element types are the integers, floating point, `decimal`, `bool`, `string`, `Guid`, `DateTime`/`DateTime64`, `Date`/`Date32` (`DateOnly`), the big integers `Int128`/`Int256`/`UInt128`/`UInt256` (`BigInteger`), and `IPv4`/`IPv6` (`IPAddress`). The array-parameter path is deliberately scoped; these cases fall back to the standard per-element expansion (`IN (…)`) instead: a **nullable** tested column (so `NOT`-Contains null semantics match `x NOT IN (…)`), element types that need a **value conversion** (e.g. a CLR enum → `Enum8`), element CLR types the driver doesn't serialize natively inside an array (currently `TimeSpan`/`Time`), and collections used as a **queryable source** (joins, `Where(…).Contains(…)`, which keep the `SELECT … UNION ALL …` rewrite). The collection is bound as a single parameter value, so very large lists can hit the server's `http_max_field_value_size` (default 128 KiB); raise that setting for extreme cases. ([#39](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/39))
+
+### Dependencies
+* Bumped `ClickHouse.Driver` from 1.1.0 to 1.3.0. Among other fixes, 1.3.0 quotes temporal (`DateTime`/`Date`) elements inside array parameters correctly, which is what lets those element types use the single-array-parameter `Contains` path above.
+
 ### Bug fixes
 * `Sum`/`SumAsync` over a `double` or `float` column no longer throws `InvalidCastException`. EF Core wraps a top-level aggregate so the empty case returns `0`, supplying that fallback as a boxed `Int32` carrying the `Float64`/`Float32` mapping; the literal generators now convert rather than unbox. The `Float32` read path also converts, since ClickHouse widens `sum(Float32)` to `Float64` (which the driver's `GetFloat()` refuses to downcast). ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46))
 

README.md

@@ -88,6 +88,8 @@ This provider is in active development. It supports **LINQ queries**, **inserts*
 
 `Where`, `OrderBy`, `Take`, `Skip`, `Select`, `First`, `Single`, `Any`, `Count`, `Distinct`, `AsNoTracking`
 
+A `Contains` filter over a captured collection (`Where(x => ids.Contains(x.Id))`) is sent as a single native `Array(T)` parameter — `has({ids:Array(Int64)}, …)` — rather than one parameter per element, so large `IN`-style filters don't hit the parameter-count ceiling. Use `EF.Constant(ids)` to inline the values or `EF.MultipleParameters(ids)` to force one parameter per element for a specific query.
+
 ### GROUP BY & Aggregates
 
 `GroupBy` with `Count`, `LongCount`, `Sum`, `Average`, `Min`, `Max` — including `HAVING` (`.Where()` after `.GroupBy()`), multiple aggregates in a single projection, and `OrderBy` on aggregate results.

RELEASENOTES.md

@@ -1,5 +1,11 @@
-v0.3.1 (Unreleased)
+v0.4.0 (Unreleased)
 ---
+### Advanced queries
+* **Large `Contains` filters no longer blow the parameter limit.** Querying with a captured collection — `Where(x => ids.Contains(x.Id))` — now sends the whole list as a single ClickHouse array parameter (`has({ids:Array(Int64)}, …)`) instead of one parameter per element. Big `IN`-style filters that previously failed once the list grew past the server's parameter ceiling now work, and the generated SQL stays small and reusable across list sizes. For a specific query you can still opt out: `EF.Constant(ids)` inlines the values as literals and `EF.MultipleParameters(ids)` keeps the old one-parameter-per-element behavior. The array shortcut applies to integer, floating-point, `decimal`, `bool`, `string`, `Guid`, date/`DateTime`, big-integer, and IP-address values; nullable columns, enums, `Time`/`TimeSpan` values, and collections used in joins keep their existing translation. (Very large lists — hundreds of thousands of elements — may hit ClickHouse's default 128 KiB parameter-value limit; raise `http_max_field_value_size` if you need that.) ([#39](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/39))
+
+### Dependencies
+* Updated the underlying `ClickHouse.Driver` to 1.3.0.
+
 ### Bug fixes
 * Summing a `double` or `float` column (`.SumAsync(x => x.Value)`) no longer throws `InvalidCastException`. Two ClickHouse-specific mismatches were biting: EF Core hands the float literal generator a boxed `Int32` `0` as the empty-result fallback, and ClickHouse widens `sum(Float32)` to `Float64` so the driver couldn't read it back as a `float`. Both the literal generation and the `Float32` read path now convert instead of hard-casting. ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46)) (Thanks to @HotTotem!)
 

src/EFCore.ClickHouse/EFCore.ClickHouse.csproj

@@ -26,7 +26,7 @@
   <ItemGroup>
     <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="10.0.2" />
     <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.2" PrivateAssets="all" IncludeAssets="runtime; build; native; contentfiles; analyzers; buildtransitive" />
-    <PackageReference Include="ClickHouse.Driver" Version="1.1.0" />
+    <PackageReference Include="ClickHouse.Driver" Version="1.3.0" />
   </ItemGroup>
 
   <ItemGroup>

src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlNullabilityProcessor.cs

@@ -1,19 +1,131 @@
 using System.Linq.Expressions;
 using ClickHouse.EntityFrameworkCore.Query.Expressions.Internal;
+using ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping;
+using Microsoft.EntityFrameworkCore;
 using Microsoft.EntityFrameworkCore.Query;
 using Microsoft.EntityFrameworkCore.Query.SqlExpressions;
+using Microsoft.EntityFrameworkCore.Storage;
 
 namespace ClickHouse.EntityFrameworkCore.Query.Internal;
 
 public class ClickHouseSqlNullabilityProcessor : SqlNullabilityProcessor
 {
+    // Element CLR types whose collections ClickHouse.Driver serializes correctly as a single bound
+    // array parameter value (verified empirically against the pinned driver). Anything not listed here
+    // falls back to EF Core's per-element expansion, which serializes each element on its own — so an
+    // unlisted or newly-mapped type stays correct (just unoptimized) rather than failing at runtime.
+    //   • DateTime covers DateTime/DateTime64; DateOnly covers Date/Date32; BigInteger covers
+    //     Int128/Int256/UInt128/UInt256; IPAddress covers IPv4/IPv6.
+    //   • TimeSpan (Time/Time64) is deliberately excluded: the driver emits its array elements without
+    //     the quoting ClickHouse needs, so `Array(Time)` parameters fail to parse (CANNOT_READ_ARRAY).
+    private static readonly HashSet<Type> ArrayParameterElementTypes =
+    [
+        typeof(byte), typeof(sbyte), typeof(short), typeof(ushort),
+        typeof(int), typeof(uint), typeof(long), typeof(ulong),
+        typeof(float), typeof(double), typeof(decimal),
+        typeof(bool), typeof(string), typeof(Guid),
+        typeof(DateTime), typeof(DateOnly),
+        typeof(System.Numerics.BigInteger), typeof(System.Net.IPAddress),
+    ];
+
     public ClickHouseSqlNullabilityProcessor(
         RelationalParameterBasedSqlProcessorDependencies dependencies,
         RelationalParameterBasedSqlProcessorParameters parameters)
         : base(dependencies, parameters)
     {
     }
 
+    /// <summary>
+    /// Rewrites <c>column IN {collectionParameter}</c> — a captured <c>int[]</c>/<c>List&lt;T&gt;</c>/etc.
+    /// used with <c>Contains</c> — into <c>has({p:Array(T)}, column)</c>, binding the whole collection
+    /// as a single native ClickHouse array parameter instead of EF Core's default one-scalar-parameter-
+    /// per-element expansion (<c>IN (p1, …, pN)</c>).
+    /// <para>
+    /// A single bound array avoids the parameter-count / query-size ceilings that large <c>IN</c> lists
+    /// hit, and keeps the query text (and plan-cache key) independent of the collection size — ClickHouse
+    /// is OLAP and does not reuse plans by parameterization, so there is no downside to a bound array over
+    /// inlined constants.
+    /// </para>
+    /// <para>
+    /// This is the provider default for a plain captured collection. Per-query overrides win: EF Core
+    /// wires the marker methods to <see cref="SqlParameterExpression.TranslationMode"/>, so
+    /// <c>EF.MultipleParameters(...)</c> keeps the one-parameter-per-element expansion and
+    /// <c>EF.Constant(...)</c> inlines the values as literals — both handled by the base implementation.
+    /// The model-wide <c>UseParameterizedCollectionMode</c> knob is intentionally not consulted here: it
+    /// also governs the collection-as-queryable path (joins, <c>Where(...).Contains(...)</c>), which
+    /// ClickHouse translates via <c>SELECT … UNION ALL …</c> and which does not support
+    /// <see cref="ParameterTranslationMode.Parameter"/>. Inline value lists and subquery <c>IN</c> are
+    /// likewise left to the base implementation.
+    /// </para>
+    /// </summary>
+    protected override SqlExpression VisitIn(
+        InExpression inExpression,
+        bool allowOptimizedExpansion,
+        out bool nullable)
+    {
+        // An unmarked collection parameter (TranslationMode == null) takes the provider default of a
+        // single array parameter; an explicit EF.Parameter(...) selects it too. EF.MultipleParameters
+        // and EF.Constant carry a non-matching TranslationMode and fall through to the base expansion.
+        if (inExpression.ValuesParameter is { } valuesParameter
+            && valuesParameter.TranslationMode is null or ParameterTranslationMode.Parameter)
+        {
+            return VisitCollectionParameterIn(inExpression, valuesParameter, allowOptimizedExpansion, out nullable);
+        }
+
+        return base.VisitIn(inExpression, allowOptimizedExpansion, out nullable);
+    }
+
+    private SqlExpression VisitCollectionParameterIn(
+        InExpression inExpression,
+        SqlParameterExpression valuesParameter,
+        bool allowOptimizedExpansion,
+        out bool nullable)
+    {
+        // Process the tested item (usually a column) for nullability first, mirroring the base
+        // VisitIn contract.
+        var item = Visit(inExpression.Item, out var itemNullable);
+
+        // The tested item carries the authoritative element store type (the column's mapping). Align
+        // the array parameter's element type to it so the parameter serializes with the column's
+        // ClickHouse type (Int64 vs Int32, FixedString(N) vs String, …).
+        var elementMapping = (item.TypeMapping ?? valuesParameter.TypeMapping?.ElementTypeMapping) as RelationalTypeMapping;
+
+        // Bail to the base per-element expansion when a single native array parameter would be wrong
+        // or unserializable:
+        //  - nullable item: `has(arr, item)` yields a concrete 0 for a NULL item rather than NULL, so
+        //    `NOT has(...)` would KEEP NULL rows whereas `x NOT IN (...)` drops them (SQL 3-valued
+        //    logic). The base path handles null compensation, so defer to it for nullable columns; the
+        //    common large-list case (non-nullable keys) still gets the array parameter.
+        //  - no element mapping → no store type to build Array(T) from;
+        //  - the element needs a value converter (e.g. a CLR enum → Enum8) → the whole collection is
+        //    handed to the driver un-converted, which it can't serialize;
+        //  - the element CLR type isn't one the driver serializes correctly inside an array (see
+        //    ArrayParameterElementTypes — notably TimeSpan/Time is excluded).
+        // The base expansion serializes each element individually, so all these cases still work.
+        if (itemNullable
+            || elementMapping is null
+            || elementMapping.Converter is not null
+            || !ArrayParameterElementTypes.Contains(Nullable.GetUnderlyingType(elementMapping.ClrType) ?? elementMapping.ClrType))
+        {
+            return base.VisitIn(inExpression, allowOptimizedExpansion, out nullable);
+        }
+
+        // `has(array, non-null item)` is never NULL and matches `item IN (...)` for a non-null item.
+        nullable = false;
+
+        var arrayMapping = new ClickHouseArrayTypeMapping(elementMapping);
+        var arrayParameter = valuesParameter.ApplyTypeMapping(arrayMapping);
+        var alignedItem = Dependencies.SqlExpressionFactory.ApplyTypeMapping(item, elementMapping)!;
+
+        return Dependencies.SqlExpressionFactory.Function(
+            "has",
+            [arrayParameter, alignedItem],
+            nullable: false,
+            argumentsPropagateNullability: [false, false],
+            typeof(bool),
+            Dependencies.TypeMappingSource.FindMapping(typeof(bool)));
+    }
+
     protected override SqlExpression VisitSqlBinary(
         SqlBinaryExpression sqlBinaryExpression,
         bool allowOptimizedExpansion,

test/EFCore.ClickHouse.FunctionalTests/EFCore.ClickHouse.FunctionalTests.csproj

@@ -9,7 +9,7 @@
     <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.*" />
     <PackageReference Include="xunit.runner.visualstudio" Version="3.*" />
     <PackageReference Include="Testcontainers.ClickHouse" Version="4.*" />
-    <PackageReference Include="ClickHouse.Driver" Version="1.1.0" />
+    <PackageReference Include="ClickHouse.Driver" Version="1.3.0" />
     <PackageReference Include="Microsoft.EntityFrameworkCore.Relational.Specification.Tests" Version="10.0.2" />
     <PackageReference Include="GitHubActionsTestLogger" Version="3.0.1">
       <PrivateAssets>all</PrivateAssets>

test/EFCore.ClickHouse.Tests/ClickHouseQuerySqlGeneratorTests.cs

@@ -52,17 +52,21 @@ public async Task InlineCollection_Contains_RoundTrips()
     }
 
     [Fact]
-    public void InlineCollection_Contains_DoesNotTranslateToHas()
+    public void CapturedCollection_Contains_TranslatesToArrayParameter()
     {
         using var ctx = new TestDbContext(_fixture.ConnectionString);
 
+        // A captured collection used with Contains binds as a single native Array(T) parameter
+        // (issue #39) rather than expanding to one parameter per element.
         var ids = new long[] { 2, 4, 6 };
 
         var sql = ctx.TestEntities
             .Where(e => ids.Contains(e.Id))
             .ToQueryString();
 
-        Assert.DoesNotContain("has(", sql);
+        Assert.Contains("has({", sql);
+        Assert.Contains(":Array(Int64)}", sql);
+        Assert.DoesNotContain(" IN (", sql);
     }
 
     [Fact]

test/EFCore.ClickHouse.Tests/EFCore.ClickHouse.Tests.csproj

@@ -9,7 +9,7 @@
     <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.*" />
     <PackageReference Include="xunit.runner.visualstudio" Version="3.*" />
     <PackageReference Include="Testcontainers.ClickHouse" Version="4.*" />
-    <PackageReference Include="ClickHouse.Driver" Version="1.1.0" />
+    <PackageReference Include="ClickHouse.Driver" Version="1.3.0" />
     <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.2" />
     <PackageReference Include="Microsoft.EntityFrameworkCore.Relational.Specification.Tests" Version="10.0.2" />
     <PackageReference Include="GitHubActionsTestLogger" Version="3.0.1">

test/EFCore.ClickHouse.Tests/ExtendedTypeMappingTests.cs

@@ -1630,19 +1630,20 @@ public async Task DbSetRoot_SkipTakeFirst_StillUseEfBaseTranslation()
     }
 
     [Fact]
-    public async Task LocalArray_Contains_DoesNotUseHas()
+    public async Task LocalArray_Contains_UsesArrayParameterNotArrayColumnHelper()
     {
-        // Local in-memory arrays must NOT be routed through the array helpers — they have no
-        // ClickHouseArrayTypeMapping, so the type-mapping gate must reject them and let EF's
-        // inline-collection pipeline emit IN-style SQL instead of has(). This is the
-        // simplest shape that pins the gate: Contains is the most likely to mistranslate if
-        // the structural pre-filter ever loosens.
+        // A captured local array must NOT be mistaken for a mapped Array(T) column by the array-column
+        // translator (whose gate, LooksLikeArrayColumnAccess, rejects it). Since issue #39 it is instead
+        // bound as a single native Array(T) *parameter* — has({p:Array(Int64)}, `id`), where the array is
+        // the parameter and the column is the tested item — rather than expanded to one parameter per
+        // element. This pins both behaviors: the parameter placeholder confirms it's the param-array path.
         await using var ctx = new ArrayDbContext(_fixture.ConnectionString);
 
         var localIds = new long[] { 1L, 3L };
         var query = ctx.Entities.Where(e => localIds.Contains(e.Id));
         var sql = query.ToQueryString();
-        Assert.DoesNotContain("has(", sql);
+        Assert.Contains("has({", sql);
+        Assert.Contains(":Array(Int64)}", sql);
 
         var results = await query.OrderBy(e => e.Id).ToListAsync();
         Assert.Equal([1L, 3L], results.Select(r => r.Id));