[Feature] Wren memory & LanceDB: enable RaBitQ to save disk space

[qdrddr]

Is your feature request related to a problem? Please describe.
Memory usage can have a significant disk footprint when SotA embedding models are used, such as text-embedding-3-large (3072 dimensions) or Qwen3-Embedding-8B (4096 dimensions), while some other models reach 8192 dimensions.

Describe the solution you'd like
I want to benefit from LanceDB's implementation of the RaBitQ data quantization that could dramatically reduce disk footprint by sacrificing a small portion of recall accuracy.

Describe alternatives you've considered
Use embedding models with low dimensionality.

Additional context
https://www.lancedb.com/blog/feature-rabitq-quantization

[goldmedal]

Interesting. I would plan how to involve this. Thanks

[qdrddr]

You might also be interested in PostgreSQL extension called VectorChord that supports data quantization, that could be used instead of Qdrant DB.

https://docs.vectorchord.ai/vectorchord/usage/quantization-types.html

https://docs.vectorchord.ai/vectorchord/getting-started/vectorchord-suite.html