What is the feature request? What problem does it solve?
To build a RAG pipeline we need to be able to create embeddings from text chunks.
The idea of building these pipelines is that they can be used as part of a private AI solution, because of this we can't just use huggingface (public) or openai's apis to generate the embeddings. We need to use a privately hosted model in VMWare to generate these embeddings.
Even though we are targetting a private AI solution embedding apis should be reasonably interchangeable.
It is important not to reinvent the wheel here and look at what langchain are doing.
Definition of done
There should be an example data job that runs and embeds chunks of text
What is the feature request? What problem does it solve?
To build a RAG pipeline we need to be able to create embeddings from text chunks.
The idea of building these pipelines is that they can be used as part of a private AI solution, because of this we can't just use huggingface (public) or openai's apis to generate the embeddings. We need to use a privately hosted model in VMWare to generate these embeddings.
Even though we are targetting a private AI solution embedding apis should be reasonably interchangeable.
It is important not to reinvent the wheel here and look at what langchain are doing.
Definition of done
There should be an example data job that runs and embeds chunks of text