Skip to content

.Net: Update Milvus memory connector to 2.3 - #5593

Merged
Mark Wallace (markwallace-microsoft) merged 2 commits into
microsoft:mainfrom
roji:Milvus2.3
Mar 25, 2024
Merged

.Net: Update Milvus memory connector to 2.3#5593
Mark Wallace (markwallace-microsoft) merged 2 commits into
microsoft:mainfrom
roji:Milvus2.3

Conversation

@roji

@roji Shay Rojansky (roji) commented Mar 21, 2024

Copy link
Copy Markdown
Member

This updates the version of the lower-level Milvus SDK from 2.2 to 2.3 (recently released).

In addition, does the following changes:

  • Changes the Milvus integration tests to use the Milvus testcontainer module for .NET; this means that a Milvus container is brought up when the tests run, removing the need to manually configure an external one.
    • As part of this, I removed the skipping to make the Milvus tests run by default; they're very reliable as far as I can tell, but if you'd rather I roll back this change (so that tests are only executed explicitly as is typical in the repo), let me know and I'll do it.
  • Implement the IMemoryStore Upsert APIs over the new 2.3 Milvus upsert operation.
  • Allow specifying the consistency level of a MilvusMemoryStore on creation.

/cc Shawn Callegari (@shawncal) Lee Miller (@lemillermicrosoft)
/cc Stephen Toub (@stephentoub) Luis Quintanilla (@luisquintanilla) Sam Patel (@SamMonoRT)

@roji
Shay Rojansky (roji) requested a review from a team as a code owner March 21, 2024 08:19
@markwallace-microsoft Mark Wallace (markwallace-microsoft) added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Mar 21, 2024
@github-actions github-actions Bot changed the title Update Milvus memory connector to 2.3 .Net: Update Milvus memory connector to 2.3 Mar 21, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Thanks, Shay.

@SamMonoRT

Copy link
Copy Markdown
Member

Shay Rojansky (@roji) - [Not blocking the PR, more for my understanding] - When I look at https://github.com/microsoft/semantic-kernel/actions/runs/8374907874/job/22931151816 for the test results page, I see it skipped over the Integration tests. How do you know if the test changes you have actually ran successfully?

@roji

Copy link
Copy Markdown
Member Author

Sam Patel (@SamMonoRT) yeah, that's a good question - I'm obviously not a maintainer on this repo but here's what I suspect is the situation.

The integration tests involve actually communicating with the service or vector database in question, and require the prior setup of that service/database; since that's hard to do for each and every service, I think the integration tests in this repo are executed manually, and it's up to the maintainer of each connector to ensure that everything works correctly (otherwise this build pipeline would have to start up all the supported vector databases, communicate to the Azure services, etc. etc.).

BTW testcontainers are meant to solve this exact problem - the tests themselves are responsible for fetching, configuring and managing the container for the service in question. This PR integrates the new Milvus testcontainer into the integration tests, but most other tests are still manual-only. Maybe with time, more connectors will integrate testcontainers into their test suite, and it would make sense to run the integration test suite in CI.

(of course, before submititng this PR I verified that all integration tests pass)

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Mar 22, 2024
@markwallace-microsoft
Mark Wallace (markwallace-microsoft) dismissed their stale review March 22, 2024 10:22

Changes cannot be merged until the integration tests are skipped by default

Merged via the queue into microsoft:main with commit 9ab9513 Mar 25, 2024
@roji
Shay Rojansky (roji) deleted the Milvus2.3 branch March 25, 2024 13:50
@roji

Copy link
Copy Markdown
Member Author

LudoCorporateShark pushed a commit to LudoCorporateShark/semantic-kernel that referenced this pull request Aug 25, 2024
This updates the version of the lower-level Milvus SDK from 2.2 to 2.3
([recently released](https://www.nuget.org/packages/Milvus.Client)).

In addition, does the following changes:
* Changes the Milvus integration tests to use the [Milvus testcontainer
module for .NET](https://testcontainers.com/modules/milvus/); this means
that a Milvus container is brought up when the tests run, removing the
need to manually configure an external one.
* As part of this, I removed the skipping to make the Milvus tests run
by default; they're very reliable as far as I can tell, but if you'd
rather I roll back this change (so that tests are only executed
explicitly as is typical in the repo), let me know and I'll do it.
* Implement the IMemoryStore Upsert APIs over the new 2.3 Milvus upsert
operation.
* Allow specifying the consistency level of a MilvusMemoryStore on
creation.

/cc @shawncal @lemillermicrosoft 
/cc @stephentoub @luisquintanilla @SamMonoRT

Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Issues or pull requests impacting the core kernel memory .NET Issue or Pull requests regarding .NET code

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

5 participants