Skip to content

Releases: meilisearch/meilisearch

v1.43.0

04 May 09:43
@dureuill dureuill
v1.43.0
475ed56
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.43.0 Latest
Latest

🌈 Enhancements

New settings indexer

The new settings indexer, a more efficient version of our internal indexer for index settings modifications, now supports filterable, sortable, facet search, and custom (asc/desc) attributes in addition to the already-supported searchable, exact, proximity precision, and embedders.

  • For Meilisearch Cloud users, the new settings indexer is disabled by default and can be enabled on a case-by-case basis for scaling purposes.
  • For OSS users, the new settings indexer can be disabled by setting the MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_SETTINGS to true.

By @Kerollmops in #6124

Improve facet search indexing performance

We reduced the time to generate the internal data structures for the facet search to work. We were previously doing a couple of full scans on the internal entries and replaced these with scans on specific entries, only the ones dedicated to the facet searchable fields.

By @Kerollmops in #6325

Improve task queue compaction integration

The GET /health route now returns HTTP 500 after a successful task queue compaction, to signal that Meilisearch should be restarted so that tasks can be enqueued again.

For Meilisearch Cloud users, this ensures that compacting the task queue will automatically restart the instance after the compaction.

🦋 Fixes

Fix lexicographic filters on strings

Fixes a bug where string facet values appearing in <, <=, >, >= and IN filters were not normalized before comparison to facet values.

This would cause some values in documents (e.g. 2026-01-01T00:00:00) to appear to be higher than their filter counterpart, due to being normalized (e.g. like 2026-01-01t00:00:00).

Thanks to @njaard for reporting the issue ❤️

By @dureuill in #6340

Fix typo tolerance v1.41 regression

Fixes the WordDelta::added_or_deleted_words function.

By @Kerollmops in #6350.

Thanks to @vladradin99 and @snurfer0 for reporting the issue in #6349 and #6324 ❤️

🔩 Miscellaneous

Thanks again @gounthar, @njaard, @vladradin99 and @snurfer0 for your contributions ❤️

Full Changelog: v1.42.1...v1.43.0

Contributors

gounthar, Kerollmops, and 6 other contributors
Assets 16

v1.42.1 🦑​

14 Apr 15:06
@ManyTheFish ManyTheFish
v1.42.1
36db1d7

Choose a tag to compare

View all tags
v1.42.1 🦑​

🪲 Bug fixes

Fix issues when using the legacy settings indexer

By @dureuill and @ManyTheFish in #6331

  • When using the experimental feature "multimodal", removing a fragment would cause an internal error in subsequent search requests
  • When using the experimental feature "multimodal", modifying fragments would sometimes cause an internal error at indexing time
  • regenerate: false would be ignored when modifying embedder settings
  • When a nested field was declared as searchable, it would not be indexed if its parents were not declared as searchable

🔩 Miscellaneous

  • Add new CI run that tests the stable settings indexer by @dureuill in #6331

Contributors

ManyTheFish and dureuill
Loading

v1.42.0 🦑​

13 Apr 09:08
@ManyTheFish ManyTheFish
v1.42.0
02ff9a4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.42.0 🦑​

✨ Enhancement

Support search fallback on remote unavailability

By @Kerollmops in #6306

We introduce a new fallback system for the sharding and replication enterprise edition feature, along with a way to determine which remote is available. The engine can avoid machines that are unavailable for a period and resume querying them once they're back online.

The following snippet shows what the /network route looks like now that this PR exposes the remote statuses/availabilities.

"remotes": {
	"prod2": {
	  "url": "http://localhost:7702",
	  "searchApiKey": "mykey",
	  "writeApiKey": "mykey",
	  "status": "available"
	},
	"prod3": {
	  "url": "http://localhost:7703",
	  "searchApiKey": "mykey",
	  "writeApiKey": "mykey",
	  "status": "unavailable"
	}
}

🔬 Experimental: Document join Filtering

By @ManyTheFish in #6314

This enhancement extends the Cross-index document hydration introduced in v1.39.0 by allowing the user to filter on the foreign indexes to retrieve the documents.

📓 Note: This implementation doesn't support a remote sharding environment

foreignKeys experimental feature

TheforeignKeys experimental feature must be activated to use the foreign filters:

curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
  -H 'Content-Type: application/json' \
  --data-binary '{"foreignKeys": true}'

foreignKeys + filter index setting

To be able to use the foreign filters, the related field must be set as a foreignKey and as a filterableAttribute in /indexes/{index_uid}/settings:

{
	// new setting, an array of foreign keys that allows multiple foreign relationships between indexes
	"foreignKeys":  [
		{
			// the path in the JSON document containing foreign document ids
			"fieldName": "actors",
			
			// the UID of the foreign index containing the documents to fetch during hydration
	        "foreignIndexUid": "actors"
		}
	],
	// the actors field must be filterable on equality
	"filterableAttributes": [
		{
	    	"attributePatterns": ["actors"],
	     	"features": {
				"facetSearch": false,
				"filter": {
					"equality": true,
					"comparison": false
				}
			}
		}
	]
}

filtering using the _foreign filter

On the search route, a new _foreign verb has been introduced and should be used as follows:

{
	"q": "<query>",

	// filters on the movie index:
	// genres = action
    // AND
    // the foreign documents from the actor index match:  birthday STARTS WITH \"1958-\" AND popularity >= 3.5
	"filter": "genres = action AND _foreign(actors, birthday STARTS WITH \"1958-\" AND popularity >= 3.5)"
}

Note: nesting foreign filters is not supported and will return an error

Example of usage

Prerequisites

  • Meilisearch running on 127.0.0.1:7700 on the document-join-hydration branch.

Step 1: Enable Foreign Keys Feature

curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
  -H 'Content-Type: application/json' \
  --data-binary '{"foreignKeys": true}'

Step 2: Create Indexes

Create the actors index
curl -X POST 'http://127.0.0.1:7700/indexes' \
  -H 'Content-Type: application/json' \
  --data-binary '{"uid": "actors", "primaryKey": "id"}'
Create the movies index
curl -X POST 'http://127.0.0.1:7700/indexes' \
  -H 'Content-Type: application/json' \
  --data-binary '{"uid": "movies", "primaryKey": "id"}'

Step 3: Add Documents to the actors Index

curl -X POST 'http://127.0.0.1:7700/indexes/actors/documents' \
  -H 'Content-Type: application/json' \
  --data-binary '[
  {"id": 1, "name": "Tom", "familyName": "Hanks", "birthDate": "1956-07-09"},
  {"id": 2, "name": "Meryl", "familyName": "Streep", "birthDate": "1949-06-22"},
  {"id": 3, "name": "Leonardo", "familyName": "DiCaprio", "birthDate": "1974-11-11"},
  {"id": 4, "name": "Emma", "familyName": "Watson", "birthDate": "1990-04-15"}
]'

Step 4: Add Documents to the movies Index

curl -X POST 'http://127.0.0.1:7700/indexes/movies/documents' \
  -H 'Content-Type: application/json' \
  --data-binary '[
  {"id": 1, "title": "Forrest Gump", "description": "The presidencies of Kennedy and Johnson, the Vietnam War, the Watergate scandal and other historical events unfold from the perspective of an Alabama man with an IQ of 75.", "actors": [1]},
  {"id": 2, "title": "The Devil Wears Prada", "description": "A smart but sensible new graduate lands a job as an assistant to Miranda Priestly, the demanding editor-in-chief of a high fashion magazine.", "actors": [2, 4]},
  {"id": 3, "title": "Inception", "description": "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", "actors": [3]},
  {"id": 4, "title": "Cast Away", "description": "A FedEx executive undergoes a physical and emotional transformation after crash landing on a deserted island.", "actors": [1]}
]'

Step 5: Configure Foreign Keys on the movies Index

curl -X PATCH 'http://127.0.0.1:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{"foreignKeys": [{"fieldName": "actors", "foreignIndexUid": "actors"}], "filterableAttributes": [{"attributePatterns": ["actors"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}'

Step 6: Configure filterable on the actors Index

curl -X PATCH 'http://127.0.0.1:7700/indexes/actors/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{"filterableAttributes": [{"attributePatterns": ["birthDate"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}'

Step 7: Perform a Federated Search

curl -X POST 'http://127.0.0.1:7700/multi-search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
  "queries": [
    {
      "indexUid": "movies",
      "q": "Forrest",
      "filter": "_foreign(actors, birthDate = \"1956-07-09\")"
    }
  ],
  "federation": {
    "limit": 20,
    "offset": 0
  }
}'

Expected Result

The federated search should return movie documents with the actors array automatically hydrated with full actor objects instead of just IDs:

{
  "hits": [
    {
      "id": 1,
      "title": "Forrest Gump",
      "description": "...",
      "actors": [
        {
          "id": 1,
          "name": "Tom",
          "familyName": "Hanks",
          "birthDate": "1956-07-09"
        }
      ],
      "_federation": {
        "indexUid": "movies",
        "queriesPosition": 0,
        "weightedRankingScore": 0.9848484848484849
      }
    }
  ],
  "processingTimeMs": 208,
  "limit": 20,
  "offset": 0,
  "estimatedTotalHits": 1
}

🪲 Bug fixes

  • Fix a race condition when writing network by @Kerollmops in #6300

    We fixed a race condition in network topology changes that could cause errors and prevent documents from being correctly indexed. Additionally, we fixed a bug in the networkTopologyChange task batching that was causing it to batch too many task types. We made sure it only batches import tasks, and only those, to avoid out-of-order task processing.

  • Throw document template errors when updating the chat settings by @Kerollmops in #6321

    We fixed an issue that prevented the engine from explicitly showing the possible document template errors users could encounter when updating the template in the chat settings. The engine now correctly checks for and throws template errors when they are detected.

  • Fix: Update Index tasks will be properly forwarded to remote nodes by @dureuill in #6299

  • Fix action mistake on the chat completions route by @Kerollmops in #6290

🔩 Miscellaneous

  • Use the latest version of heed with nested rtxns support by @Kerollmops in #6316

    This PR bumps the versions of crates that use heed to the latest version, v0.22.1. This version finally stabilized a long-standing piece of work we were doing with Howard Chu: nested read transactions. We no longer have to rely on unstable pre-releases, but rather on a clean, stable version of LMDB (still a fork, but a better one).

  • Add section to CONTRIBUTING.md to bump mini-dashboard version and testing section to right place by @curquiza in #6195

  • Make the no-agent AGENTS.ms more permissive by @Kerollmops in #6260

  • Remove deleted test commands by @Strift in #6283

  • Fix OpenAPI schema generation for chat completions route by @qdequele in #6274

  • Rename OpenAPI route names for search rules and compact by @qdequele in #6298

  • Update README with new features and demos by @qdequele in #6297

  • Prevent shell injection in benchmark workflows by @curquiza & @Kerollmopsin #6308 & #6318

  • Rename so...

Read more

Contributors

Kerollmops, Strift, and 4 other contributors
Loading

v1.41.0 🏈

30 Mar 13:55
@YoEight YoEight
v1.41.0
8dd5a95
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.41.0 🏈

✨ Enhancement

  • Support dynamic search rules with pinning by @YoEight in #6182
    Introduce the Dynamic Search Rules (DSR) experimental feature, making it easy to promote the right content for the right searches with flexible, condition-based pinning. In this first version, rules can be triggered by query-based conditions such as empty queries or literal substring matches, as well as time windows. Multiple documents can be pinned in a defined order, and pins still work with filtering, pagination, facet distribution, hybrid search, and federated search. This feature is behind the dynamicSearchRules experimental feature flag.

    Rules can be created or updated with PATCH /dynamic-search-rules/{uid} and removed with DELETE /dynamic-search-rules/{uid}. In this first version, a rule can define query- or time-based conditions and pin specific documents at fixed positions in the results list.

    Example of rule creation/update 
      // PATCH /dynamic-search-rules/featured-wireless-headphones
  {
    "description": "Promote featured products for wireless headphone searches",
    "active": true,
    "conditions": [
      { "scope": "query", "contains": "headphone" }
    ],
    "actions": [
      {
        "selector": { "indexUid": "products", "id": "featured-headphones-001" },
        "action": { "type": "pin", "position": 0 }
      },
      {
        "selector": { "indexUid": "products", "id": "featured-headphones-002" },
        "action": { "type": "pin", "position": 1 }
      }
    ]
  }

  • Default to useNetwork: true in sharded instances by @dureuill in #6278

    This release contains breaking changes for users of the network experimental feature.

    Search requests now use the network by default when (replicated) sharding is enabled
    § Breaking changes

    When network.leader is set in the instance, useNetwork now defaults to true in search requests when omitted.

    So, when a network is configured, the following:

    Search request without an explicit useNetwork 
    // POST /indexes/movies/search
{
  "q": "Batman dark knight returns 1",
  "filter": "genres IN [Action, Adventure]",
  "facets": ["genres"],
  "limit": 5
  // No `useNetwork` is passed
}

    is now equivalent to a federated search over multiples remotes so that all shards in the network are covered exactly once.

    Equivalent federated search request 
    // POST /multi-search
{
    "federation": {
        "limit": 5,
        "facetsByIndex": {
            "movies": [
                "genres"
            ]
        },
        "merge": {}
    },
    // assuming 3 remotes 0, 1, 2 and 3 shards a, b, c such that
    // 0 owns a, b
    // 1 owns b, c
    // 2 owns c, a
    "queries": [
        {
            "indexUid": "movies",
            "federationOptions": {
                "remote": "0"
            },
            "q": "Batman dark knight returns 1",
            "filter": ["genres IN [Action, Adventure]", "_shard = a"]
        },
        {
            "indexUid": "movies",
            "federationOptions": {
                "remote": "0" // useNetwork picked remote 0 for both shard a and b
            },
            "q": "Batman dark knight returns 1",
		     "filter": ["genres IN [Action, Adventure]", "_shard = b"]
        },
        {
            "indexUid": "movies",
            "federationOptions": {
                "remote": "2"
            },
            "q": "Batman dark knight returns 1",
		     "filter": ["genres IN [Action, Adventure]", "_shard = c"]
        }
    ]
}

    Responding with documents from all the shards:

    Response 
    {
  "hits": [
    {
      "id": 123025,
      "title": "Batman: The Dark Knight Returns, Part 1",
      "overview": "Batman has not been seen for ten years. A new breed of criminal ravages Gotham City, forcing 55-year-old Bruce Wayne back into the cape and cowl. But, does he still have what it takes to fight crime in a new era?",
      "genres": [
        "Action",
        "Animation",
        "Mystery"
      ],
      "poster": "https://image.tmdb.org/t/p/w500/kkjTbwV1Xnj8wBL52PjOcXzTbnb.jpg",
      "release_date": 1345507200,
      "_federation": {
        "indexUid": "mieli",
        "queriesPosition": 2,
        "weightedRankingScore": 0.9894586894586894,
        "remote": "2"
      }
    },
    {
      "id": 142061,
      "title": "Batman: The Dark Knight Returns, Part 2",
      "overview": "Batman has stopped the reign of terror that The Mutants had cast upon his city.  Now an old foe wants a reunion and the government wants The Man of Steel to put a stop to Batman.",
      "genres": [
        "Action",
        "Animation",
        "Mystery"
      ],
      "poster": "https://image.tmdb.org/t/p/w500/arEZYd6uMOFTILne9Ux0A8qctMe.jpg",
      "release_date": 1357171200,
      "_federation": {
        "indexUid": "mieli",
        "queriesPosition": 1,
        "weightedRankingScore": 0.9894558963186414,
        "remote": "0"
      }
    },
    {
      "id": 16234,
      "title": "Batman Beyond: Return of the Joker",
      "overview": "The Joker is back with a vengeance, and Gotham's newest Dark Knight needs answers as he stands alone to face Gotham's most infamous Clown Prince of Crime.",
      "genres": [
        "Animation",
        "Family",
        "Action",
        "Science Fiction"
      ],
      "poster": "https://image.tmdb.org/t/p/w500/7RlBs0An83fqAuKfwH5gKMcqgMc.jpg",
      "release_date": 976579200,
      "_federation": {
        "indexUid": "mieli",
        "queriesPosition": 1,
        "weightedRankingScore": 0.9427964918160996,
        "remote": "0"
      }
    },
    {
      "id": 155,
      "title": "The Dark Knight",
      "overview": "Batman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves to be effective, but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham as the Joker.",
      "genres": [
        "Drama",
        "Action",
        "Crime",
        "Thriller"
      ],
      "poster": "https://image.tmdb.org/t/p/w500/qJ2tW6WMUDux911r6m7haRef0WH.jpg",
      "release_date": 1216166400,
      "_federation": {
        "indexUid": "mieli",
        "queriesPosition": 1,
        "weightedRankingScore": 0.5784178187403994,
        "remote": "0"
      }
    },
    {
      "id": 49026,
      "title": "The Dark Knight Rises",
      "overview": "Following the death of District Attorney Harvey Dent, Batman assumes responsibility for Dent's crimes to protect the late attorney's reputation and is subsequently hunted by the Gotham City Police Department. Eight years later, Batman encounters the mysterious Selina Kyle and the villainous Bane, a new terrorist leader who overwhelms Gotham's finest. The Dark Knight resurfaces to protect a city that has branded him an enemy.",
      "genres": [
        "Action",
        "Crime",
        "Drama",
        "Thriller"
      ],
      "poster": "https://image.tmdb.org/t/p/w500/vzvKcPQ4o7TjWeGIn0aGC9FeVNu.jpg",
      "release_date": 1342396800,
      "_federation": {
        "indexUid": "mieli",
        "queriesPosition": 2,
        "weightedRankingScore": 0.5772657450076805,
        "remote": "2"
      }
    }
  ],
  "query": "Batman dark knight returns 1",
  "processingTimeMs": 173,
  "limit": 5,
  "offset": 0,
  "estimatedTotalHits": 47,
  "facetDistribution": {
    "genres": {
      "Action": 46,
      "Adventure": 15,
      "Animation": 34,
      "Comedy": 3,
      "Crime": 14,
      "Drama": 6,
      "Family": 15,
      "Fantasy": 8,
      "Horror": 1,
      "Mystery": 4,
      "Romance": 1,
      "Science Fiction": 14,
      "TV Movie": 4,
      "Thriller": 4,
      "Western": 1
    }
  },
  "facetStats": {},
  "requestUid": "019bbcf4-a609-7701-8d82-d370611adfb3",
  "remoteErrors": {}
}

    This change allows to naturally query all the documents in a sharded context.
    When network.leader is not present (in particular, when no network was defined), then the behavior is identical to previous versions.

  • useNetwork picks local instance if it owns the shard by @dureuill in #6287
    To prevent unnecessary networ...

Read more

Contributors

YoEight, Kerollmops, and 3 other contributors
Loading

v1.40.0 🦈

23 Mar 10:30
@Kerollmops Kerollmops
v1.40.0
7fbd466
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.40.0 🦈

This release introduced support for the distinct attribute in federated search, enabling cross-index distinct attributes with facet distribution support. Additionally, significant performance improvements were delivered, including faster federated search (approximately 100ms faster), optimized JSON document generation for better handling of large documents and a much better memory usage for large workloads.

✨ Enhancement

  • Support distinct in federated search by @dureuill in #6214

    The distinct attribute can now be passed to the federation object in federated search to apply a global, cross-index and cross-remote distinct computation to the results.

    Example of a federated search request with distinct 
     {
   "federation": {
     "distinct": "genres", // ✨ NEW
     "facetsByIndex": { // recovering facet distribution is also supported with distinct
       "comics": [
         "genres"
       ],
       "movies": [
         "genres"
       ]
     },
     "mergeFacets": {} // merging facet distributions is also supported with distinct
   },
   "queries": [
     {
       "indexUid": "comics",
       "q": "batman",
       "attributesToRetrieve": ["title", "genres"],
       "useNetwork": true // distinct is also supported with network queries
     },
     {
       "indexUid": "movies",
       "q": "superman",
       "attributesToRetrieve": ["title", "genres"],
       "useNetwork": true
     }
   ]
 }
    Sample response to a federated search request with distinct 
     {
   "hits": [
     {
       "title": "Batman",
       "genres": [
         "Family",
         "Adventure",
         "Comedy",
         "Science Fiction",
         "Crime"
       ],
       "_federation": {
         "indexUid": "comics",
         "queriesPosition": 0,
         "weightedRankingScore": 1.0,
         "remote": "ms2"
       }
     },
     {
       "title": "Batman",
       "genres": [
         "Fantasy",
         "Action"
       ],
       "_federation": {
         "indexUid": "comics",
         "queriesPosition": 0,
         "weightedRankingScore": 1.0,
         "remote": "ms1"
       }
     },
     {
       "title": "Batman & Bill",
       "genres": [
         "Documentary"
       ],
       "_federation": {
         "indexUid": "comics",
         "queriesPosition": 0,
         "weightedRankingScore": 0.9848484848484848,
         "remote": "ms1"
       }
     },
     {
       "title": "Superman: Red Son",
       "genres": [],
       "_federation": {
         "indexUid": "movies",
         "queriesPosition": 1,
         "weightedRankingScore": 0.9848484848484849,
         "remote": "ms0"
       }
     },
     {
       "title": "Superman, Spider-Man or Batman",
       "genres": [
         "Drama"
       ],
       "_federation": {
         "indexUid": "movies",
         "queriesPosition": 1,
         "weightedRankingScore": 0.9848484848484849,
         "remote": "ms0"
       }
     }
   ],
   "processingTimeMs": 15,
   "limit": 5,
   "offset": 0,
   "estimatedTotalHits": 11,
   "facetDistribution": {
     "genres": {
       "Action": 1,
       "Adventure": 1,
       "Comedy": 3,
       "Crime": 2,
       "Documentary": 1,
       "Drama": 1,
       "Family": 1,
       "Fantasy": 1,
       "Horror": 2,
       "Romance": 1,
       "Science Fiction": 1,
       "Thriller": 1,
       "Western": 1
     }
   },
   "facetStats": {},
   "requestUid": "019d05c7-ea65-77a1-8274-22a8ba9e26db",
   "remoteErrors": {}
 }

    Note the following to apply the distinct attribute at the federation level:

    1. Applying distinct at the query level at the same time as the federation level is disallowed and will return a HTTP 400 error.
    2. The chosen distinct field will apply to all remotes and indexes, so it must be a filterable attribute for all participating remotes and indexes.
    3. While Meilisearch attempts to compute the most accurate facet distribution, in distributed contexts this cannot be guaranteed as the distinct algorithm is not applied on all of the remote documents.

  • Improve performance of federated search by @dureuill in #6229

    Improves performance of federated search: about 100ms faster for all requests. Improves reliability of the HTTP server: the server will no longer be blocked when too many federated search requests are being processed.

  • Optimize the generation of JSON documents by @Kerollmops in #6257

    Addresses performance issues that users might encounter when requesting large documents. Additionally, performance is enhanced when users request only a small subset of fields from large documents.

  • Use the latest version of mimalloc to improve memory usage by @Kerollmops in #6201

    Updates mimalloc from v2 to v3, improving memory sharing between threads and significantly reducing memory usage on large workloads. It also overrides the allocator to use mimalloc at linking time, allowing LMDB, Meilisearch, and other C libraries to share their allocations for better overall memory efficiency. @Kerollmops wrote a blog post about the story behind this improvement.

  • Add POST /tasks/compact for task queue compaction by @YoEight in #6193

    Compacts the task queue database and reclaim space so new tasks can keep being enqueued, without deleting existing tasks. This feature is behind the taskQueueCompactionRoute experimental feature flag.

Warning

Once task queue compaction completes, all write operations are blocked until the server is restarted.

🔐 Security

  • Bump rustls-webpki from 0.103.8 to 0.103.10 in #6273
  • Bump tar from 0.4.44 to 0.4.45 in #6271
  • Bump actix-web-lab from 0.24.3 to 0.26.0 in #6218
  • Bump quinn-proto from 0.11.13 to 0.11.14 in #6217

🪲 Bug fixes

🔩 Miscellaneous

New Contributors

Contributors

YoEight, GuillaumeGomez, and 5 other contributors
Loading

v1.39.0 🐳​

16 Mar 12:54
@ManyTheFish ManyTheFish
v1.39.0
21a9178
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.39.0 🐳​

What's Changed

✨ Enhancement

🔬 Cross-index document hydration

Add a new ForeignKeys setting allowing to hydrate documents with documents coming from other indexes.

📓 Note: This implementation doesn't support a remote sharding environment

foreignKeys experimental feature

A new experimental feature, foreignKeys, has been added to the /experimental-feature route that must be set to true to activate the hydration.

curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
  -H 'Content-Type: application/json' \
  --data-binary '{"foreignKeys": true}'

foreignKeys index setting

A new index setting, foreignKeys, has been added to the /indexes/{index_uid}/settings:

// new setting, an array of foreign keys that allows multiple foreign relationships between indexes
"foreignKeys":  [
	{
		// the path in the JSON document containing foreign document ids
		"fieldName": "actors",
		
		// the UID of the foreign index containing the documents to fetch during hydration
        "foreignIndexUid": "actors"
	}
]

Example

With this new feature, a document shaped as follows:

{
  "id": 1,
  "title": "Forrest Gump",
  // Document IDs in foreign index
  "actors": [
    1
  ]
}

Will be hydrated as follows in a search response:

{
  "id": 1,
  "title": "Forrest Gump",
  "actors": [
	{
	  "id": 1,
	  "name": "Tom",
	  "familyName": "Hanks",
	  "birthDate": "1956-07-09"
	}
  ]
}

By @ManyTheFish in #6047

Disable proxy response buffering on Server-Sent Events (SSE)

Add X-Accel-Buffering: no on POST /chats/{workspace_uid}/chat/completions when the streaming mode is activated.
By @YoEight in #6228

🪲 Bug fixes

  • Fix a memory leak in the indexation pipeline by @Kerollmops in #6212

    We fixed an important memory leak caused by an invalid use of bumpalo. If you've seen Meilisearch using more and more memory over time, this issue is no longer visible in the latest version. If you want to read more about the bumpalo-trap we felt in, here is a good summary. It looks like this leak was introduced in v1.12, so approximately a year ago...

  • Avoid losing tasks by applying deltas on the wtxn view by @Kerollmops in #6222

    We addressed a small performance regression introduced in v1.38.1 that affected sending updates to the engine during task deletion. We've restored the task deletion performance to match v1.38.0 levels and ensured no race conditions occur while still allowing updates during deletion.

🔩 Miscellaneous

Full Changelog: v1.38.0...v1.39.0

Contributors

ManyTheFish
Loading

v1.38.2 🐍

11 Mar 11:36
@dureuill dureuill
v1.38.2
238de27
This commit was signed with the committer’s verified signature.
dureuill Louis Dureuil
SSH Key Fingerprint: Gp59QShA/YuuOe/+7rjuCfziYkXhffjZsVTtwvgjg+Y
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.38.2 🐍

Meilisearch v1.38.2 fixes a regression introduced in Meilisearch v1.38.0 where Meilisearch could sometimes stop processing tasks after an automatic task queue cleanup performed while more tasks where being added.

We recommend that all users in v1.38.0 and v1.38.1 migrate to v1.38.2.

Note that the dumpless upgrade process can take up to one minute before the HTTP server becomes available when migrating from a previous v1.38 version.

Full Changelog: v1.38.1...v1.38.2

Loading

v1.38.1 🐍

11 Mar 10:11
@Kerollmops Kerollmops
v1.38.1
fb41808
This commit was signed with the committer’s verified signature.
Kerollmops Clément Renault
GPG key ID: 58648EC4005711EE
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.38.1 🐍

Meilisearch v1.38.1 fixes a regression we have seen in task processing, which caused the engine to stop processing tasks.

🪲 Bug fixes

Full Changelog: v1.38.0...v1.38.1

Contributors

Kerollmops
Loading

v1.38.0 🐍

09 Mar 08:06
@Kerollmops Kerollmops
v1.38.0
807a5db
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.38.0 🐍

Meilisearch v1.38 drastically improves indexing performance for embeddings, cleans up the task queue of orphan tasks and batches, ensures they are correctly deleted now, fixes issues users may encounter with OpenAI or VoyageAI when generating embeddings, and improves the tooling for generating routes in the documentation.

✨ Enhancement

  • Embeddings: Use the latest version of hannoy with improved performances by @Kerollmops in #6184
    We drastically reduced embedding indexing time by eliminating full database scans. It is more efficient to add a couple of embeddings to a large database now.
  • Tasks: Optimize and fix task deletion by @Mubelotix and @Kerollmops in #5828

🪲 Bug fixes

  • Embedders: Fix the intermittent connection reset by peer errors when embedding documents or search queries using a remote embedder. By @dureuill in #6191

🔩 Miscellaneous

  • Enforce that routes appear in the reference documentation by @dureuill and @ManyTheFish in #6200
    The reference API is now generated from the source code, but it used to miss some actix-declared routes. It is now mandatory to attach a route when using the newly-provided routes::routes and routes::path macros, which ensures the route is added to the reference. Please refer to the documentation of the individual macros and the many examples throughout the codebase for more details.
  • Chore: remove verbose debug log on list documents by @StephaneRob in #6198
  • Add guide to CONTRIBUTING.md to test doc locally by @curquiza in #6180
  • Docs: capitalize LangChain and MCP in README by @24f1000637 in #6183
  • Docs: improve documentation of the parameters of the search route by @curquiza in #6194

New Contributors

Contributors

Kerollmops, StephaneRob, and 5 other contributors
Loading

v1.37.0

02 Mar 09:16
@dureuill dureuill
v1.37.0
d6c0f4b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.37.0

Important

This release contains breaking changes for users of the network experimental feature.

Meilisearch v1.37 introduces replicated sharding, removes the vectorStoreSetting experimental feature, stabilizes our new vector store for best performance, adds a security fix and miscellaneous improvements.

✨ Improvements

§ Replicated sharding

Note

Replicated sharding requires Meilisearch Enterprise Edition (EE).

  • Users of Meilisearch Cloud, please contact support if you need replicated sharding.
  • Users of the Community Edition, please contact the sales if you want to use replicated sharding in production.

§ Breaking changes

  • network objects sent to the PATCH /network route must now contain at least one shard object containing at least one remote when leader is not null.

Existing databases will be migrated automatically when upgraded with --experimental-dumpless-upgrade when leader is not null, such that for each remote:

  1. A shard with the same name as the remote is created
  2. This shard has exactly one remote in its remotes list: the remote with the same name as the shard.

This change will not cause any document to be resharded.

To be able to upgrade without resharding, the migration uses the same name for remotes and for shards. However, in new configurations, we recommend using different names for shards and remotes.

Example of migration

For instance, the following network object:

{
  "leader": "ms-00",
  "self": "ms-01",
  "remotes": {
    "ms-00": { /* .. */ },
    "ms-01": { /* .. */ }
  }
}

is converted to:

{
  "leader": "ms-00",
  "self": "ms-01",
  "remotes": {
    "ms-00": { /* .. */ },
    "ms-01": { /* .. */ }
  },
  "shards": {  // ✨ NEW
    "ms-00": {  // shard named like the remote
      "remotes": ["ms-00"] // is owned by the remote
    },
    "ms-01": {
      "remotes": ["ms-01"]
    }
  }
}

Addition of network.shards

The network object for routes PATCH /network and GET /network now contains the new field shards, which is an object whose values are shard objects, and keys the name of each shard.

Each shard object contains a single field remotes, which is an array of strings, each string representing the name of an existing remote.

Convenience fields

The shard objects in PATCH /network contain the additional fields addRemotes and removeRemotes meant for convenience:

  • pass an array of remote names to shard.addRemotes to add these remotes to the list of remotes of a shard.
  • pass an array of remote names to shard.removeRemotes to remove these remotes from the list of remotes of a shard.
  • if present and non-null, shard.remotes will completely override the existing list of remotes for a shard.
  • if several of these options are present and non-null, then the order of application is shard.remotes, then shard.addRemotes, then shard.removeShards.
Adding a new shard with some remotes 
// PATCH /network
{
  // assuming that remotes `ms-0`, `ms-1`, `ms-2` where sent in a previous call to PATCH /network
  "shards": {
    "s-a": { // new shard
      "remotes": ["ms-0", "ms-1"]
    }
  }
}

Remotes ms-0 and ms-1 own the new shard s-a.

Fully overriding the list of remotes owning a shard 
// PATCH /network
{
  // assuming remotes `ms-0`, `ms-1`, `ms-2`
  // assuming shard `s-a`, owned by `ms-0` and `ms-1`
  "shards": {
    "s-a": {
      "remotes": ["ms-2"]
    }
  }
}

ms-2 is now the sole owner of s-a, replacing ms-0 and ms-1.

Adding a remote without overriding the list of remotes owning a shard 
// PATCH /network
{
  // assuming remotes `ms-0`, `ms-1`, `ms-2`
  // assuming shard `s-a`, owned by `ms-2`
  "shards": {
    "s-a": {
      "addRemotes": ["ms-0"]
    }
  }
}

ms-0 and ms-2 are now the owners of s-a.

Removing a remote without overriding the list of remotes owning a shard 
// PATCH /network
{
  // assuming remotes `ms-0`, `ms-1`, `ms-2`
  // assuming shard `s-a`, owned by `ms-0` and `ms-2`
  "shards": {
    "s-a": {
      "removeRemotes": ["ms-2"]
    }
  }
}

ms-0 is now the sole owner of s-a.

Entirely removing a shard from the list of shards

Set the shard to null:

// PATCH /network
{
  "shards": {
    "s-a": null
  }
}

Or set its remotes list to the empty list:

// PATCH /network
{
  "shards": {
    "s-a": {
      "remotes": []
    }
  }
}

network.shards validity

When network.leader is not null, each shard object in network.shards must:

  1. Only contain remotes that exist in the list of remotes.
  2. Contain at least one remote.

Additionally, network.shards must contain at least one shard.

Failure to meet any of these conditions will cause the PATCH /network route to respond with 400 invalid_network_shards.

Change in sharding logic

Documents are now sharded according to the list of shards declared in the network rather than the list of remotes. All remotes owning a shard will process the documents that belong to this shard, allowing for replication.

Example of replication

The following configuration defines 3 remotes 0, 1 and 2, and 3 shards A, B, C, such that each remote owns two shards, achieving replication (losing one remote does not lose any document).

{
  "leader": "0",
  "self": "0",
  "remotes": {
    "0": { /* .. */ },
    "1": { /* .. */ },
    "2": { /* .. */ }
  },
  "shards": {
    "A": {
      "remotes": ["0", "1"]
    },
    "B": {
      "remotes": ["1", "2"]
    },
    "C": {
      "remotes": ["2", "0"]
    }
  }
}
  • Full replication is supported by having all remotes own all the shards.
  • Unbalanced replication is supported by having some remotes own more shards than other remotes.
  • "Watcher" remotes are supported by having remotes that own no shards. Watcher remotes are not very useful in this release, and might be upgraded in a future release, so that they keep all documents without indexing them, allowing to "respawn" shards for other remotes.

useNetwork takes network.shards into account

When useNetwork: true is passed to a search query, it is expanded to multiple queries such that each shard declared in network.shards appears exactly once, associated with a remote that owns that shard.

This ensures that there is no missing or duplicate documents in the results.

_shard filters

When the network experimental feature is enabled, then it becomes possible to filter documents depending on the shard they belong to.

Given s-a and s-b the names of two shards declared in network.shards, then:

  • _shard = "s-a" in a filter parameter to the search or documents fetch will return the documents that belong to s-a.
  • _shard != "s-a" will return the documents that do not belong to s-a
  • _shard IN ["s-a", "s-b"] will return the documents that belong to s-a or to s-b.

You can use these new filters in manual remote federated search to create a partitioning over all shards in the network.

Important

To avoid duplicate or missing documents in results, for manually crafted remote federated search requests, all shards should appear in exactly one query.

Tip

Search requests built with useNetwork: true already build a correct partitioning over shards. They should be preferred to manually crafted remote federated search requests in replicated sharding scenarios.

Update instructions

When updating your Meilisearch network using dumpless upgrade, please observe the following guidelines:

  1. Do not call the PATCH /network route until all remotes of the network are finished updating
  2. If using the search routes with useNetwork: true, call them on un-updated remotes. Calling it on already updated remotes will cause un-updated remotes to fail the search as they don't know about the _shard filters.

By @dureuill in #6128

§ Remove vectorStoreSetting experimental feature

The new HNSW vector store (hannoy) has been stabilized and is now the only supported vector store in Meilisearch.

As a result, updating to v1.37.0 will migrate all remaining legacy vector store indexes (using arroy) to hannoy, and the vectorStoreSetting experimental feature is no longer available.

By @Kerollmops in #6176

Improve indexing performance for embeddings

We removed a computationally expensive step from vector indexing.

On a DB with 20M documents, this removes 300s per indexing batch of 1100s.

By @Kerollmops in #6175

§ 🔒 Security

  • Bump mini-dashboard (local web interface) which
    • now stores API key in RAM instead of in the localStorage
    • bumps dependencies with po...
Read more

Contributors

Kerollmops, Strift, and 2 other contributors
Loading