Skip to content

Commit 1a7fff6

Browse files
shiva-istariShiva
andauthored
feat(bulk): add HNSW vector index support to bulk loader (#9598)
**Description** Adds HNSW vector index support to Dgraph's bulk loader, enabling bulk loading of vector predicates with automatic index construction. **Checklist** - [x] The PR title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) syntax, leading with `fix:`, `feat:`, `chore:`, `ci:`, etc. - [x] Code compiles correctly and linting (via trunk) passes locally - [x] Tests added for new functionality, or regression tests for bug fixes added as applicable - [ ] For public APIs, new features, etc., a PR on the [docs repo](https://github.com/dgraph-io/dgraph-docs) staged and linked here. This process can be simplified by going to the [public docs site](https://docs.dgraph.io/) and clicking the "Edit this page" button at the bottom of page(s) relevant to your changes. Ensure that you indicate in the PR that this is an **unreleased** feature so that it does not get merged into the main docs prematurely. --------- Co-authored-by: Shiva <shiva@Shivajis-MacBook-Pro.local>
1 parent 0043595 commit 1a7fff6

File tree

7 files changed

+1563
-29
lines changed

7 files changed

+1563
-29
lines changed

dgraph/cmd/bulk/loader.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ type state struct {
9797
mapFileId uint32 // Used atomically to name the output files of the mappers.
9898
dbs []*badger.DB
9999
tmpDbs []*badger.DB // Temporary DB to write the split lists to avoid ordering issues.
100+
vectorTmpDb *badger.DB // Shared temporary DB for vector data and HNSW index building.
100101
writeTs uint64 // All badger writes use this timestamp
101102
namespaces *sync.Map // To store the encountered namespaces.
102103
errorLog *errorLogger // Error logger for --log_errors
@@ -586,6 +587,11 @@ func (ld *loader) cleanup() {
586587
x.Check(db.Close())
587588
x.Check(os.RemoveAll(opts.Dir))
588589
}
590+
if ld.vectorTmpDb != nil {
591+
opts := ld.vectorTmpDb.Opts()
592+
x.Check(ld.vectorTmpDb.Close())
593+
x.Check(os.RemoveAll(opts.Dir))
594+
}
589595
if ld.errorLog != nil {
590596
if err := ld.errorLog.Close(); err != nil {
591597
glog.Warningf("error closing error log: %v", err)

0 commit comments

Comments
 (0)