fixed join_docs.py concatenate

[nickprock]

Related Issues

• fixes JoinDocuments should use highest score when multiple retrievers recall the same document #4916

Proposed Changes:

I merged the two lists keeping only the document with the highest score

How did you test it?

Manual verification.

Checklist

• I have read the contributors guidelines and the code of conduct

[ZanSara]

Hey @nickprock! When fixing issues and modifying the behavior of some node, we normally ask contributors to add automated tests to verify that their changes behave as expected.

In this case, the tests for JoinDocuments live in tests/nodes/test_join_documents.py.

You just need to add at the end of the file a piece of code similar to this:

@pytest.mark.unit
@pytest.mark.parametrize("join_mode", ["concatenate", "merge", "reciprocal_rank_fusion"])
def test_joindocuments_keep_only_highest_ranking_duplicate(join_mode):
    inputs = [
        {"documents": [Document(content="text document 1", content_type="text", score=0.2)]},
        {"documents": [Document(content="text document 2", content_type="text", score=0.3)]},
        {"documents": [Document(content="text document 2", content_type="text", score=0.7)]},
    ]
    expected_outputs = [
        {"documents": [Document(content="text document 1", content_type="text", score=0.2)]},
        {"documents": [Document(content="text document 2", content_type="text", score=0.7)]},
    ]

    join_docs = JoinDocuments(join_mode=join_mode)
    result, _ = join_docs.run(inputs)
    assert len(result["documents"]) == 2
    assert result["documents"] == expected_outputs

(note: I haven't tested this code, you might have to adjust it a bit).

Once the test is added this PR is ready for another review 🙂

[ZanSara]

A comment about the release notes

[ZanSara]

Also, don't forget to update your branch every now and then, to make sure your PR is not too out of date with main.

[nickprock]

Hi @ZanSara I hope the PR is ok now.

[ZanSara]

One last detail and we're ready to merge 😊

[nickprock]

Hi @ZanSara,
I made all the required changes and all the tests were ok after this commit.
After branch alignment with main keeps giving me this error:

ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

[anakin87]

Hey, @nickprock! 👋

We have problems with main, which we are trying to solve in #6029.

When these problems are solved, we can take care of pushing this PR to the finish line...

[ZanSara]

LGTM! Thank you for your contribution!