GH-44501: [C#] Use an index lookup for O(1) field index access

[vthemelis]

Fixes #44501

Rationale for this change

This should speed up the reasonably common operation of looking up a column via name in Schema or RecordBatch.

I would argue that the fact that this operation was previously O(N) (now O(1)) is unintuitive and could easily lead to accidental performance woes.

Note that I would like to also replace the existing Lookups with signature string -> Field but unfortunately those use StringComparer.Default instead of StringComparer.CurrentCulture (which I'm using to make the changed public function backwards compatible). Not sure if this is intentional.

What changes are included in this PR?

This PR simply memoizes the index lookup of the fields. It should not break any existing behaviour.

It also fixes a recent regression about the behaviour of the changed function in the case of using a custom comparator and have no matches in the schema.

Are these changes tested?

All code paths changed are covered by new unit tests.

Are there any user-facing changes?

Yes, the lookup function throws an exception when there are no matches fuels rather than returning -1. This was a regression introduces O(days) ago.

• GitHub Issue: [C#] Column(string) method in RecordBatch is linear to the number of columns #44501

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #44501 has been automatically assigned in GitHub to PR creator.

[vthemelis]

@CurtHagenlocher it looks like my tests are failing in CI but I can't replicate this locally.

[vthemelis]

Oh, I see the issue. The CI does a merge into main first and this has changed since my PR: #44576

The PR above has changed the lookup behaviour when the field is not matched.

CC @georgevanburgh

[vthemelis]

@CurtHagenlocher and @georgevanburgh I've reverted to the original behaviour (throwing an InvalidOperationException) for now. Let me know if you prefer keeping the return -1 regression.

[CurtHagenlocher]

I'm a bit tied up with work but will look at this by Friday.

[vthemelis]

I'm a bit tied up with work but will look at this by Friday.

Thank you very much!

[CurtHagenlocher]

Thanks for the change! (and sorry for the delay) I left a few suggestions.

[CurtHagenlocher]

This is a breaking change. Is there a specific reason for it?

[vthemelis]

This is actually fixing a regression that was added in #44576

The function would previously throw InvalidOperationException due to .First(...). The description of the PR is erroneous in asserting that -1 is returned in the case of no match as the exception would first be thrown by the argument of the .IndexOf(...) function.

I noticed this because I wrote my unit tests before that PR was merged. Are you happy with keeping the return -1 regression?

[CurtHagenlocher]

I think for new code, we should use Ordinal. I don't think CurrentCulture really makes sense here.

[vthemelis]

If I change to Ordinal then this would be a breaking change as this is used in GetFieldIndex(string). Are you okay with changing the behaviour?

[vthemelis]

Of course, I can then only use the lookup if the user has provided Ordinal as the comparator but this would defeat the purpose of this PR which is to make lookup of Columns in RecordBatch O(1).

[vthemelis]

Or I could change this in a separate PR that implements your suggestions here #44650 (comment) ?

[CurtHagenlocher]

Can we also expose this in a way that lets users get all the indexes for a particular field name?

[vthemelis]

Sure, I'm happy to add this, though it feels like it's adding further to the scope of the PR. I'm happy to implement any API suggestions that you may have; what would be your preferred way?

[vthemelis]

Thanks for the change! (and sorry for the delay) I left a few suggestions.

Thank you very much for your review @CurtHagenlocher ! I added some replies to your comments. Let me know your thoughts and will proceed with changes.

[vthemelis]

Hi @CurtHagenlocher , do you have some time to look into this?

[vthemelis]

Hi @CurtHagenlocher, I've undone the style change as you suggested. Let me know your views on the other questions and I'm happy to change this PR accordingly.

[vthemelis]

If you'd prefer, I can arrange separate PRs for aspects suggested?

[vthemelis]

Hi @CurtHagenlocher , can you still have a look at this?

[Audrius-GR]

@CurtHagenlocher would it be possible to review this? We've hit this recently which inadvertently introduced a large slowdown in our code.

[adamreeve]

It would be nice to get this feature added to speed up field and column lookups, but from my understanding there are two main issues:

• This doesn't account for duplicate field names
• It adds more technical debt to the inconsistent string comparisons issue by continuing the use of StringComparer.CurrentCulture ([C#] Inconsistent String Comparisons in Schema Field Retrieval Methods #44650)

I think Curt's suggestion to deprecate exising APIs and introduce new better APIs (#44650 (comment)) makes sense. This would be fairly disruptive though as it would also require deprecating IArrowRecord.Column and its implementations in RecordBatch and StructArray, but that's probably still better than making a breaking behaviour change.

How about this as a potential way forward?

• Mark Schema.GetFieldIndex(string name, IEqualityComparer<string> comparer = default) and IRecordType.GetFieldIndex as obsolete
• Add a new Schema and IRecordType method that uses StringComparer.Ordinal by default. I can't think of a great name, some options might be FieldIndex, LookupFieldIndex, GetFirstFieldIndex orGetIndexForField
• Add a new IEnumerable<int> Schema.GetFieldIndices(string name, IEqualityComparer<string> comparer = default) method that also uses StringComparer.Ordinal by default to support the duplicate field names use case.
• Document all the relevant Schema methods and properties to mention what type of string comparison they do by default
• Mark RecordBatch.Column(string columnName), RecordBatch.Column(string columnName, IEqualityComparer<string> comparer) and IArrowRecord.Column(string columnName, IEqualityComparer<string> comparer = default) as obsolote.
• Add new RecordBatch and IArrowRecord methods that use the new Schema methods that default to StringComparer.Ordinal. Again I'm not sure what a good name would be, maybe LookupColumn, GetColumn or GetColumnByName
• If we're making new methods anyway, we should throw KeyNotFoundException or a new custom exception type (FieldNotFoundException?) rather than InvalidOperationException in the case a field isn't found.
• Optionally make _fieldsIndexLookup initialized lazily so it could be used by the new and old methods. It would probably make sense to support comparers other than Ordinal and CurrentCulture too. This could get complicated though, we'd need to consider thread safety and what happens if the same schema is queried with one comparer and then a different comparer. I can imagine that happening when code that uses the new and obsolete methods is mixed.

[adamreeve]

I should add that the above changes could all be done as a follow up to this PR so I think we could merge this nearly as-is to get the performance improvement, and then follow up later to fix the comparison inconsistency.

Given the breaking change to return -1 instead of throw InvalidOperationException has already been released in 19.0.0, I don't think we should break that behaviour again but should stick with returning -1 now.

[adamreeve]

Can we add a test that verifies we can use a different comparer to the default, eg. getting a field named "f0" using "F0" and StringComparer.OrdinalIgnoreCase?

[CurtHagenlocher]

Closing as the C# implementation has moved to a new repository.