support support for ngram indexer#197
Open
jomart1985 wants to merge 270 commits intodevfrom
Open
Conversation
- Should close all but the first SkipBuffer.
Thanks to alian555.
Alain Barbet <alian@amisw.com>
Improved the README file's build instructions
Update for boost-1.5
- Thanks to kojik1010.
Commit e3f8992 ported the code to use Boost.Filesystem V3 API, so the warning about not being able to use Boost 1.50+ is no longer true.
Remove outdated note about Boost incompatibility from the README.
I've observed crashes where one thread is in the middle of initializing ZZ_CMAP and other is trying to use the partially initialized array and crashes. Use boost::once to ensure that only one thread handles the initialization and no thread uses the data until it is fully initialized.
Fix races when initializing static arrays in StandardTokenizerImpl
The argument should be named startOffset, not endOffset, otherwise the function is a no-op.
Use StringUtils::toString() before trying to concatenate or use operator<< which is more type-safe.
Fix some issues identified using clang's static analysis
Add support for compiling with -std=c++11.
Before 1.54, there was no support for varadic calls to boost:call_once(), so make the arrays static members to avoid the need to pass them to the static init methods.
Fix compatibility with Boost versions before 1.54
8628278 broke compilation due to a typo (boost:call_once instead of boost::call_once). Additionally, VC++ compilation with precompiled header was broken, because LuceneInc.h must be included as the very first header.
This commit addresses critical crashes that occur when trying to read certain Lucene index files where field data may be corrupted or have incompatible format. Root causes: 1. Assertion failures in FieldsReader::doc() when bits flag or compressed field format is incompatible with expected values 2. Null pointer dereference in boost::shared_ptr when trying to add fields to a document with potentially corrupted data The fixes include: - Replace assertions with safe handling for invalid bit flags and format incompatibilities - Add null checks before dereferencing shared pointers in critical paths - Create field objects separately before adding them to documents - Add proper error handling for decompression failures - Ensure uncompress and string conversion methods never return null values These minimal changes maintain the original logic but make the code more robust when dealing with unexpected or corrupt index data. Instead of crashing, the code now gracefully handles these edge cases and continues processing where possible.
Fix crash when reading corrupted or incompatible Lucene index files
- If `boost::filesystem::directory_entry::status` fails, return false instead of throwing a `boost::filesystem::filesystem_error` exception. - During file enumeration, if an `ENOENT` error occurs, re-enumerate all directory entries to increase robustness.
Boost.System has been header only since Boost 1.69.0
Handle file enumeration exceptions in FileUtils::listDirectory
Fix build new cmake
Fix typo in MAX_VARINT32_LENGTH constant in BufferedIndexInput.cpp
Update DefaultSimilarity.cpp
Fix old comment about C++ standard
BitSet: Partial fix for Boost 1.90
Use conditional compilation to support both old and new Boost.Bind API: - Boost >= 1.73.0: Use boost/bind/bind.hpp - Boost < 1.73.0: Use boost/bind.hpp This approach maintains backward compatibility while fixing deprecation warnings in newer Boost versions.
Use new Boost.Bind API to fix deprecation warnings
Several tests have custom mock classes. Unfortunately these frequently have identical names across tests, which creates problems when building with LTO, as everything is merged into a single test executable. GCC rightfully complains about this, since classes with the same name are assumed to have the same shape, and that is just not true here. Therefore rename the mock classes with the initials of the containing test. With this the entire test suite compiles and passes when built with LTO.
While trying to verify tests previously excluded in Gentoo (gentoo/gentoo@b9d1c7a) I noticed that ParallelMultiSearcherTest & SortTest would work, but hang in ~ThreadPool() on threadGroup.join_all(), preventing the test executable from terminating cleanly. Stopping the io_context makes join_all() work immediately.
Stop io_context before joining threads in ThreadPool destructor
Use unique class names for inner test mock classes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
i hope if you support support for ngram indexer