missing option for trapping exceptions during indexing

[satra]

i have run into several reasons why indexing a doc can fail (api error, pdfreader error, pdf error, etc.,.). however, the current CLI and functions for indexing simply error out when this happens. a more graceful strategy could be implemented, to move the skipped pdfs/docs into a list, and reporting failed instances at the end instead of erroring out.

for the moment, i'm reusing the indexing functions and adding to a list (https://github.com/sensein/paperqa-test/blob/53d5dcf6af3d44645668a01103247d6d8b0de86a/index_abcd.py#L26). but this iterative process is slowly and relies on monitoring.

[dosubot]

Paper-qa’s indexing already traps exceptions for most document failures: when a file can’t be indexed (API error, PDF parsing, etc.), it logs the error, marks the doc as failed in the index, saves progress, and continues with the rest—so a single bad file won’t crash the batch. Failed docs are tracked in the index metadata and error logs (like pqa_index/processing_errors.txt), and the CLI prints a summary at the end with counts of successful and failed files. You can also use CLI flags like --reprocess-failed to retry only the failures later. However, there isn’t a dedicated CLI flag to output a list of failed docs directly at the end, though the infrastructure is there if you want to extend the CLI to print or export that list after indexing. For details, see the core logic in process_file in paperqa/agents/search.py and related discussion.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}