Skip to content

Commit 528cc29

Browse files
committed
Handle SQLite cache errors gracefully instead of aborting upload
Summary: - Fix sqlite3.OperationalError: disk I/O error in the file handle cache from crashing the entire upload process - PersistentCache.get() now catches sqlite3.DatabaseError and returns None (cache miss) instead of propagating the exception - Cache failures are logged as warnings so they remain visible for debugging Context The upload file handle cache is a SQLite database that stores server-returned handles for already-uploaded images. It is purely an optimization to skip re-uploading — if the cache is unavailable, the server's fetch_offset endpoint confirms prior uploads at the cost of one extra HTTP round-trip per image. Previously, any SQLite error during cache lookup (disk I/O error, corrupt database, etc.) propagated up through _continue_or_fail which treated it as a fatal error, aborting the upload of all remaining images. This was observed on Windows with 4 concurrent upload workers rapidly opening/closing SQLite connections to the same cache file.
1 parent abc0056 commit 528cc29

File tree

1 file changed

+10
-1
lines changed

1 file changed

+10
-1
lines changed

mapillary_tools/history.py

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,12 +118,21 @@ def get(self, key: str) -> str | None:
118118

119119
s = time.perf_counter()
120120

121-
with store.KeyValueStore(self._file, flag="r") as db:
121+
try:
122+
db_ctx = store.KeyValueStore(self._file, flag="r")
123+
except sqlite3.DatabaseError as ex:
124+
LOG.warning(f"Failed to open cache database: {ex}")
125+
return None
126+
127+
with db_ctx as db:
122128
try:
123129
raw_payload: bytes | None = db.get(key) # data retrieved from db[key]
124130
except Exception as ex:
125131
if self._table_not_found(ex):
126132
return None
133+
if isinstance(ex, sqlite3.DatabaseError):
134+
LOG.warning(f"Cache read error for {key}: {ex}")
135+
return None
127136
raise ex
128137

129138
if raw_payload is None:

0 commit comments

Comments
 (0)