Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .jules/bolt.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,6 @@
## 2025-02-12 - [Fast Path Execution in Directory Traversal and Log Parsing]
**Learning:** Checking for string existence (`if "silence_" not in stderr`) before invoking regex matchers provides significant speed improvements when parsing large blocks of text. Similarly, moving expensive I/O operations like `os.path.realpath` inside conditional blocks prevents redundant disk access when configuration (like path exclusions) isn't utilized.
**Action:** When working on large text processing or disk operations, verify if early exit conditions or conditional execution can bypass the expensive system or library calls.
## 2023-11-20 - [SQLite job store query performance bottleneck]
**Learning:** In a single-file SQLite database used as a job queue with a WAL journal mode, relying on full table scans for frequent `list_jobs` polling causes a measurable CPU and memory sorting (filesort) bottleneck. SQLite can quickly evaluate queries locally, but as row count scales to tens of thousands, scanning without an index becomes an issue.
**Action:** When implementing an SQLite table meant to be polled or queried sequentially (e.g., a job table with status and timestamps), always define composite indexes corresponding to the `ORDER BY` and `WHERE` clauses from the start. Remember to use `conn.executescript()` instead of `conn.execute()` when initializing schemas with multiple statements.
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@
### Added
- 다중 파일 업로드 선택 시 즉각적인 파일 개수 피드백 및 제한 초과 경고 메시지 추가
- 일괄 업로드 폼에 대상 바이트 프리셋 버튼과 총 파일 크기 미리보기를 추가하여 사용성을 개선했습니다.

- `job_store.py` 데이터베이스 테이블에 조회 최적화를 위한 복합 인덱스를 추가하여 `list_jobs` 쿼리 성능 개선 (Bolt)
9 changes: 7 additions & 2 deletions job_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,12 @@
output_name TEXT,
error TEXT,
temp_dir TEXT
)
);
-- Bolt ⚡: Optimization for list_jobs queries.
-- Composite indexes drastically speed up queries by preventing full table scans and memory file sorts
-- when querying and ordering by status and created_at.
CREATE INDEX IF NOT EXISTS idx_jobs_status_created_id ON jobs(status, created_at, id);
CREATE INDEX IF NOT EXISTS idx_jobs_created_id ON jobs(created_at, id);
"""

_COLUMNS = (
Expand Down Expand Up @@ -95,7 +100,7 @@ def __init__(self, db_path: str) -> None:
self._db_path = str(db_path)
self._lock = threading.Lock()
with self._connect() as conn:
conn.execute(_SCHEMA)
conn.executescript(_SCHEMA)

@contextmanager
def _connect(self) -> Iterator[sqlite3.Connection]:
Expand Down