Feature/phase2 doclaynet layout#15
Merged
Merged
Conversation
- _Row: final_tables -> num_candidate_tables + num_crop_tables; best_iou_final -> best_iou_candidate + best_iou_crop - crop IoU: final_crop_tables = [r for r in final_tables if r.score >= table_threshold], matches batch runner crop filter - summary: reports both candidate and crop mean IoU so threshold effect is visible - Q3 sim: fallback fires only when primary_tables >= 1 (new rule); zero-table primary pages contribute 0.0, not fallback IoU - FP rate: fp_crop uses num_crop_tables > 0, not num_candidate_tables > 0 - notebook cell 4b348fa6: sort by best_iou_crop, show both candidate + crop columns
…p debug cell - run_layout_batch.py + eval_layout_iou.py: --table-threshold default 0.5 -> 0.3 - notebook Steps 4/5/5b: drop explicit --table-threshold 0.5 (now uses new default) - Step 5c: keep --table-threshold 0.3 explicit (comparison reference) - Step 5d (new): re-runs primary detector on val_005241 with per-box IoU/GT and IoU/otherPrimary to diagnose dedup collapse; reads regions JSON + shows crops
…dup label - run_layout_batch.py: clear regions/*.json + crops/*.png at start of each run - notebook spot-check: filter crops by current manifest page_ids (no stale files) - eval_layout_iou.py: Q3 simulation header notes pre-dedup proxy limitation - notebook: Step 5e - run eval with --dedup-iou 0.7 to test val_005241 NMS collapse
…cells
- run_layout_batch.py + eval_layout_iou.py: --dedup-iou default 0.5 -> 0.7
- eval_layout_iou.py: diagnostic.csv -> diagnostic_{pos,neg}.csv (mode suffix) to avoid overwrite
- notebook: Step 6 cells (6a batch, 6b positive eval, 6c FP eval) for seed=42 n=200
- notebook cell 4b348fa6: reads diagnostic_pos.csv first
- _greedy_match(pred_boxes, gt_boxes, threshold): greedy IoU assignment - _Row: matched_50, matched_75 (GT tables matched at IoU >= 0.50/0.75) - per-page print: adds m50=X/Y - summary: table-level section with gt_total, crops, matched@0.5/0.75, recall, precision, missed, extra - notebook cell 4b348fa6: shows matched_50/75 columns + aggregate footer
- scripts/smoke_structure.py: picks --n crops from LAYOUT_OUTPUT/crops/, runs full structure pipeline (model load, inference, normalize_tatr_prediction, validate_grid_geometry), prints rows/cols/cells/valid per crop Reuses _load_model quirks from run_phase1a_colab.py: use_fast=False + size fix - notebook: Step 7 markdown + code cell (!python scripts/smoke_structure.py --n 10)
scripts/smoke_structure.py: - _ReasonCollector shim: captures validate_grid_geometry failure strings - print failure reasons inline for WARN crops - seed-based random crop sampling (--seed, default 42) instead of alpha-first - write smoke_structure.csv (crop, rows, cols, cells, valid, failure_reasons) notebooks/04_phase2_layout.ipynb: - Step 7: --n 50 --seed 42 - Step 7b cell: reads smoke_structure.csv, tallies failure reason counts src/layout_parsing.py: - DEFAULT_TABLE_SCORE=0.3, DEFAULT_TABLE_DEDUP_IOU=0.7 named constants - detect_layout defaults updated from 0.5/0.5 to match scripts
src/tatr_postprocess.py: - _dedup_bands: greedy 1-D NMS on row/col bands sorted by start coord - dedup_row_col_bands: public wrapper applies it to row_boxes + col_boxes tests/test_tatr_postprocess.py: - 9 new tests covering: no overlap, overlap score selection, chain dedup, col-axis, exact-threshold boundary, passthrough keys, valid-after-dedup scripts/smoke_structure.py: - --dedup-bands flag: applies dedup_row_col_bands before normalize_tatr_prediction notebooks/04_phase2_layout.ipynb: - Step 7c cell: --dedup-bands A/B run on same seed=42, n=50 crops
src/tatr_postprocess.py: - normalize_tatr_prediction now calls dedup_row_col_bands before building grid; row/col band NMS is the default for all callers, not opt-in - validate_grid_geometry rejects empty row_boxes or col_boxes (no-row grid fix) scripts/smoke_structure.py: - removed --dedup-bands flag (always applies dedup_row_col_bands before normalize) notebooks/04_phase2_layout.ipynb: - Step 7c: annotated as historical A/B reference (flag removed) - Step 7d: updated to run without flag; notes 285 OK / 1 WARN final result tests/test_tatr_postprocess.py: - test_validate_grid_geometry_rejects_missing_rows_or_cols
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.