diff --git a/android/cartfollow-devlog.md b/android/cartfollow-devlog.md index 5549681e4..71340bac0 100644 --- a/android/cartfollow-devlog.md +++ b/android/cartfollow-devlog.md @@ -3,7 +3,7 @@ > 所属项目:自主跟随购物车原型 > 代码位置:`dev/OpenBot/android/robot/src/main/java/org/openbot/cartfollow/` > 开发分支:`feature/human-cart-simulator`(Phase 1)/ `feature/distance-control`(Phase 2)/ `feature/person-crop-collector`(Phase 3 起) -> 最后更新:2026-07-07 +> 最后更新:2026-07-08 --- @@ -13,19 +13,20 @@ Human Cart Simulator 是购物车跟随功能的上位机核心模块,在 Open ### 文件清单 -| 文件 | 行数 | 作用 | -|------|------|------| -| `HumanCartSimulatorFragment.java` | 464 | 主 UI Fragment:摄像头预览 + 检测框绘制 + 确认/重拍/取消 + 倒计时 + 距离调试显示 | -| `ControlGenerator.java` | 110 | 控制算法:基于 DistanceState 决定 forward,转向由 xError 决定 | -| `FollowStateMachine.java` | 262 | 完整状态机:管理两阶段目标初始化、重识别、倒计时、跟随、丢失、搜索、停止 | -| `FollowState.java` | 14 | 状态机枚举:`IDLE / CAPTURE_TARGET / LOCKED_PENDING_CONFIRM / CONFIRMED_ARMED / REACQUIRE_TARGET / READY_TO_FOLLOW / FOLLOW / LOST / SEARCH / STOP` | -| `DistanceState.java` | 7 | 距离状态枚举:`TOO_FAR / OK / TOO_CLOSE / UNKNOWN` | -| `ImageSetpointDistanceEstimator.java` | 112 | 图像伺服距离估计器:基于初始化 setpoint 输出 height_scale / area_scale / bottom_shift / state / confidence | -| `TargetMemory.java` | 167 | 目标记忆:confirmedBbox、面积、上下身 HSV 颜色直方图、动态位置、距离 setpoint | -| `TargetMatcher.java` | 79 | 目标匹配:position + size + color + confidence 融合评分(ReID 接口预留) | -| `ReIDFeatureExtractor.java` | 5 | ReID 接口占位,尚未接入真实 embedding 推理 | -| `HumanCommandInterpreter.java` | 52 | 中文指令解释器:支持 DistanceState 重载,输出距离感知指令 | -| `fragment_human_cart_simulator.xml` | 218 | 布局文件:OverlayView + 指令文本 + 快照确认面板 + 倒计时 + 调试信息 + 底部面板 | +| 文件 | 作用 | +|------|------| +| `HumanCartSimulatorFragment.java` | 主 UI Fragment:摄像头预览、检测框绘制、目标确认、ReID 调度、行为决策 debug 与人类指令显示 | +| `FollowStateMachine.java` / `FollowState.java` | 管理初始化、确认、重捕获、跟随、谨慎跟随、身份不确定、搜索与停止 | +| `ActionArbitrator.java` / `BehaviorAction.java` / `BehaviorDecisionResult.java` | 阶段 A 行为层:将状态与证据映射为 `FOLLOW_SLOW / MOTION_STOP / LOCAL_SEARCH / BLOCKED_WAIT` 等动作 | +| `IdentityEvidence.java` / `DistanceEvidence.java` / `TraversabilityEvidence.java` / `SystemSafetyEvidence.java` | 统一证据结构,供 ActionArbitrator 和 debug 面板使用 | +| `ControlGenerator.java` | 控制算法:基于 DistanceState 决定 forward,转向由 xError 决定;当前仍只用于模拟提示,不直接控制底盘 | +| `DistanceState.java` / `ImageSetpointDistanceEstimator.java` | 初始化标定 + 图像伺服距离估计,输出 `TOO_FAR / OK / TOO_CLOSE / UNKNOWN` | +| `TargetMemory.java` | 目标记忆:confirmed bbox、颜色特征、距离 setpoint、previous/last bbox 和 ReID gallery | +| `TargetMatcher.java` | legacy 目标匹配:position + size + color + confidence,用于 ReID 不可用时的保守降级 | +| `ReIDFeatureExtractor.java` / `TfliteReIDFeatureExtractor.java` | ReID 抽象接口与 TFLite 推理实现,当前本地测试模型为 `osnet_x0_25_market1501.tflite` | +| `ReIDCoordinator.java` / `ReIDMatchResult.java` / `BboxContinuityEvidence.java` | 管理 ReID gallery、候选人推理、best/second/margin、bbox 连续性与推理耗时 | +| `HumanCommandInterpreter.java` | 将状态、距离和行为动作转换为 Human Cart Simulator 的中文动作提示 | +| `fragment_human_cart_simulator.xml` | 布局文件:OverlayView、指令文本、快照确认面板、倒计时、调试信息和底部面板 | ### 集成点(在 OpenBot App 中的入口) @@ -83,6 +84,10 @@ Human Cart Simulator 是购物车跟随功能的上位机核心模块,在 Open | 导航集成 | 已完成 | 已注册到主菜单 "Cart Simulator" 入口 | | **Person Crop Collector** | 已完成(Phase 3 前置) | 已注册到主菜单,可采集真实 OpenBot person bbox crop、`session_info.json` 与 `metadata.csv` | | **真实 crop 数据 PC 端 ReID 复测** | 已完成首轮 | 基于 `images_openbot_clean`、`osnet_x0_25_market1501.pth`、`diverse gallery` 完成 pairwise / gallery-probe / target-follow 模拟 | +| **Person Sequence Collector** | 已完成 | 可采集无人帧、多人检测、bbox、crop 和人工事件,用于 PC sequence replay | +| **阶段 A 行为层** | 已完成并通过手机体验 | `Evidence -> BehaviorDecisionResult -> BehaviorAction -> HumanCommand` 已接入 Human Cart Simulator | +| **阶段 B Android ReID** | 已完成首版并通过手机运行 | TFLite ReID 可运行,debug 字段正常,实机约 30 FPS;仍需阶段 C 轨迹与身份信念层抑制跟错人 | +| **阶段 C 目标轨迹与身份信念层** | 已完成首版代码接入 | 新增短时 trackId、lockedTrackId、targetBelief 和 suspectedTrack debug,待手机验收调参 | ### 2.2 Phase 3 首轮 ReID 实验结果(2026-07-06) @@ -173,7 +178,7 @@ ReID margin 可作为身份置信证据,但不能单独恢复 FOLLOW。 | 功能 | 优先级 | 说明 | |------|--------|------| | **`vehicle.setControl()` 集成** | 中(阶段6) | 当前 Control 仅显示在 UI 上,未实际发送给底盘。硬件联调阶段在 `processFrame()` 中调用 `vehicle.setControl()` | -| **目标重锁定增强** | 高 | 当前 LOST/SEARCH 恢复复用 TargetMatcher,尚未接入 Android 端 ReID embedding 与多线索融合。阶段3 继续处理 | +| **目标轨迹与身份信念层** | 高 | ReID 已接入并能运行,但目标离开/返回和干扰者场景仍可能跟错或恢复过慢;下一阶段需要 `TargetTrackManager + IdentityBeliefAccumulator` | | **参数持久化** | 低 | 当前调参仅内存生效,重启恢复默认 | | **参数 UI 面板** | 低 | K_TURN / MAX_FORWARD / 阈值等参数需通过代码修改,没有 UI 界面 | | **bottomShift 参与判态** | 低 | 当前 bottomShift 仅用于显示,未参与距离状态判断。待 90° 旋转下方向实测验证后决定是否纳入 | @@ -250,21 +255,38 @@ STOP ──(用户重新开始)──→ CAPTURE_TARGET - [x] 新增 Person Crop Collector,从 OpenBot Android 导出真实 person bbox crop - [x] PC 端验证 confirmedGallery / reid_score / reid_margin(首轮基于 `osnet_x0_25 + diverse gallery-k=8`) -- [ ] Android 端部署 osnet_x0_25(ONNX Runtime Mobile 主线) -- [ ] 将 ReID 输出接入 `TargetMemory / TargetMatcher / FollowStateMachine` -- [ ] 多人/遮挡/重现场景验证,并特别验证目标缺席时的 false accept 抑制 +- [x] Android 端部署 `osnet_x0_25` TFLite 首版,Human Cart Simulator 中 `reidAvailable=true` +- [x] 将 ReID 输出接入 `IdentityEvidence / FollowStateMachine / ActionArbitrator` +- [x] 多人、目标离开、目标返回、遮挡场景完成首轮手机观察 +- [ ] 用阶段 C 的 track/belief 层继续降低跟错人风险,提高目标返回后的恢复速度 + +### Phase 4:目标轨迹与身份信念层(当前下一步) -### Phase 4:单目深度辅助 +- [x] 新增轻量 `TargetTrackManager`,用 bbox IoU / center distance / area ratio 维护短时 trackId +- [x] 新增 `IdentityBeliefAccumulator`,对每个 track 累计 targetBelief +- [x] locked target 不因干扰者单帧 ReID 高分被抢走,状态机恢复改为 belief 优先 +- [x] 目标返回后通过 suspected track + 多帧 belief 稳定恢复到 `REACQUIRE_TARGET / FOLLOW_CAUTION` +- [x] debug 面板显示 `trackId / lockedTrackId / targetBelief / trackAge / missedFrames / beliefReason` +- [ ] 手机实测验收:目标离开、干扰者进入、目标返回、目标在场干扰者穿越、遮挡 -- [ ] MiDaS / Depth Anything Android 部署调研 -- [ ] 作为距离/障碍风险辅助,不替代安全状态机 +### Phase 5:距离控制继续收敛 -### Phase 5:局部可通行空间与跟随式避障 +- [x] 初始化距离标定 + 图像伺服首版已完成 +- [ ] 结合阶段 C 的稳定目标 track 重新验证 `TOO_FAR / OK / TOO_CLOSE` +- [ ] 评估 bottomShift 是否纳入距离状态判断 + +### Phase 6:局部可通行空间与跟随式避障 - [ ] LEFT / CENTER / RIGHT 三方向 free score - [ ] 候选动作 SLOW_FORWARD / LEFT_ARC / RIGHT_ARC / BLOCKED_WAIT -### Phase 6:硬件联调 +### Phase 7:性能评估与增强开关 + +- [ ] 记录 detector、ReID、track/belief、ActionArbitrator 耗时 +- [ ] 根据手机性能决定是否提高 ReID 频率 +- [ ] 评估 MiDaS / Depth Anything Android 部署,只作为距离/障碍风险增强 + +### Phase 8:硬件联调 - [ ] 接通 `vehicle.setControl()` 到底盘 - [ ] 真实车速/转向半径/延迟标定 @@ -288,7 +310,10 @@ STOP ──(用户重新开始)──→ CAPTURE_TARGET | `6d9aa5f` | 2026-07-06 | Add capture controls and status panel | | `765eb82` | 2026-07-06 | Put Person ID input on its own row for tap accessibility | | `771345e` | 2026-07-06 | Rotate crop by sensorOrientation before saving to disk | -| pending | 2026-07-07 | Add PersonSequenceCollector for continuous sequence data collection | +| recorded | 2026-07-07 | Add PersonSequenceCollector for continuous sequence data collection | +| recorded | 2026-07-08 | Add phase A behavior decision layer and Human Cart Simulator action debug | +| recorded | 2026-07-08 | Add phase B TFLite ReID evidence path for Human Cart Simulator | +| pending | 2026-07-08 | Add phase C TargetTrack and IdentityBelief layer | --- @@ -445,3 +470,295 @@ STOP - 采集目录导出到 PC 后,用 `tools/reid_pc_test/prepare_openbot_crops_dataset.py` 整理为 `images_openbot_clean/` - 主菜单 → "Person Sequence Collector" 进入连续时序采集页 - Sequence 采集时事件按钮只需在事件开始/结束时各按一次,PC replay 会用容忍窗口处理人工反应延迟 + +--- + +## 9. Phase B:ReID 身份证据接入与安全重捕获闭环(2026-07-08) + +本阶段根据 Human Cart Simulator 阶段 A 的手机验收反馈推进:阶段 A 的行为层基本可用,但目标重新进入、干扰者进入等场景仍可能因为旧 `TargetMatcher` 单帧匹配而出现“跟错人”。因此 Phase B 的首要目标是切断 `LOST / SEARCH -> FOLLOW` 的单帧恢复路径,并把 ReID 作为身份置信度证据接入状态机。 + +已完成代码改动: + +- 新增 `ReIDMatchResult`、`BboxContinuityEvidence`、`TfliteReIDFeatureExtractor`、`ReIDCoordinator`。 +- `FollowState` 新增 `FOLLOW_CAUTION` 与 `IDENTITY_UNCERTAIN`。 +- `TargetMemory` 增加 previous bbox 记录,用于 bbox 连续性和简单 prediction 计算。 +- `IdentityEvidence` 扩展为同时携带 legacy score、ReID score / margin、bbox gate、稳定帧数和候选切换次数。 +- `FollowStateMachine` 改为支持外部 `IdentityEvidence` 输入;`LOST / SEARCH` 不再允许单帧匹配直接恢复 `FOLLOW`,而是先进入 `REACQUIRE_TARGET`,再经多帧稳定证据恢复。 +- `ActionArbitrator` 增加 `IDENTITY_UNCERTAIN` 和 `FOLLOW_CAUTION` 的动作解释。 +- `HumanCartSimulatorFragment` 接入 `ReIDCoordinator`,在 debug 面板显示 `reidAvailable / gallerySize / bestScore / secondScore / margin / weak-mid-strong / bboxDefault-bboxStrict-prediction / stableMatchCount / candidateSwitchCount / reidLatencyMs / reidReason`。 + +TFLite 路线说明: + +- 首版复用当前工程已有 TensorFlow Lite 2.4,不新增 ONNX Runtime 依赖。 +- 默认模型路径为 `assets/networks/reid/osnet_x0_25_market1501.tflite`。 +- 该模型文件属于本地测试资产,`.gitignore` 已忽略 `*.tflite`,默认不提交。 +- 如果模型不存在或加载失败,App 不崩溃,debug 显示 `reidAvailable=false`,状态机退回更保守的 bbox / color / motion 逻辑。 + +构建验证: + +```powershell +$env:JAVA_HOME='D:\Java\jdk-17' +$env:Path="$env:JAVA_HOME\bin;$env:Path" +.\gradlew.bat :robot:assembleDebug +``` + +结果:构建通过,生成 `robot/build/outputs/apk/debug/robot-debug.apk`。当前本机输出过 Android SDK XML version warning,但未影响构建结果。 + +后续手机验收重点: + +- 无 ReID 模型时:Human Cart Simulator 应正常打开,debug 显示 `reidAvailable=false`,目标丢失后不应单帧恢复 `FOLLOW`。 +- 放入 TFLite 模型后:确认目标后 `gallerySize` 应逐步达到 8 或实际可用数量;多人、目标离开、目标返回、遮挡场景下观察 `bestScore / margin / stableMatchCount` 是否符合预期。 +- 目标返回时允许 `IDENTITY_UNCERTAIN -> REACQUIRE_TARGET -> FOLLOW`,但不允许单帧高分直接恢复 `FOLLOW`。 + +--- + +## 10. Phase B 实机验收更新:ReID 已跑通,但需要 TargetTrack / IdentityBelief(2026-07-08) + +### 10.1 当前实机状态 + +最新 APK 已安装到手机,Human Cart Simulator 中 ReID 首版链路已经成功运行: + +- TFLite 模型路径:`assets/networks/reid/osnet_x0_25_market1501.tflite` +- 模型输入:`[1,3,256,128]` +- 模型输出:`[1,512]` +- debug 面板显示的 ReID 字段基本正常 +- 实机帧率约 30 FPS,首轮性能可接受,后续如有必要仍可继续压榨调度策略 + +当前结论:ReID Android 接入已经从“能否加载/能否推理”进入“如何安全使用推理结果”的阶段。 + +### 10.2 当前体验问题 + +手机实测仍暴露两个关键问题: + +1. 目标离开画面或多人接近时,虽然统计上比纯 bbox / color 匹配更安全,但仍可能识别错目标并跟着别人走。 +2. 目标重新回到画面后,重捕获有时偏慢,甚至无法及时确认。 + +这说明当前 `ReIDMatchResult + bbox gate + stable frame` 仍偏“单帧候选驱动”。ReID 分数和 margin 有帮助,但不能直接等价于目标身份。 + +### 10.3 下一步代码方向 + +下一步建议在 `ReIDCoordinator` 与 `FollowStateMachine` 之间新增轻量轨迹和身份信念层: + +```text +Detector persons + -> TargetTrackManager + -> IdentityBeliefAccumulator + -> FollowStateMachine + -> ActionArbitrator +``` + +建议新增或细化类型: + +- `TargetTrack` + - `trackId` + - `lastBbox` + - `previousBbox` + - `ageFrames` + - `missedFrames` + - `lastSeenTimestampMs` + - `isNearPredictionRegion` + +- `TargetTrackManager` + - 用 bbox IoU、center distance、area ratio 做短时 person bbox 关联。 + - 只维护最近几秒的轻量 track,不做复杂多目标跟踪。 + - 输出当前候选 tracks,供 ReID 和状态机使用。 + +- `IdentityBelief` + - `trackId` + - `targetBelief` + - `reidContribution` + - `bboxContribution` + - `predictionContribution` + - `switchPenalty` + - `stableFrames` + - `beliefReason` + +- `IdentityBeliefAccumulator` + - 对每个 track 累计身份信念。 + - ReID strong/mid、bbox 连续、prediction 命中时加分。 + - 候选切换、bbox 跳变、面积突变、margin 低、多人与目标混淆时扣分。 + +### 10.4 状态机接入原则 + +后续恢复跟随不应再由“单帧候选高分”触发,而应由“稳定 track + 稳定 belief”触发: + +```text +IDENTITY_UNCERTAIN / SEARCH + -> 疑似目标 track 连续稳定 + -> REACQUIRE_TARGET + -> FOLLOW_CAUTION + -> FOLLOW_CONFIDENT +``` + +安全边界保持不变: + +- 身份不确定时线速度为 0。 +- 搜索阶段可以更积极地原地扫描和提高 ReID 频率,但不能向前跟随。 +- 已锁定目标时,干扰者一帧 ReID 高分不能直接抢走目标。 +- hard `STOP` 仍只作为搜索失败、风险过高或安全异常后的兜底终态。 + +### 10.5 下一轮手机验收新增观察项 + +debug 面板建议新增: + +```text +trackId +trackAge +missedFrames +targetBelief +beliefReason +activeTrackCount +lockedTrackId +``` + +验收场景: + +1. 单人正常跟随:trackId 应稳定,targetBelief 应逐步升高。 +2. 目标离开:进入 motion_stop / search,lockedTrack 不应被立即替换。 +3. 干扰者进入:干扰者可形成新 track,但 targetBelief 不应快速超过恢复阈值。 +4. 目标返回:先成为疑似目标,连续稳定后恢复 `REACQUIRE_TARGET -> FOLLOW_CAUTION / FOLLOW_CONFIDENT`。 +5. 目标在场且干扰者穿越:lockedTrack 应尽量保持,必要时进入 `FOLLOW_CAUTION / IDENTITY_UNCERTAIN`,不冒进切换。 + +--- + +## 11. Phase C:目标轨迹与身份信念层首版实现(2026-07-08) + +### 11.1 新增代码 + +| 文件 | 作用 | +|------|------| +| `TargetTrack.java` | 记录短时 track 的 `trackId / lastBbox / previousBbox / ageFrames / missedFrames / stableFrames`。 | +| `TargetTrackManager.java` | 用 bbox IoU、中心距离、面积比例将连续检测框关联为 track,并维护 `lockedTrackId / suspectedTrackId`。 | +| `IdentityBelief.java` | 定义 `BELIEF_CONFIRM=0.75 / BELIEF_CAUTION=0.55 / BELIEF_LOST=0.30` 和 belief debug 字段。 | +| `IdentityBeliefAccumulator.java` | 融合 ReID、bbox continuity、prediction、locked target、track age、candidate switch 和 missed frame,输出带 belief 的 `IdentityEvidence`。 | + +### 11.2 接入点 + +- `HumanCartSimulatorFragment` 每帧检测后先调用 `TargetTrackManager.update()`。 +- ReID 单帧输出不再直接交给状态机,而是先通过 `IdentityBeliefAccumulator.update()` 转换为累计身份信念。 +- 用户点击 Confirm 时调用 `lockClosest(memory.getLastBbox())`,建立 `lockedTrackId`。 +- overlay 现在显示 `T b=`;locked track 绿色,suspected track 黄色。 +- debug 面板新增 `activeTrackCount / trackId / lockedTrackId / suspectedTrackId / trackAge / missedFrames / belief / beliefReason`。 +- `FollowStateMachine` 在 `IdentityEvidence.hasBelief()` 时优先使用 `targetBelief + beliefStableFrames + bbox/prediction/lockedTrack` 判断 `FOLLOW / FOLLOW_CAUTION / REACQUIRE_TARGET / IDENTITY_UNCERTAIN`。 + +### 11.3 构建验证 + +构建命令: + +```powershell +$env:JAVA_HOME='D:\Java\jdk-17' +$env:Path="$env:JAVA_HOME\bin;$env:Path" +.\gradlew.bat :robot:assembleDebug +``` + +结果:构建通过,生成 debug APK。构建日志仍有 TensorFlow Lite manifest namespace warning 和 Kotlin/Javac target warning,均未阻塞构建。 + +### 11.4 手机验收重点 + +1. 单人正常跟随:`trackId` 应稳定,`targetBelief` 应逐步升高并保持。 +2. 目标离开画面:动作应进入 motion stop / local search,不继续前进。 +3. 目标离开后干扰者进入:干扰者可形成新 track,但不应快速获得恢复 `FOLLOW` 的 belief。 +4. 目标返回:应先成为 suspected track,经多帧 belief 稳定后恢复到 `REACQUIRE_TARGET / FOLLOW_CAUTION`。 +5. 目标在场时干扰者穿越:locked track 不应被一帧高 ReID 分数抢走;必要时进入 `FOLLOW_CAUTION / IDENTITY_UNCERTAIN`。 + +--- + +## 12. Phase C 诊断采集与 UI 简化(2026-07-08) + +### 12.1 实现目的 + +阶段 C 首轮实机验收后,仍观察到两类问题: + +- 目标回到画面后长期停留在黄框,迟迟不转为绿框; +- 非目标人物偶发转绿,表现为疑似跟错人。 + +本轮没有继续修改 ReID、belief、relock 或状态机阈值,而是先在 Human Cart Simulator 中加入诊断采集能力,用真实日志解释问题发生在哪个环节。 + +### 12.2 新增代码 + +| 文件 | 作用 | +|------|------| +| `cartfollow/diagnostics/CartFollowDiagnosticConfig.java` | 管理 frame log、crop、overlay 采样间隔和 JPEG 参数。 | +| `cartfollow/diagnostics/CartFollowDiagnosticSession.java` | 创建 `cartfollow_diagnostics//`,初始化 CSV、gallery/crops/overlays 目录和 `session_info.json`。 | +| `cartfollow/diagnostics/CartFollowDiagnosticSaver.java` | 单线程异步写 `frame_log.csv`、`identity_log.csv`、`events.csv`,并低频保存 locked/suspected/best_reid crop 与初始化 gallery snapshot。 | +| `HumanCartSimulatorFragment.java` | 接入诊断 session 生命周期、人工事件按钮、低频日志保存和简洁/完整 debug 切换。 | +| `fragment_human_cart_simulator.xml` | 新增 `调试详情` 按钮和 `目标离开画面 / 目标回到画面` 事件按钮。 | + +### 12.3 输出目录与文件 + +输出位置: + +```text +/sdcard/Android/data/org.openbot/files/Pictures/cartfollow_diagnostics/ +└── cart_diag_/ + ├── frame_log.csv + ├── identity_log.csv + ├── events.csv + ├── session_info.json + ├── crops/ + ├── gallery/ + └── overlays/ +``` + +`frame_log.csv` 记录每 200 ms 左右的状态机、行为动作、人类指令和人数: + +```text +session_id,frame_id,timestamp_ms,elapsed_ms,fps,num_persons, +follow_state,selected_action,action_reason,safety_block_reason,command_text +``` + +`identity_log.csv` 记录每 200 ms 左右的 track、ReID、bbox gate、belief 和 crop 路径: + +```text +session_id,frame_id,timestamp_ms, +track_id,locked_track_id,suspected_track_id,active_track_count, +track_age,missed_frames,best_score,second_score,margin,gallery_size, +weak_ok,mid_ok,strong_ok,bbox_default_ok,bbox_strict_ok,prediction_ok, +target_belief,belief_stable_frames,belief_uncertain_frames, +candidate_switch_count,belief_reason,reid_reason, +locked_crop_path,suspected_crop_path,best_reid_crop_path +``` + +`events.csv` 记录人工事件: + +```text +session_id,timestamp_ms,frame_id,event_type,note +``` + +当前人工事件只包括: + +- `target_left` +- `target_return` +- `session_stop` + +### 12.4 UI 行为 + +- 默认左上角只显示简洁 debug:`fps / state / action / persons / track / locked / suspected / belief / best / margin`。 +- 点击 `调试详情` 后显示原完整 debug,再次点击 `收起详情` 回到简洁显示。 +- `目标离开画面` 按钮在目标确认前禁用。 +- 用户点击 `确认` 后诊断 session 正式启用,事件按钮可用。 +- 第一次点击事件按钮写入 `target_left`,按钮文本切换为 `目标回到画面`。 +- 第二次点击写入 `target_return`,按钮文本切回 `目标离开画面`。 +- 该按钮只写日志,不改变状态机、ReID、track、belief 或 action。 + +### 12.5 构建验证 + +构建命令: + +```powershell +$env:JAVA_HOME='D:\Java\jdk-17' +$env:Path="$env:JAVA_HOME\bin;$env:Path" +.\gradlew.bat :robot:assembleDebug +``` + +结果:构建通过,生成 debug APK。构建日志仍包含 TensorFlow Lite manifest namespace warning、Kotlin/Javac target warning 和若干 deprecated API warning,均未阻塞构建。 + +### 12.6 手机验收重点 + +1. Start 后、确认目标前,事件按钮应禁用。 +2. 确认目标后,事件按钮启用,默认显示 `目标离开画面`。 +3. 目标离开时点击一次,`events.csv` 应出现 `target_left`。 +4. 目标返回时再点击一次,`events.csv` 应出现 `target_return`。 +5. 默认左上角不再显示大块完整 debug;点击 `调试详情` 后可展开完整字段。 +6. Stop / Cancel / Retake / 页面暂停后,诊断 session 应关闭,按钮禁用并复位。 +7. 导出诊断目录后,应能用 `events.csv` 附近的 `frame_log.csv` 和 `identity_log.csv` 判断黄框不转绿或非目标转绿的原因。 diff --git a/android/robot/src/main/java/org/openbot/cartfollow/ActionArbitrator.java b/android/robot/src/main/java/org/openbot/cartfollow/ActionArbitrator.java new file mode 100644 index 000000000..0d61b338e --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/ActionArbitrator.java @@ -0,0 +1,100 @@ +package org.openbot.cartfollow; + +import android.graphics.RectF; + +public class ActionArbitrator { + + public BehaviorDecisionResult decide( + FollowState state, + IdentityEvidence identity, + DistanceEvidence distance, + TraversabilityEvidence traversability, + SystemSafetyEvidence safety, + TargetMemory memory, + int frameW) { + if (safety != null && safety.emergencyStop) { + return result(state, BehaviorAction.EMERGENCY_STOP, "emergency_stop", safety.reason, 0f); + } + if (safety != null && (!safety.communicationOk || !safety.detectorOk)) { + return result(state, BehaviorAction.EMERGENCY_STOP, "system_not_ready", safety.reason, 0f); + } + if (state == FollowState.STOP) { + return result(state, BehaviorAction.HARD_STOP, "state_stop", "hard_stop_state", 0f); + } + if (state == FollowState.REACQUIRE_TARGET) { + float conf = identity == null ? 0f : identity.confidence; + return result(state, BehaviorAction.REACQUIRE_HOLD, "reacquire_confirming", null, conf); + } + if (state == FollowState.IDENTITY_UNCERTAIN) { + float conf = identity == null ? 0f : identity.confidence; + return result(state, BehaviorAction.MOTION_STOP, "identity_uncertain", "motion_stop", conf); + } + if (state == FollowState.LOCKED_PENDING_CONFIRM + || state == FollowState.CONFIRMED_ARMED + || state == FollowState.READY_TO_FOLLOW + || state == FollowState.CAPTURE_TARGET + || state == FollowState.IDLE) { + return result(state, BehaviorAction.MOTION_STOP, "not_ready_to_follow", "motion_stop", 0f); + } + if (state == FollowState.LOST || state == FollowState.SEARCH) { + BehaviorAction searchAction = searchAction(memory, frameW); + if (searchAction == BehaviorAction.MOTION_STOP) { + return result(state, searchAction, "target_lost_no_last_bbox", "motion_stop", 0f); + } + return result(state, searchAction, "target_lost_last_side", null, 0.2f); + } + if (identity != null && !identity.matched) { + return result(state, BehaviorAction.MOTION_STOP, "identity_unmatched", identity.reason, 0f); + } + if (traversability != null && traversability.centerBlocked) { + return result(state, BehaviorAction.BLOCKED_WAIT, "center_blocked", traversability.reason, 0f); + } + if (distance != null + && (distance.state == DistanceState.UNKNOWN || distance.state == DistanceState.TOO_CLOSE)) { + return result( + state, + BehaviorAction.MOTION_STOP, + "distance_" + distance.state.name().toLowerCase(), + distance.reason, + distance.confidence); + } + if ((state == FollowState.FOLLOW || state == FollowState.FOLLOW_CAUTION) + && distance != null + && distance.state == DistanceState.OK) { + return result( + state, + BehaviorAction.FOLLOW_CAUTION, + state == FollowState.FOLLOW_CAUTION ? "follow_caution_distance_ok" : "identity_ok_distance_ok", + null, + identityConfidence(identity)); + } + return result( + state, + BehaviorAction.FOLLOW_SLOW, + "identity_ok_follow_allowed", + null, + identityConfidence(identity)); + } + + private static BehaviorDecisionResult result( + FollowState state, + BehaviorAction action, + String actionReason, + String safetyBlockReason, + float confidence) { + return new BehaviorDecisionResult(state, action, actionReason, safetyBlockReason, confidence); + } + + private static float identityConfidence(IdentityEvidence identity) { + return identity == null ? 0f : identity.confidence; + } + + private static BehaviorAction searchAction(TargetMemory memory, int frameW) { + if (memory == null || frameW <= 0) return BehaviorAction.MOTION_STOP; + RectF lastBbox = memory.getLastBbox(); + if (lastBbox == null) return BehaviorAction.MOTION_STOP; + return lastBbox.centerX() < frameW / 2f + ? BehaviorAction.LOCAL_SEARCH_LEFT + : BehaviorAction.LOCAL_SEARCH_RIGHT; + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/BboxContinuityEvidence.java b/android/robot/src/main/java/org/openbot/cartfollow/BboxContinuityEvidence.java new file mode 100644 index 000000000..219600f55 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/BboxContinuityEvidence.java @@ -0,0 +1,79 @@ +package org.openbot.cartfollow; + +import android.graphics.RectF; + +public class BboxContinuityEvidence { + public static final float DEFAULT_CENTER_MAX = 0.25f; + public static final float DEFAULT_X_MAX = 0.25f; + public static final float DEFAULT_AREA_MIN = 0.50f; + public static final float DEFAULT_AREA_MAX = 2.00f; + public static final float STRICT_CENTER_MAX = 0.18f; + public static final float STRICT_X_MAX = 0.18f; + public static final float STRICT_AREA_MIN = 0.60f; + public static final float STRICT_AREA_MAX = 1.67f; + + public final float centerJumpRatio; + public final float xJumpRatio; + public final float areaRatio; + public final float predictionError; + public final boolean bboxDefaultOk; + public final boolean bboxStrictOk; + public final boolean predictionOk; + public final String reason; + + public BboxContinuityEvidence( + float centerJumpRatio, + float xJumpRatio, + float areaRatio, + float predictionError, + String reason) { + this.centerJumpRatio = centerJumpRatio; + this.xJumpRatio = xJumpRatio; + this.areaRatio = areaRatio; + this.predictionError = predictionError; + this.bboxDefaultOk = + centerJumpRatio <= DEFAULT_CENTER_MAX + && xJumpRatio <= DEFAULT_X_MAX + && areaRatio >= DEFAULT_AREA_MIN + && areaRatio <= DEFAULT_AREA_MAX; + this.bboxStrictOk = + centerJumpRatio <= STRICT_CENTER_MAX + && xJumpRatio <= STRICT_X_MAX + && areaRatio >= STRICT_AREA_MIN + && areaRatio <= STRICT_AREA_MAX; + this.predictionOk = predictionError >= 0f && predictionError <= STRICT_CENTER_MAX; + this.reason = reason; + } + + public static BboxContinuityEvidence unavailable(String reason) { + return new BboxContinuityEvidence(1f, 1f, 0f, -1f, reason); + } + + public static BboxContinuityEvidence from( + RectF candidate, RectF last, RectF previous, int frameW, int frameH) { + if (candidate == null || last == null || frameW <= 0 || frameH <= 0) { + return unavailable("bbox_reference_not_available"); + } + float diag = (float) Math.hypot(frameW, frameH); + float centerJump = + (float) + (Math.hypot(candidate.centerX() - last.centerX(), candidate.centerY() - last.centerY()) + / Math.max(1f, diag)); + float xJump = Math.abs(candidate.centerX() - last.centerX()) / Math.max(1f, frameW); + float areaRatio = area(candidate) / Math.max(1f, area(last)); + float predictionError = -1f; + if (previous != null) { + float predX = last.centerX() + (last.centerX() - previous.centerX()); + float predY = last.centerY() + (last.centerY() - previous.centerY()); + predictionError = + (float) + (Math.hypot(candidate.centerX() - predX, candidate.centerY() - predY) + / Math.max(1f, diag)); + } + return new BboxContinuityEvidence(centerJump, xJump, areaRatio, predictionError, "ok"); + } + + private static float area(RectF b) { + return Math.max(0f, b.width()) * Math.max(0f, b.height()); + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/BehaviorAction.java b/android/robot/src/main/java/org/openbot/cartfollow/BehaviorAction.java new file mode 100644 index 000000000..efe1f3416 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/BehaviorAction.java @@ -0,0 +1,13 @@ +package org.openbot.cartfollow; + +public enum BehaviorAction { + FOLLOW_SLOW, + FOLLOW_CAUTION, + MOTION_STOP, + LOCAL_SEARCH_LEFT, + LOCAL_SEARCH_RIGHT, + BLOCKED_WAIT, + REACQUIRE_HOLD, + HARD_STOP, + EMERGENCY_STOP +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/BehaviorDecisionResult.java b/android/robot/src/main/java/org/openbot/cartfollow/BehaviorDecisionResult.java new file mode 100644 index 000000000..b8c186046 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/BehaviorDecisionResult.java @@ -0,0 +1,37 @@ +package org.openbot.cartfollow; + +public class BehaviorDecisionResult { + public final FollowState state; + public final BehaviorAction selectedAction; + public final String actionReason; + public final String safetyBlockReason; + public final float confidence; + public final DistanceEvidence distanceEvidence; + public final TraversabilityEvidence traversabilityEvidence; + + public BehaviorDecisionResult( + FollowState state, + BehaviorAction selectedAction, + String actionReason, + String safetyBlockReason, + float confidence) { + this(state, selectedAction, actionReason, safetyBlockReason, confidence, null, null); + } + + public BehaviorDecisionResult( + FollowState state, + BehaviorAction selectedAction, + String actionReason, + String safetyBlockReason, + float confidence, + DistanceEvidence distanceEvidence, + TraversabilityEvidence traversabilityEvidence) { + this.state = state; + this.selectedAction = selectedAction; + this.actionReason = actionReason; + this.safetyBlockReason = safetyBlockReason; + this.confidence = confidence; + this.distanceEvidence = distanceEvidence; + this.traversabilityEvidence = traversabilityEvidence; + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/DistanceEvidence.java b/android/robot/src/main/java/org/openbot/cartfollow/DistanceEvidence.java new file mode 100644 index 000000000..e1e4c3eb5 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/DistanceEvidence.java @@ -0,0 +1,13 @@ +package org.openbot.cartfollow; + +public class DistanceEvidence { + public final DistanceState state; + public final float confidence; + public final String reason; + + public DistanceEvidence(DistanceState state, float confidence, String reason) { + this.state = state == null ? DistanceState.UNKNOWN : state; + this.confidence = confidence; + this.reason = reason; + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/FollowState.java b/android/robot/src/main/java/org/openbot/cartfollow/FollowState.java index aa59c4223..204f5eaad 100644 --- a/android/robot/src/main/java/org/openbot/cartfollow/FollowState.java +++ b/android/robot/src/main/java/org/openbot/cartfollow/FollowState.java @@ -8,6 +8,8 @@ public enum FollowState { REACQUIRE_TARGET, READY_TO_FOLLOW, FOLLOW, + FOLLOW_CAUTION, + IDENTITY_UNCERTAIN, LOST, SEARCH, STOP diff --git a/android/robot/src/main/java/org/openbot/cartfollow/FollowStateMachine.java b/android/robot/src/main/java/org/openbot/cartfollow/FollowStateMachine.java index c635798ae..004938b48 100644 --- a/android/robot/src/main/java/org/openbot/cartfollow/FollowStateMachine.java +++ b/android/robot/src/main/java/org/openbot/cartfollow/FollowStateMachine.java @@ -19,7 +19,10 @@ public static class FrameResult { public final boolean tooClose; public final Bitmap snapshot; public final int countdownSec; + public float matchScore; public ImageSetpointDistanceEstimator.DistanceEstimate distanceEstimate; + public BehaviorDecisionResult behaviorDecision; + public IdentityEvidence identityEvidence; public FrameResult( FollowState state, @@ -40,6 +43,7 @@ public FrameResult( this.tooClose = tooClose; this.snapshot = snapshot; this.countdownSec = countdownSec; + this.matchScore = 0f; } } @@ -47,8 +51,15 @@ public FrameResult( public int REACQUIRE_MATCH_N = 8; public int FOLLOW_LOST_M = 10; public long LOST_TO_SEARCH_MS = 800; - public long SEARCH_TIMEOUT_MS = 5000; + public long SEARCH_TIMEOUT_MS = 18000; + public long IDENTITY_UNCERTAIN_TIMEOUT_MS = 18000; public long COUNTDOWN_MS = 3000; + public int CAUTION_STABLE_FRAMES = 3; + public int UNCERTAIN_FRAMES = 3; + public int UNCERTAIN_RECOVER_FRAMES = 2; + public int REACQUIRE_STRICT_FRAMES = 1; + public int REACQUIRE_DEFAULT_FRAMES = 2; + public int LOST_RECOVER_FRAMES = 3; private final TargetMatcher matcher; private final ControlGenerator controlGenerator; @@ -58,6 +69,10 @@ public FrameResult( private int captureCount = 0; private int matchCount = 0; private int lostCount = 0; + private int midDefaultStreak = 0; + private int strongDefaultStreak = 0; + private int strongStrictStreak = 0; + private int unstableStreak = 0; private long stateEnterTime = 0L; private Bitmap snapshot = null; @@ -79,6 +94,7 @@ public void startCapture() { memory.clear(); snapshot = null; captureCount = 0; + resetEvidenceCounters(); state = FollowState.CAPTURE_TARGET; } } @@ -105,11 +121,22 @@ public void cancel() { captureCount = 0; matchCount = 0; lostCount = 0; + resetEvidenceCounters(); state = FollowState.IDLE; } public FrameResult onFrame( List persons, Bitmap frame, int frameW, int frameH, int sensorOrientation) { + return onFrame(persons, frame, frameW, frameH, sensorOrientation, null); + } + + public FrameResult onFrame( + List persons, + Bitmap frame, + int frameW, + int frameH, + int sensorOrientation, + IdentityEvidence externalIdentity) { List safePersons = persons == null ? new ArrayList<>() : persons; long now = System.currentTimeMillis(); @@ -152,15 +179,25 @@ public FrameResult onFrame( case REACQUIRE_TARGET: { TargetMatcher.MatchResult m = matcher.match(safePersons, frame, memory, frameW, frameH); - if (m.matched) matchCount++; + IdentityEvidence id = identityFrom(m, externalIdentity); + updateEvidenceCounters(id, m, safePersons); + if (reacquireReady(id, m, safePersons)) matchCount++; else matchCount = 0; - if (matchCount >= REACQUIRE_MATCH_N) { + Recognition selected = selectedCandidate(id, m); + if (selected != null + && (strongStrictStreak >= REACQUIRE_STRICT_FRAMES + || strongDefaultStreak >= REACQUIRE_DEFAULT_FRAMES + || matchCount >= REACQUIRE_MATCH_N)) { state = FollowState.READY_TO_FOLLOW; stateEnterTime = now; - memory.updateDynamic(m.best); + memory.updateDynamic(selected); + resetEvidenceCounters(); } - return new FrameResult( - state, new Control(0f, 0f), m.best, null, safePersons, m.matched, false, null, -1); + FrameResult fr = + new FrameResult( + state, new Control(0f, 0f), selected, null, safePersons, id.matched, false, null, -1); + fillIdentity(fr, id); + return fr; } case READY_TO_FOLLOW: { @@ -172,76 +209,186 @@ public FrameResult onFrame( FollowState.FOLLOW, new Control(0f, 0f), null, null, safePersons, false, false, null, -1); } TargetMatcher.MatchResult m = matcher.match(safePersons, frame, memory, frameW, frameH); - return new FrameResult( - FollowState.READY_TO_FOLLOW, new Control(0f, 0f), m.best, null, safePersons, m.matched, false, null, Math.max(0, cd)); + IdentityEvidence id = identityFrom(m, externalIdentity); + Recognition selected = selectedCandidate(id, m); + FrameResult fr = + new FrameResult( + FollowState.READY_TO_FOLLOW, + new Control(0f, 0f), + selected, + null, + safePersons, + id.matched, + false, + null, + Math.max(0, cd)); + fillIdentity(fr, id); + return fr; } case FOLLOW: { TargetMatcher.MatchResult m = matcher.match(safePersons, frame, memory, frameW, frameH); - if (m.matched) { - memory.updateDynamic(m.best); + IdentityEvidence id = identityFrom(m, externalIdentity); + updateEvidenceCounters(id, m, safePersons); + Recognition selected = selectedCandidate(id, m); + if (selected != null && followConfident(id, m, safePersons)) { + memory.updateDynamic(selected); lostCount = 0; ControlGenerator.Result res = controlGenerator.generateFromTarget( - m.best, safePersons, frameW, frameH, sensorOrientation, memory); + selected, safePersons, frameW, frameH, sensorOrientation, memory); FrameResult fr = new FrameResult( - FollowState.FOLLOW, res.control, m.best, null, safePersons, true, res.tooClose, null, -1); + FollowState.FOLLOW, res.control, selected, null, safePersons, true, res.tooClose, null, -1); + fillIdentity(fr, id); + fr.distanceEstimate = res.distanceEstimate; + return fr; + } + if (selected != null && followCaution(id, m, safePersons)) { + state = FollowState.FOLLOW_CAUTION; + memory.updateDynamic(selected); + ControlGenerator.Result res = + controlGenerator.generateFromTarget( + selected, safePersons, frameW, frameH, sensorOrientation, memory); + FrameResult fr = + new FrameResult( + FollowState.FOLLOW_CAUTION, + res.control, + selected, + null, + safePersons, + true, + res.tooClose, + null, + -1); + fillIdentity(fr, id); fr.distanceEstimate = res.distanceEstimate; return fr; } lostCount++; if (lostCount >= FOLLOW_LOST_M) { - state = FollowState.LOST; + state = FollowState.IDENTITY_UNCERTAIN; stateEnterTime = now; + resetEvidenceCounters(); } - return new FrameResult( - state, new Control(0f, 0f), null, null, safePersons, false, false, null, -1); + FrameResult fr = + new FrameResult( + state, new Control(0f, 0f), null, null, safePersons, false, false, null, -1); + fillIdentity(fr, id); + return fr; } - case LOST: { + case FOLLOW_CAUTION: { TargetMatcher.MatchResult m = matcher.match(safePersons, frame, memory, frameW, frameH); - if (m.matched) { + IdentityEvidence id = identityFrom(m, externalIdentity); + updateEvidenceCounters(id, m, safePersons); + Recognition selected = selectedCandidate(id, m); + if (selected != null && midDefaultStreak >= CAUTION_STABLE_FRAMES) { state = FollowState.FOLLOW; - lostCount = 0; - memory.updateDynamic(m.best); + memory.updateDynamic(selected); + ControlGenerator.Result res = + controlGenerator.generateFromTarget( + selected, safePersons, frameW, frameH, sensorOrientation, memory); + FrameResult fr = + new FrameResult( + FollowState.FOLLOW, res.control, selected, null, safePersons, true, res.tooClose, null, -1); + fillIdentity(fr, id); + fr.distanceEstimate = res.distanceEstimate; + return fr; + } + if (selected != null && followCaution(id, m, safePersons)) { + memory.updateDynamic(selected); ControlGenerator.Result res = controlGenerator.generateFromTarget( - m.best, safePersons, frameW, frameH, sensorOrientation, memory); + selected, safePersons, frameW, frameH, sensorOrientation, memory); FrameResult fr = new FrameResult( - FollowState.FOLLOW, res.control, m.best, null, safePersons, true, res.tooClose, null, -1); + FollowState.FOLLOW_CAUTION, + res.control, + selected, + null, + safePersons, + true, + res.tooClose, + null, + -1); + fillIdentity(fr, id); fr.distanceEstimate = res.distanceEstimate; return fr; } + unstableStreak++; + if (unstableStreak >= UNCERTAIN_FRAMES) { + state = FollowState.IDENTITY_UNCERTAIN; + stateEnterTime = now; + resetEvidenceCounters(); + } + FrameResult fr = + new FrameResult( + state, new Control(0f, 0f), null, null, safePersons, false, false, null, -1); + fillIdentity(fr, id); + return fr; + } + + case IDENTITY_UNCERTAIN: { + TargetMatcher.MatchResult m = matcher.match(safePersons, frame, memory, frameW, frameH); + IdentityEvidence id = identityFrom(m, externalIdentity); + updateEvidenceCounters(id, m, safePersons); + Recognition selected = selectedCandidate(id, m); + if (selected != null + && (strongStrictStreak >= UNCERTAIN_RECOVER_FRAMES + || midDefaultStreak >= UNCERTAIN_RECOVER_FRAMES)) { + state = FollowState.REACQUIRE_TARGET; + matchCount = 0; + stateEnterTime = now; + } else if (now - stateEnterTime >= IDENTITY_UNCERTAIN_TIMEOUT_MS) { + state = FollowState.STOP; + } + FrameResult fr = + new FrameResult( + state, new Control(0f, 0f), selected, null, safePersons, false, false, null, -1); + fillIdentity(fr, id); + return fr; + } + + case LOST: { + TargetMatcher.MatchResult m = matcher.match(safePersons, frame, memory, frameW, frameH); + IdentityEvidence id = identityFrom(m, externalIdentity); + updateEvidenceCounters(id, m, safePersons); + Recognition selected = selectedCandidate(id, m); + if (selected != null && lostRecoverReady(id, m, safePersons)) { + state = FollowState.REACQUIRE_TARGET; + matchCount = 0; + stateEnterTime = now; + } if (now - stateEnterTime >= LOST_TO_SEARCH_MS) { state = FollowState.SEARCH; stateEnterTime = now; } - return new FrameResult( - state, new Control(0f, 0f), null, null, safePersons, false, false, null, -1); + FrameResult fr = + new FrameResult( + state, new Control(0f, 0f), selected, null, safePersons, false, false, null, -1); + fillIdentity(fr, id); + return fr; } case SEARCH: { TargetMatcher.MatchResult m = matcher.match(safePersons, frame, memory, frameW, frameH); - if (m.matched) { - state = FollowState.FOLLOW; - lostCount = 0; - memory.updateDynamic(m.best); - ControlGenerator.Result res = - controlGenerator.generateFromTarget( - m.best, safePersons, frameW, frameH, sensorOrientation, memory); - FrameResult fr = - new FrameResult( - FollowState.FOLLOW, res.control, m.best, null, safePersons, true, res.tooClose, null, -1); - fr.distanceEstimate = res.distanceEstimate; - return fr; + IdentityEvidence id = identityFrom(m, externalIdentity); + updateEvidenceCounters(id, m, safePersons); + Recognition selected = selectedCandidate(id, m); + if (selected != null && lostRecoverReady(id, m, safePersons)) { + state = FollowState.REACQUIRE_TARGET; + matchCount = 0; + stateEnterTime = now; } if (now - stateEnterTime >= SEARCH_TIMEOUT_MS) { state = FollowState.STOP; } - return new FrameResult( - state, new Control(0f, 0f), null, null, safePersons, false, false, null, -1); + FrameResult fr = + new FrameResult( + state, new Control(0f, 0f), selected, null, safePersons, false, false, null, -1); + fillIdentity(fr, id); + return fr; } case STOP: @@ -251,6 +398,136 @@ public FrameResult onFrame( } } + private void fillIdentity(FrameResult fr, IdentityEvidence id) { + if (fr == null || id == null) return; + fr.identityEvidence = id; + fr.matchScore = id.confidence; + } + + private IdentityEvidence identityFrom(TargetMatcher.MatchResult m, IdentityEvidence external) { + if (external != null) return external; + return new IdentityEvidence( + m.matched ? m.score : 0f, + m.score, + m.matched, + m.matched ? "legacy_matched" : "legacy_not_matched", + ReIDMatchResult.unavailable("reid_not_connected", 0), + BboxContinuityEvidence.from( + m.best == null ? null : m.best.getLocation(), + memory.getLastBbox(), + memory.getPreviousBbox(), + 1, + 1), + 0, + 0, + m.best); + } + + private Recognition selectedCandidate(IdentityEvidence id, TargetMatcher.MatchResult m) { + if (id != null && id.bestCandidate != null) return id.bestCandidate; + return m == null ? null : m.best; + } + + private boolean followConfident( + IdentityEvidence id, TargetMatcher.MatchResult m, List persons) { + if (id != null && id.hasBelief()) { + return id.beliefConfirmed() + && id.beliefStableFrames >= 2 + && (id.bboxDefaultOk() || id.predictionOk() || id.trackId == id.lockedTrackId); + } + if (id != null && id.reidAvailable()) { + return id.weakOk() && id.bboxDefaultOk(); + } + return m != null && m.matched && (persons == null || persons.size() <= 1 || id.bboxDefaultOk()); + } + + private boolean followCaution( + IdentityEvidence id, TargetMatcher.MatchResult m, List persons) { + if (id != null && id.hasBelief()) { + return id.beliefCaution() + && (id.bboxDefaultOk() || id.predictionOk() || id.trackId == id.lockedTrackId); + } + if (id != null && id.reidAvailable()) { + return id.bboxDefaultOk() && (id.weakOk() || id.midOk()); + } + return m != null && m.matched && id != null && id.bboxDefaultOk() && persons != null && persons.size() <= 1; + } + + private boolean reacquireReady( + IdentityEvidence id, TargetMatcher.MatchResult m, List persons) { + if (id != null && id.hasBelief()) { + return id.beliefConfirmed() + && id.beliefStableFrames >= 3 + && (id.bboxDefaultOk() || id.predictionOk() || id.trackId == id.lockedTrackId); + } + if (id != null && id.reidAvailable()) { + return (id.strongOk() && id.bboxDefaultOk()) || (id.midOk() && id.bboxStrictOk()); + } + return m != null && m.matched && id != null && id.bboxStrictOk() && persons != null && persons.size() <= 1; + } + + private boolean lostRecoverReady( + IdentityEvidence id, TargetMatcher.MatchResult m, List persons) { + if (id != null && id.hasBelief()) { + return id.beliefConfirmed() && id.beliefStableFrames >= LOST_RECOVER_FRAMES; + } + if (id != null && id.reidAvailable()) { + return strongStrictStreak >= LOST_RECOVER_FRAMES + || strongDefaultStreak >= LOST_RECOVER_FRAMES + || midDefaultStreak >= LOST_RECOVER_FRAMES; + } + return m != null + && m.matched + && id != null + && id.bboxStrictOk() + && persons != null + && persons.size() <= 1 + && strongStrictStreak >= LOST_RECOVER_FRAMES; + } + + private void updateEvidenceCounters( + IdentityEvidence id, TargetMatcher.MatchResult m, List persons) { + if (id != null && id.hasBelief()) { + boolean safeBbox = id.bboxDefaultOk() || id.predictionOk() || id.trackId == id.lockedTrackId; + boolean midDefault = id.targetBelief >= IdentityBelief.BELIEF_CAUTION && safeBbox; + boolean strongDefault = id.targetBelief >= IdentityBelief.BELIEF_CONFIRM && safeBbox; + boolean strongStrict = + id.targetBelief >= IdentityBelief.BELIEF_CONFIRM + && (id.bboxStrictOk() || id.predictionOk() || id.trackId == id.lockedTrackId); + midDefaultStreak = midDefault ? midDefaultStreak + 1 : 0; + strongDefaultStreak = strongDefault ? strongDefaultStreak + 1 : 0; + strongStrictStreak = strongStrict ? strongStrictStreak + 1 : 0; + if (midDefault || strongDefault || strongStrict) unstableStreak = 0; + return; + } + boolean reidAvailable = id != null && id.reidAvailable(); + boolean midDefault; + boolean strongDefault; + boolean strongStrict; + if (reidAvailable) { + midDefault = id.midOk() && id.bboxDefaultOk(); + strongDefault = id.strongOk() && id.bboxDefaultOk(); + strongStrict = id.strongOk() && id.bboxStrictOk(); + } else { + boolean legacySafe = + m != null && m.matched && id != null && persons != null && persons.size() <= 1; + midDefault = legacySafe && id.bboxDefaultOk(); + strongDefault = legacySafe && id.bboxDefaultOk(); + strongStrict = legacySafe && id.bboxStrictOk(); + } + midDefaultStreak = midDefault ? midDefaultStreak + 1 : 0; + strongDefaultStreak = strongDefault ? strongDefaultStreak + 1 : 0; + strongStrictStreak = strongStrict ? strongStrictStreak + 1 : 0; + if (midDefault || strongDefault || strongStrict) unstableStreak = 0; + } + + private void resetEvidenceCounters() { + midDefaultStreak = 0; + strongDefaultStreak = 0; + strongStrictStreak = 0; + unstableStreak = 0; + } + private static Recognition selectLargest(List persons) { Recognition target = null; float maxArea = -1f; diff --git a/android/robot/src/main/java/org/openbot/cartfollow/HumanCartSimulatorFragment.java b/android/robot/src/main/java/org/openbot/cartfollow/HumanCartSimulatorFragment.java index 46c3fd54b..20a727c60 100644 --- a/android/robot/src/main/java/org/openbot/cartfollow/HumanCartSimulatorFragment.java +++ b/android/robot/src/main/java/org/openbot/cartfollow/HumanCartSimulatorFragment.java @@ -6,6 +6,7 @@ import android.graphics.Matrix; import android.graphics.Paint; import android.graphics.RectF; +import android.app.Activity; import android.os.Bundle; import android.os.Handler; import android.os.HandlerThread; @@ -24,6 +25,9 @@ import java.util.Locale; import org.jetbrains.annotations.NotNull; import org.openbot.R; +import org.openbot.cartfollow.diagnostics.CartFollowDiagnosticConfig; +import org.openbot.cartfollow.diagnostics.CartFollowDiagnosticSaver; +import org.openbot.cartfollow.diagnostics.CartFollowDiagnosticSession; import org.openbot.common.CameraFragment; import org.openbot.databinding.FragmentHumanCartSimulatorBinding; import org.openbot.env.ImageUtils; @@ -68,6 +72,20 @@ public class HumanCartSimulatorFragment extends CameraFragment { private final TargetMatcher matcher = new TargetMatcher(); private final FollowStateMachine stateMachine = new FollowStateMachine(matcher, controlGenerator); + private final ActionArbitrator actionArbitrator = new ActionArbitrator(); + private final TargetTrackManager targetTrackManager = new TargetTrackManager(); + private final IdentityBeliefAccumulator beliefAccumulator = new IdentityBeliefAccumulator(); + private ReIDCoordinator reidCoordinator; + private final CartFollowDiagnosticConfig diagnosticConfig = new CartFollowDiagnosticConfig(); + private final CartFollowDiagnosticSaver diagnosticSaver = new CartFollowDiagnosticSaver(); + private CartFollowDiagnosticSession diagnosticSession; + private boolean diagnosticActive = false; + private boolean targetEventAwaitingReturn = false; + private boolean showFullDebug = false; + private long lastDiagnosticFrameLogMs = 0L; + private long lastDiagnosticCropMs = 0L; + private long lastDiagnosticGalleryMs = 0L; + private Bitmap latestConfirmSnapshot; private final List drawBoxes = new ArrayList<>(); private int drawFrameWidth = 0; @@ -109,6 +127,7 @@ public View onCreateView( @Override public void onViewCreated(@NonNull View view, @Nullable Bundle savedInstanceState) { super.onViewCreated(view, savedInstanceState); + reidCoordinator = new ReIDCoordinator(requireActivity(), getNumThreads()); binding.confidenceValue.setText((int) (minConfidence * 100) + "%"); binding.plusConfidence.setOnClickListener( @@ -136,19 +155,60 @@ public void onViewCreated(@NonNull View view, @Nullable Bundle savedInstanceStat setAnalyserResolution(Enums.Preview.HD.getValue()); binding.trackingOverlay.addCallback(canvas -> drawOverlay(canvas)); + binding.btnDebugDetails.setOnClickListener( + v -> { + showFullDebug = !showFullDebug; + binding.btnDebugDetails.setText(showFullDebug ? "收起详情" : "调试详情"); + }); + resetTargetEventButton(); + binding.btnTargetEvent.setOnClickListener(v -> recordTargetEvent()); - binding.btnConfirm.setOnClickListener(v -> stateMachine.confirm()); - binding.btnRetake.setOnClickListener(v -> stateMachine.retake()); - binding.btnCancel.setOnClickListener(v -> stateMachine.cancel()); + binding.btnConfirm.setOnClickListener( + v -> { + if (reidCoordinator != null) reidCoordinator.confirmGallery(); + int lockedTrackId = targetTrackManager.lockClosest(stateMachine.getMemory().getLastBbox()); + beliefAccumulator.lockTrack(lockedTrackId); + activateDiagnosticSession(); + if (latestConfirmSnapshot != null) { + diagnosticSaver.saveGallerySnapshotAsync( + latestConfirmSnapshot, diagnosticSession, "confirmed_snapshot"); + } + stateMachine.confirm(); + }); + binding.btnRetake.setOnClickListener( + v -> { + if (reidCoordinator != null) reidCoordinator.reset(); + targetTrackManager.reset(); + beliefAccumulator.reset(); + stopDiagnosticSession(); + startDiagnosticSession(); + stateMachine.retake(); + }); + binding.btnCancel.setOnClickListener( + v -> { + if (reidCoordinator != null) reidCoordinator.reset(); + targetTrackManager.reset(); + beliefAccumulator.reset(); + stopDiagnosticSession(); + stateMachine.cancel(); + }); binding.startSwitch.setChecked(false); binding.startSwitch.setOnClickListener( v -> { if (binding.startSwitch.isChecked()) { binding.modelSpinner.setEnabled(false); + if (reidCoordinator != null) reidCoordinator.reset(); + targetTrackManager.reset(); + beliefAccumulator.reset(); + startDiagnosticSession(); stateMachine.startCapture(); } else { binding.modelSpinner.setEnabled(true); + if (reidCoordinator != null) reidCoordinator.reset(); + targetTrackManager.reset(); + beliefAccumulator.reset(); + stopDiagnosticSession(); stateMachine.cancel(); resetUiToIdle(); } @@ -157,7 +217,7 @@ public void onViewCreated(@NonNull View view, @Nullable Bundle savedInstanceStat private void resetUiToIdle() { updateCommandText(getString(R.string.cart_sim_idle)); - updateDebugInfo(FollowState.IDLE, new Control(0f, 0f), 0, 0f, null); + updateDebugInfo(FollowState.IDLE, new Control(0f, 0f), 0, 0f, null, null, null); if (binding != null) { binding.confirmPanel.setVisibility(View.GONE); binding.countdownText.setVisibility(View.GONE); @@ -236,6 +296,7 @@ public synchronized void onResume() { @Override public synchronized void onPause() { + stopDiagnosticSession(); handlerThread.quitSafely(); try { handlerThread.join(); @@ -247,6 +308,12 @@ public synchronized void onPause() { super.onPause(); } + @Override + public void onDestroy() { + diagnosticSaver.shutdown(); + super.onDestroy(); + } + protected synchronized void runInBackground(final Runnable r) { if (handler != null) handler.post(r); } @@ -299,14 +366,62 @@ protected void processFrame(Bitmap bitmap, ImageProxy image) { int frameW = getMaxAnalyseImageSize().getWidth(); int frameH = getMaxAnalyseImageSize().getHeight(); + targetTrackManager.update( + mappedRecognitions, frameW, frameH, SystemClock.elapsedRealtime()); + FollowState currentState = stateMachine.getState(); + Detector.Recognition largestPerson = selectLargest(mappedRecognitions); + if (currentState == FollowState.CAPTURE_TARGET) { + if (reidCoordinator != null) { + reidCoordinator.collectInitializationCandidate(workingFrame, largestPerson); + } + maybeSaveGalleryCandidate(workingFrame, largestPerson); + } + TargetMatcher.MatchResult legacyMatch = + matcher.match( + mappedRecognitions, workingFrame, stateMachine.getMemory(), frameW, frameH); + IdentityEvidence identity = + reidCoordinator == null + ? null + : reidCoordinator.evaluate( + mappedRecognitions, + workingFrame, + stateMachine.getMemory(), + currentState, + frameW, + frameH, + legacyMatch.score, + legacyMatch.matched, + legacyMatch.best); + if (identity != null) { + TargetTrack reidCandidateTrack = + targetTrackManager.getTrackForRecognition(identity.bestCandidate); + identity = + beliefAccumulator.update( + identity, + targetTrackManager, + reidCandidateTrack, + stateMachine.getMemory(), + frameW, + frameH); + } FollowStateMachine.FrameResult fr = stateMachine.onFrame( - mappedRecognitions, workingFrame, frameW, frameH, sensorOrientation); + mappedRecognitions, workingFrame, frameW, frameH, sensorOrientation, identity); + fr.behaviorDecision = decideBehavior(fr, frameW, frameH); updateDrawState(fr, frameW, frameH, sensorOrientation); - updateCommandText(commandForState(fr)); + String commandText = commandForState(fr); + updateCommandText(commandText); float fps = lastProcessingTimeMs > 0 ? 1000f / lastProcessingTimeMs : 0f; - updateDebugInfo(fr.state, fr.control, fr.persons.size(), fps, fr.distanceEstimate); + maybeSaveDiagnostics(workingFrame, fr, fps, commandText, frameW, frameH, sensorOrientation); + updateDebugInfo( + fr.state, + fr.control, + fr.persons.size(), + fps, + fr.distanceEstimate, + fr.behaviorDecision, + fr.identityEvidence); updateUiForState(fr); binding.trackingOverlay.postInvalidate(); } @@ -325,12 +440,23 @@ private synchronized void updateDrawState( for (Detector.Recognition r : fr.persons) { if (r == null || r.getLocation() == null) continue; int colorType = COLOR_NORMAL; - if (r == fr.target) { + TargetTrack track = targetTrackManager.getTrackForRecognition(r); + if (track != null && targetTrackManager.isLockedTrack(track)) { + colorType = COLOR_TARGET; + } else if (track != null && track.trackId == targetTrackManager.getSuspectedTrackId()) { + colorType = COLOR_CANDIDATE; + } else if (r == fr.target) { colorType = fr.matched ? COLOR_TARGET : COLOR_FAIL; } else if (r == fr.candidate) { colorType = COLOR_CANDIDATE; } - drawBoxes.add(new DrawBox(new RectF(r.getLocation()), colorType)); + String label = null; + if (track != null) { + label = + String.format( + Locale.US, "T%d b=%.2f", track.trackId, beliefAccumulator.getBeliefForTrack(track)); + } + drawBoxes.add(new DrawBox(new RectF(r.getLocation()), colorType, label)); } drawFrameWidth = frameW; drawFrameHeight = frameH; @@ -362,19 +488,19 @@ private void drawOverlay(Canvas canvas) { RectF rect = new RectF(box.location); matrix.mapRect(rect); Paint paint; - String label = null; + String label = box.label; switch (box.colorType) { case COLOR_TARGET: paint = targetBoxPaint; - label = "目标"; + if (label == null) label = "目标"; break; case COLOR_CANDIDATE: paint = candidateBoxPaint; - label = "候选"; + if (label == null) label = "候选"; break; case COLOR_FAIL: paint = failBoxPaint; - label = "匹配失败"; + if (label == null) label = "匹配失败"; break; default: paint = personBoxPaint; @@ -389,6 +515,26 @@ private void drawOverlay(Canvas canvas) { } private String commandForState(FollowStateMachine.FrameResult fr) { + if (fr.behaviorDecision != null) { + switch (fr.behaviorDecision.selectedAction) { + case LOCAL_SEARCH_LEFT: + return HumanCommandInterpreter.CMD_TURN_LEFT; + case LOCAL_SEARCH_RIGHT: + return HumanCommandInterpreter.CMD_TURN_RIGHT; + case BLOCKED_WAIT: + return "前方受阻,请停止等待"; + case MOTION_STOP: + case HARD_STOP: + case EMERGENCY_STOP: + return HumanCommandInterpreter.CMD_STOP; + case REACQUIRE_HOLD: + return "疑似目标,请停止确认"; + case FOLLOW_SLOW: + case FOLLOW_CAUTION: + default: + break; + } + } switch (fr.state) { case IDLE: return "待命,打开 Start 开始采集目标"; @@ -403,10 +549,13 @@ private String commandForState(FollowStateMachine.FrameResult fr) { case READY_TO_FOLLOW: return fr.countdownSec >= 0 ? fr.countdownSec + " 秒后启动" : "准备启动"; case FOLLOW: + case FOLLOW_CAUTION: if (fr.distanceEstimate != null) { return interpreter.interpret(fr.control, fr.state, fr.distanceEstimate.state); } return interpreter.interpret(fr.control, fr.state, fr.tooClose); + case IDENTITY_UNCERTAIN: + return "身份不确定,请停止"; case LOST: return "目标丢失,请停止"; case SEARCH: @@ -432,6 +581,7 @@ private void updateUiForState(FollowStateMachine.FrameResult fr) { boolean showConfirm = fr.state == FollowState.LOCKED_PENDING_CONFIRM; binding.confirmPanel.setVisibility(showConfirm ? View.VISIBLE : View.GONE); if (showConfirm && fr.snapshot != null) { + latestConfirmSnapshot = fr.snapshot; binding.snapshotView.setImageBitmap(fr.snapshot); } boolean showCountdown = fr.state == FollowState.READY_TO_FOLLOW; @@ -443,12 +593,166 @@ private void updateUiForState(FollowStateMachine.FrameResult fr) { }); } + private void startDiagnosticSession() { + stopDiagnosticSession(); + diagnosticSession = new CartFollowDiagnosticSession(requireContext().getApplicationContext()); + diagnosticSession.initCsvFiles(); + diagnosticActive = false; + targetEventAwaitingReturn = false; + latestConfirmSnapshot = null; + lastDiagnosticFrameLogMs = 0L; + lastDiagnosticCropMs = 0L; + lastDiagnosticGalleryMs = 0L; + resetTargetEventButton(); + } + + private void activateDiagnosticSession() { + if (diagnosticSession == null) { + startDiagnosticSession(); + } + diagnosticActive = true; + targetEventAwaitingReturn = false; + String detectorName = getModel() == null ? "" : getModel().name; + boolean reidAvailable = reidCoordinator != null && reidCoordinator.isAvailable(); + int gallerySize = reidCoordinator == null ? 0 : reidCoordinator.getGallerySize(); + diagnosticSession.writeSessionInfo( + diagnosticConfig, detectorName, minConfidence, reidAvailable, gallerySize); + resetTargetEventButton(); + Toast.makeText( + requireContext(), + "Diagnostic: " + diagnosticSession.sessionDir.getAbsolutePath(), + Toast.LENGTH_SHORT) + .show(); + } + + private void stopDiagnosticSession() { + if (diagnosticSession != null && diagnosticActive) { + diagnosticSaver.saveEventAsync(diagnosticSession, frameNum, "session_stop", ""); + } + diagnosticActive = false; + diagnosticSession = null; + targetEventAwaitingReturn = false; + lastDiagnosticFrameLogMs = 0L; + lastDiagnosticCropMs = 0L; + lastDiagnosticGalleryMs = 0L; + resetTargetEventButton(); + } + + private void resetTargetEventButton() { + if (binding == null) return; + Activity activity = getActivity(); + if (activity == null) return; + activity.runOnUiThread( + () -> { + if (binding == null) return; + binding.btnTargetEvent.setEnabled(diagnosticActive); + binding.btnTargetEvent.setText( + targetEventAwaitingReturn ? "目标回到画面" : "目标离开画面"); + }); + } + + private void recordTargetEvent() { + if (!diagnosticActive || diagnosticSession == null) return; + String eventType = targetEventAwaitingReturn ? "target_return" : "target_left"; + diagnosticSaver.saveEventAsync(diagnosticSession, frameNum, eventType, ""); + targetEventAwaitingReturn = !targetEventAwaitingReturn; + resetTargetEventButton(); + } + + private void maybeSaveDiagnostics( + Bitmap workingFrame, + FollowStateMachine.FrameResult fr, + float fps, + String commandText, + int frameW, + int frameH, + int sensorOrientation) { + if (!diagnosticActive || diagnosticSession == null || fr == null) return; + long now = SystemClock.elapsedRealtime(); + boolean shouldLog = + lastDiagnosticFrameLogMs == 0L + || now - lastDiagnosticFrameLogMs >= diagnosticConfig.frameLogIntervalMs; + boolean shouldSaveCrop = + lastDiagnosticCropMs == 0L || now - lastDiagnosticCropMs >= diagnosticConfig.cropIntervalMs; + if (!shouldLog && !shouldSaveCrop) return; + if (shouldLog) lastDiagnosticFrameLogMs = now; + if (shouldSaveCrop) lastDiagnosticCropMs = now; + + Detector.Recognition locked = recognitionForTrack(targetTrackManager.getLockedTrack()); + TargetTrack suspectedTrack = targetTrackManager.getTrackById(targetTrackManager.getSuspectedTrackId()); + Detector.Recognition suspected = recognitionForTrack(suspectedTrack); + Detector.Recognition bestReid = + reidCoordinator == null ? null : reidCoordinator.getLastBestCandidate(); + diagnosticSaver.saveFrameAsync( + workingFrame, + diagnosticSession, + diagnosticConfig, + frameNum, + frameW, + frameH, + sensorOrientation, + fps, + fr.persons == null ? 0 : fr.persons.size(), + fr.state.name(), + fr.behaviorDecision, + commandText, + fr.identityEvidence, + locked, + suspected, + bestReid, + shouldSaveCrop); + } + + private void maybeSaveGalleryCandidate(Bitmap frame, Detector.Recognition candidate) { + if (diagnosticSession == null || frame == null || candidate == null) return; + if (candidate.getLocation() == null) return; + long now = SystemClock.elapsedRealtime(); + if (lastDiagnosticGalleryMs != 0L + && now - lastDiagnosticGalleryMs < diagnosticConfig.cropIntervalMs) { + return; + } + Bitmap crop = cropPerson(frame, candidate.getLocation(), diagnosticConfig.paddingRatio); + if (crop == null) return; + lastDiagnosticGalleryMs = now; + diagnosticSaver.saveGallerySnapshotAsync( + crop, diagnosticSession, "gallery_candidate_" + frameNum); + crop.recycle(); + } + + private static Detector.Recognition recognitionForTrack(TargetTrack track) { + return track == null || !track.isVisible() ? null : track.recognition; + } + + private static Bitmap cropPerson(Bitmap frame, RectF bbox, float paddingRatio) { + if (frame == null || bbox == null) return null; + float padX = bbox.width() * paddingRatio; + float padY = bbox.height() * paddingRatio; + int left = clamp((int) (bbox.left - padX), 0, frame.getWidth() - 1); + int top = clamp((int) (bbox.top - padY), 0, frame.getHeight() - 1); + int right = clamp((int) (bbox.right + padX), left + 1, frame.getWidth()); + int bottom = clamp((int) (bbox.bottom + padY), top + 1, frame.getHeight()); + int width = right - left; + int height = bottom - top; + if (width <= 0 || height <= 0) return null; + try { + return Bitmap.createBitmap(frame, left, top, width, height); + } catch (Exception e) { + return null; + } + } + + private static int clamp(int value, int min, int max) { + return Math.max(min, Math.min(max, value)); + } + private void updateDebugInfo( FollowState state, Control control, int persons, float fps, - ImageSetpointDistanceEstimator.DistanceEstimate dist) { + ImageSetpointDistanceEstimator.DistanceEstimate dist, + BehaviorDecisionResult behaviorDecision, + IdentityEvidence identityEvidence) { if (binding == null) return; float forward = (control.getLeft() + control.getRight()) / 2f; float turn = (control.getRight() - control.getLeft()) / 2f; @@ -466,10 +770,36 @@ private void updateDebugInfo( } else { distLine = "dist=-"; } - String info = + String behaviorLine; + if (behaviorDecision != null) { + behaviorLine = + String.format( + Locale.US, + "action=%s\nactionReason=%s\nsafetyBlock=%s\nactionConf=%.2f", + behaviorDecision.selectedAction.name(), + behaviorDecision.actionReason, + behaviorDecision.safetyBlockReason == null ? "-" : behaviorDecision.safetyBlockReason, + behaviorDecision.confidence); + if (behaviorDecision.traversabilityEvidence != null) { + TraversabilityEvidence trav = behaviorDecision.traversabilityEvidence; + behaviorLine += + String.format( + Locale.US, + "\ncenterBlocked=%s\nfreeLCR=%.2f/%.2f/%.2f\ntravReason=%s", + trav.centerBlocked, + trav.leftFreeScore, + trav.centerFreeScore, + trav.rightFreeScore, + trav.reason); + } + } else { + behaviorLine = "action=-\nactionReason=-\nsafetyBlock=-\nactionConf=0.00"; + } + String identityLine = buildIdentityDebugLine(identityEvidence); + String fullInfo = String.format( Locale.US, - "state=%s\nforward=%.2f\nturn=%.2f\nleft=%.2f\nright=%.2f\npersons=%d\nfps=%.1f\n%s", + "state=%s\nforward=%.2f\nturn=%.2f\nleft=%.2f\nright=%.2f\npersons=%d\nfps=%.1f\n%s\n%s\n%s", state.name(), forward, turn, @@ -477,14 +807,162 @@ private void updateDebugInfo( control.getRight(), persons, fps, - distLine); + distLine, + behaviorLine, + identityLine); + String compactInfo = + String.format( + Locale.US, + "fps=%.1f\nstate=%s\naction=%s\npersons=%d\ntrack=%d locked=%d suspected=%d\nbelief=%.2f\nbest=%.3f margin=%.3f", + fps, + state.name(), + behaviorDecision == null ? "-" : behaviorDecision.selectedAction.name(), + persons, + identityEvidence == null ? -1 : identityEvidence.trackId, + identityEvidence == null ? -1 : identityEvidence.lockedTrackId, + identityEvidence == null ? -1 : identityEvidence.suspectedTrackId, + identityEvidence == null ? 0f : identityEvidence.targetBelief, + identityEvidence == null || identityEvidence.reidMatch == null + ? 0f + : identityEvidence.reidMatch.bestScore, + identityEvidence == null || identityEvidence.reidMatch == null + ? 0f + : identityEvidence.reidMatch.margin); + String info = showFullDebug ? fullInfo : compactInfo; requireActivity().runOnUiThread(() -> binding.debugInfo.setText(info)); } + private String buildIdentityDebugLine(IdentityEvidence identity) { + if (identity == null) { + return "reidAvailable=false\ngallerySize=0\nbestScore=0.000\nsecondScore=0.000\nmargin=0.000\nweak/mid/strong=false/false/false\nbboxDefault=false bboxStrict=false prediction=false\nstableMatchCount=0\ncandidateSwitchCount=0\nreidLatencyMs=0\nreidReason=-\nactiveTrackCount=0\ntrackId=-1 lockedTrackId=-1 suspectedTrackId=-1\ntrackAge=0 missedFrames=0\nbelief=0.00 beliefStable=0 beliefUncertain=0\nbeliefReason=-"; + } + ReIDMatchResult reid = identity.reidMatch; + BboxContinuityEvidence bbox = identity.bboxContinuity; + boolean reidAvailable = reid != null && reid.reidAvailable; + int gallerySize = reid == null ? 0 : reid.gallerySize; + float best = reid == null ? 0f : reid.bestScore; + float second = reid == null ? 0f : reid.secondScore; + float margin = reid == null ? 0f : reid.margin; + long latency = reid == null ? 0L : reid.latencyMs; + String reason = reid == null ? identity.reason : reid.reason; + return String.format( + Locale.US, + "reidAvailable=%s\ngallerySize=%d\nbestScore=%.3f\nsecondScore=%.3f\nmargin=%.3f\nweak/mid/strong=%s/%s/%s\nbboxDefault=%s bboxStrict=%s prediction=%s\nstableMatchCount=%d\ncandidateSwitchCount=%d\nreidLatencyMs=%d\nreidReason=%s\nactiveTrackCount=%d\ntrackId=%d lockedTrackId=%d suspectedTrackId=%d\ntrackAge=%d missedFrames=%d\nbelief=%.2f reidC=%.2f bboxC=%.2f predC=%.2f switchP=%.2f\nbeliefStable=%d beliefUncertain=%d\nbeliefReason=%s", + reidAvailable, + gallerySize, + best, + second, + margin, + identity.weakOk(), + identity.midOk(), + identity.strongOk(), + bbox != null && bbox.bboxDefaultOk, + bbox != null && bbox.bboxStrictOk, + bbox != null && bbox.predictionOk, + identity.stableMatchCount, + identity.candidateSwitchCount, + latency, + reason == null ? "-" : reason, + identity.activeTrackCount, + identity.trackId, + identity.lockedTrackId, + identity.suspectedTrackId, + identity.trackAge, + identity.missedFrames, + identity.targetBelief, + identity.reidContribution, + identity.bboxContribution, + identity.predictionContribution, + identity.switchPenalty, + identity.beliefStableFrames, + identity.beliefUncertainFrames, + identity.beliefReason == null ? "-" : identity.beliefReason); + } + + private BehaviorDecisionResult decideBehavior( + FollowStateMachine.FrameResult fr, int frameW, int frameH) { + IdentityEvidence identity = + fr.identityEvidence != null + ? fr.identityEvidence + : new IdentityEvidence( + fr.matched ? fr.matchScore : 0f, + fr.matchScore, + fr.matched, + fr.matched ? "matched" : "not_matched"); + DistanceEvidence distance; + if (fr.distanceEstimate != null) { + distance = + new DistanceEvidence( + fr.distanceEstimate.state, + fr.distanceEstimate.confidence, + fr.distanceEstimate.failureReason); + } else { + distance = new DistanceEvidence(DistanceState.UNKNOWN, 0f, "distance_not_available"); + } + TraversabilityEvidence traversability = estimateTraversability(fr, frameW, frameH); + SystemSafetyEvidence safety = + new SystemSafetyEvidence( + false, true, detector != null, detector == null ? "detector_not_ready" : "ok"); + BehaviorDecisionResult decision = + actionArbitrator.decide( + fr.state, identity, distance, traversability, safety, stateMachine.getMemory(), frameW); + return new BehaviorDecisionResult( + decision.state, + decision.selectedAction, + decision.actionReason, + decision.safetyBlockReason, + decision.confidence, + distance, + traversability); + } + + private TraversabilityEvidence estimateTraversability( + FollowStateMachine.FrameResult fr, int frameW, int frameH) { + if (fr == null || fr.persons == null || frameW <= 0 || frameH <= 0) { + return new TraversabilityEvidence(1f, 1f, 1f, false, "default_clear"); + } + boolean centerBlocked = false; + float centerFreeScore = 1f; + for (Detector.Recognition person : fr.persons) { + if (person == null || person == fr.target || person.getLocation() == null) continue; + RectF b = person.getLocation(); + float cxRatio = b.centerX() / frameW; + boolean inCenter = cxRatio >= 0.33f && cxRatio <= 0.67f; + boolean lowerBodyRisk = b.bottom >= frameH * 0.55f; + boolean largeEnough = b.width() * b.height() >= frameW * frameH * 0.03f; + if (inCenter && lowerBodyRisk && largeEnough) { + centerBlocked = true; + centerFreeScore = Math.min(centerFreeScore, 0.2f); + } + } + return new TraversabilityEvidence( + 1f, + centerFreeScore, + 1f, + centerBlocked, + centerBlocked ? "non_target_in_center_corridor" : "default_clear"); + } + protected Model getModel() { return model; } + private static Detector.Recognition selectLargest(List persons) { + Detector.Recognition target = null; + float maxArea = -1f; + if (persons == null) return null; + for (Detector.Recognition r : persons) { + if (r == null || r.getLocation() == null) continue; + RectF loc = r.getLocation(); + float area = loc.width() * loc.height(); + if (area > maxArea) { + maxArea = area; + target = r; + } + } + return target; + } + @Override protected void setModel(Model model) { if (this.model != model) { @@ -505,10 +983,12 @@ protected int getNumThreads() { private static class DrawBox { final RectF location; final int colorType; + final String label; - DrawBox(RectF location, int colorType) { + DrawBox(RectF location, int colorType, String label) { this.location = location; this.colorType = colorType; + this.label = label; } } } diff --git a/android/robot/src/main/java/org/openbot/cartfollow/IdentityBelief.java b/android/robot/src/main/java/org/openbot/cartfollow/IdentityBelief.java new file mode 100644 index 000000000..5a26d601a --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/IdentityBelief.java @@ -0,0 +1,38 @@ +package org.openbot.cartfollow; + +public class IdentityBelief { + public static final float BELIEF_CONFIRM = 0.75f; + public static final float BELIEF_CAUTION = 0.55f; + public static final float BELIEF_LOST = 0.30f; + + public final int trackId; + public final float targetBelief; + public final float reidContribution; + public final float bboxContribution; + public final float predictionContribution; + public final float switchPenalty; + public final int stableFrames; + public final int uncertainFrames; + public final String beliefReason; + + public IdentityBelief( + int trackId, + float targetBelief, + float reidContribution, + float bboxContribution, + float predictionContribution, + float switchPenalty, + int stableFrames, + int uncertainFrames, + String beliefReason) { + this.trackId = trackId; + this.targetBelief = targetBelief; + this.reidContribution = reidContribution; + this.bboxContribution = bboxContribution; + this.predictionContribution = predictionContribution; + this.switchPenalty = switchPenalty; + this.stableFrames = stableFrames; + this.uncertainFrames = uncertainFrames; + this.beliefReason = beliefReason; + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/IdentityBeliefAccumulator.java b/android/robot/src/main/java/org/openbot/cartfollow/IdentityBeliefAccumulator.java new file mode 100644 index 000000000..d2b1109a8 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/IdentityBeliefAccumulator.java @@ -0,0 +1,298 @@ +package org.openbot.cartfollow; + +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import org.openbot.tflite.Detector.Recognition; + +public class IdentityBeliefAccumulator { + private final Map beliefs = new HashMap<>(); + private final Map stableFrames = new HashMap<>(); + private final Map uncertainFrames = new HashMap<>(); + + private int lockedTrackId = -1; + private int lastSelectedTrackId = -1; + private int candidateSwitchCount = 0; + + public void reset() { + beliefs.clear(); + stableFrames.clear(); + uncertainFrames.clear(); + lockedTrackId = -1; + lastSelectedTrackId = -1; + candidateSwitchCount = 0; + } + + public void lockTrack(int trackId) { + lockedTrackId = trackId; + if (trackId >= 0) { + beliefs.put(trackId, IdentityBelief.BELIEF_CONFIRM); + stableFrames.put(trackId, 1); + uncertainFrames.put(trackId, 0); + lastSelectedTrackId = trackId; + } + } + + public float getBeliefForTrack(TargetTrack track) { + if (track == null) return 0f; + Float value = beliefs.get(track.trackId); + return value == null ? 0f : value; + } + + public IdentityEvidence update( + IdentityEvidence base, + TargetTrackManager trackManager, + TargetTrack reidCandidateTrack, + TargetMemory memory, + int frameW, + int frameH) { + if (trackManager == null) return base; + int managerLockedId = trackManager.getLockedTrackId(); + if (managerLockedId != lockedTrackId) { + lockedTrackId = managerLockedId; + if (lockedTrackId >= 0 && !beliefs.containsKey(lockedTrackId)) { + beliefs.put(lockedTrackId, IdentityBelief.BELIEF_CONFIRM); + stableFrames.put(lockedTrackId, 1); + } + } + + List tracks = trackManager.getTracks(); + TargetTrack lockedTrack = trackManager.getLockedTrack(); + TargetTrack selectedTrack = null; + IdentityBelief selectedBelief = null; + BboxContinuityEvidence selectedBbox = null; + + for (TargetTrack track : tracks) { + IdentityBelief belief = + updateTrackBelief(base, track, reidCandidateTrack, lockedTrack, memory, frameW, frameH); + BboxContinuityEvidence bbox = bboxEvidence(track, memory, frameW, frameH); + if (!track.isVisible()) continue; + if (selectedTrack == null || shouldSelect(track, belief, selectedTrack, selectedBelief)) { + selectedTrack = track; + selectedBelief = belief; + selectedBbox = bbox; + } + } + + if (lockedTrack != null + && lockedTrack.isVisible() + && getBeliefForTrack(lockedTrack) >= IdentityBelief.BELIEF_LOST) { + selectedTrack = lockedTrack; + selectedBelief = + new IdentityBelief( + lockedTrack.trackId, + getBeliefForTrack(lockedTrack), + 0f, + 0f, + 0f, + 0f, + getStableFrames(lockedTrack.trackId), + getUncertainFrames(lockedTrack.trackId), + "locked_track_priority"); + selectedBbox = bboxEvidence(lockedTrack, memory, frameW, frameH); + } + + if (selectedTrack == null || selectedBelief == null) { + trackManager.setSuspectedTrackId(-1); + return withBelief( + base, + null, + null, + null, + -1, + managerLockedId, + -1, + trackManager.getActiveTrackCount(), + "no_visible_track"); + } + + if (selectedTrack.trackId != lastSelectedTrackId) { + if (lastSelectedTrackId >= 0) candidateSwitchCount++; + lastSelectedTrackId = selectedTrack.trackId; + } + + int suspectedTrackId = -1; + if (selectedTrack.trackId != managerLockedId + && selectedBelief.targetBelief >= IdentityBelief.BELIEF_CAUTION) { + suspectedTrackId = selectedTrack.trackId; + } + trackManager.setSuspectedTrackId(suspectedTrackId); + + return withBelief( + base, + selectedTrack, + selectedBbox, + selectedBelief, + selectedTrack.trackId, + managerLockedId, + suspectedTrackId, + trackManager.getActiveTrackCount(), + selectedBelief.beliefReason); + } + + private IdentityBelief updateTrackBelief( + IdentityEvidence base, + TargetTrack track, + TargetTrack reidCandidateTrack, + TargetTrack lockedTrack, + TargetMemory memory, + int frameW, + int frameH) { + float old = getBeliefForTrack(track); + if (old <= 0f) old = track.trackId == lockedTrackId ? IdentityBelief.BELIEF_CAUTION : 0.08f; + + boolean isLocked = track.trackId == lockedTrackId; + boolean lockedVisible = lockedTrack != null && lockedTrack.isVisible(); + boolean isReidCandidate = reidCandidateTrack != null && reidCandidateTrack.trackId == track.trackId; + + BboxContinuityEvidence bbox = bboxEvidence(track, memory, frameW, frameH); + float reidContribution = reidContribution(base, isReidCandidate); + float bboxContribution = bbox.bboxStrictOk ? 0.15f : (bbox.bboxDefaultOk ? 0.10f : 0f); + float predictionContribution = bbox.predictionOk ? 0.06f : 0f; + float lockedContribution = isLocked && track.isVisible() ? 0.08f : 0f; + float ageContribution = track.ageFrames >= 3 && track.isVisible() ? 0.04f : 0f; + float switchPenalty = + isReidCandidate && lastSelectedTrackId >= 0 && track.trackId != lastSelectedTrackId ? 0.12f : 0f; + float lockedProtectionPenalty = + !isLocked && lockedVisible && isReidCandidate ? 0.18f : 0f; + float missedPenalty = Math.min(0.30f, track.missedFrames * 0.08f); + float weakMarginPenalty = + isReidCandidate && base != null && base.reidAvailable() && !base.weakOk() ? 0.08f : 0f; + + float decay = track.isVisible() ? 0.80f : 0.70f; + float positive = + reidContribution + + bboxContribution + + predictionContribution + + lockedContribution + + ageContribution; + float penalty = switchPenalty + lockedProtectionPenalty + missedPenalty + weakMarginPenalty; + float belief = clamp(old * decay + positive - penalty, 0f, 1f); + + int stable = belief >= IdentityBelief.BELIEF_CAUTION ? getStableFrames(track.trackId) + 1 : 0; + int uncertain = + belief < IdentityBelief.BELIEF_CAUTION ? getUncertainFrames(track.trackId) + 1 : 0; + beliefs.put(track.trackId, belief); + stableFrames.put(track.trackId, stable); + uncertainFrames.put(track.trackId, uncertain); + + String reason = + String.format( + Locale.US, + "belief=%.2f old=%.2f reid=%.2f bbox=%.2f pred=%.2f locked=%s switchPenalty=%.2f missed=%d", + belief, + old, + reidContribution, + bboxContribution, + predictionContribution, + isLocked, + switchPenalty + lockedProtectionPenalty, + track.missedFrames); + return new IdentityBelief( + track.trackId, + belief, + reidContribution, + bboxContribution, + predictionContribution, + switchPenalty + lockedProtectionPenalty, + stable, + uncertain, + reason); + } + + private IdentityEvidence withBelief( + IdentityEvidence base, + TargetTrack selectedTrack, + BboxContinuityEvidence bbox, + IdentityBelief beliefDetails, + int trackId, + int lockedTrackId, + int suspectedTrackId, + int activeTrackCount, + String reason) { + ReIDMatchResult reid = + base == null ? ReIDMatchResult.unavailable("identity_base_missing", 0) : base.reidMatch; + Recognition candidate = selectedTrack == null ? null : selectedTrack.recognition; + float belief = selectedTrack == null ? 0f : getBeliefForTrack(selectedTrack); + boolean matched = selectedTrack != null && belief >= IdentityBelief.BELIEF_CAUTION; + float confidence = Math.max(base == null ? 0f : base.confidence, belief); + int stable = trackId >= 0 ? getStableFrames(trackId) : 0; + int uncertain = trackId >= 0 ? getUncertainFrames(trackId) : 0; + int rawSwitchCount = base == null ? 0 : base.candidateSwitchCount; + return new IdentityEvidence( + matched ? belief : 0f, + confidence, + matched, + reason, + reid, + bbox, + Math.max(base == null ? 0 : base.stableMatchCount, stable), + rawSwitchCount + candidateSwitchCount, + candidate, + trackId, + lockedTrackId, + suspectedTrackId, + activeTrackCount, + selectedTrack == null ? 0 : selectedTrack.ageFrames, + selectedTrack == null ? 0 : selectedTrack.missedFrames, + belief, + beliefDetails == null ? 0f : beliefDetails.reidContribution, + beliefDetails == null ? 0f : beliefDetails.bboxContribution, + beliefDetails == null ? 0f : beliefDetails.predictionContribution, + beliefDetails == null ? 0f : beliefDetails.switchPenalty, + stable, + uncertain, + reason); + } + + private boolean shouldSelect( + TargetTrack track, + IdentityBelief belief, + TargetTrack selectedTrack, + IdentityBelief selectedBelief) { + if (selectedTrack == null || selectedBelief == null) return true; + if (track.trackId == lockedTrackId && belief.targetBelief >= IdentityBelief.BELIEF_LOST) { + return true; + } + if (selectedTrack.trackId == lockedTrackId + && selectedBelief.targetBelief >= IdentityBelief.BELIEF_LOST) { + return false; + } + if (belief.targetBelief != selectedBelief.targetBelief) { + return belief.targetBelief > selectedBelief.targetBelief; + } + return track.ageFrames > selectedTrack.ageFrames; + } + + private static BboxContinuityEvidence bboxEvidence( + TargetTrack track, TargetMemory memory, int frameW, int frameH) { + if (track == null || memory == null || track.lastBbox == null) { + return BboxContinuityEvidence.unavailable("track_bbox_not_available"); + } + return BboxContinuityEvidence.from( + track.lastBbox, memory.getLastBbox(), memory.getPreviousBbox(), frameW, frameH); + } + + private static float reidContribution(IdentityEvidence base, boolean isReidCandidate) { + if (!isReidCandidate || base == null) return 0f; + if (base.strongOk()) return 0.25f; + if (base.midOk()) return 0.18f; + if (base.weakOk()) return 0.10f; + return base.matched ? 0.05f : 0f; + } + + private int getStableFrames(int trackId) { + Integer value = stableFrames.get(trackId); + return value == null ? 0 : value; + } + + private int getUncertainFrames(int trackId) { + Integer value = uncertainFrames.get(trackId); + return value == null ? 0 : value; + } + + private static float clamp(float value, float min, float max) { + return Math.max(min, Math.min(max, value)); + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/IdentityEvidence.java b/android/robot/src/main/java/org/openbot/cartfollow/IdentityEvidence.java new file mode 100644 index 000000000..206cc79e9 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/IdentityEvidence.java @@ -0,0 +1,158 @@ +package org.openbot.cartfollow; + +import org.openbot.tflite.Detector.Recognition; + +public class IdentityEvidence { + public final float score; + public final float confidence; + public final boolean matched; + public final String reason; + public final ReIDMatchResult reidMatch; + public final BboxContinuityEvidence bboxContinuity; + public final int stableMatchCount; + public final int candidateSwitchCount; + public final Recognition bestCandidate; + public final int trackId; + public final int lockedTrackId; + public final int suspectedTrackId; + public final int activeTrackCount; + public final int trackAge; + public final int missedFrames; + public final float targetBelief; + public final float reidContribution; + public final float bboxContribution; + public final float predictionContribution; + public final float switchPenalty; + public final int beliefStableFrames; + public final int beliefUncertainFrames; + public final String beliefReason; + + public IdentityEvidence(float score, float confidence, boolean matched, String reason) { + this(score, confidence, matched, reason, null, null, 0, 0, null); + } + + public IdentityEvidence( + float score, + float confidence, + boolean matched, + String reason, + ReIDMatchResult reidMatch, + BboxContinuityEvidence bboxContinuity, + int stableMatchCount, + int candidateSwitchCount, + Recognition bestCandidate) { + this( + score, + confidence, + matched, + reason, + reidMatch, + bboxContinuity, + stableMatchCount, + candidateSwitchCount, + bestCandidate, + -1, + -1, + -1, + 0, + 0, + 0, + 0f, + 0f, + 0f, + 0f, + 0f, + 0, + 0, + null); + } + + public IdentityEvidence( + float score, + float confidence, + boolean matched, + String reason, + ReIDMatchResult reidMatch, + BboxContinuityEvidence bboxContinuity, + int stableMatchCount, + int candidateSwitchCount, + Recognition bestCandidate, + int trackId, + int lockedTrackId, + int suspectedTrackId, + int activeTrackCount, + int trackAge, + int missedFrames, + float targetBelief, + float reidContribution, + float bboxContribution, + float predictionContribution, + float switchPenalty, + int beliefStableFrames, + int beliefUncertainFrames, + String beliefReason) { + this.score = score; + this.confidence = confidence; + this.matched = matched; + this.reason = reason; + this.reidMatch = reidMatch; + this.bboxContinuity = bboxContinuity; + this.stableMatchCount = stableMatchCount; + this.candidateSwitchCount = candidateSwitchCount; + this.bestCandidate = bestCandidate; + this.trackId = trackId; + this.lockedTrackId = lockedTrackId; + this.suspectedTrackId = suspectedTrackId; + this.activeTrackCount = activeTrackCount; + this.trackAge = trackAge; + this.missedFrames = missedFrames; + this.targetBelief = targetBelief; + this.reidContribution = reidContribution; + this.bboxContribution = bboxContribution; + this.predictionContribution = predictionContribution; + this.switchPenalty = switchPenalty; + this.beliefStableFrames = beliefStableFrames; + this.beliefUncertainFrames = beliefUncertainFrames; + this.beliefReason = beliefReason; + } + + public boolean hasBelief() { + return trackId >= 0 || activeTrackCount > 0 || lockedTrackId >= 0; + } + + public boolean beliefConfirmed() { + return hasBelief() && targetBelief >= IdentityBelief.BELIEF_CONFIRM; + } + + public boolean beliefCaution() { + return hasBelief() && targetBelief >= IdentityBelief.BELIEF_CAUTION; + } + + public boolean reidAvailable() { + return reidMatch != null && reidMatch.reidAvailable; + } + + public boolean weakOk() { + return reidAvailable() && reidMatch.weakOk; + } + + public boolean midOk() { + return reidAvailable() && reidMatch.midOk; + } + + public boolean strongOk() { + return reidAvailable() && reidMatch.strongOk; + } + + public boolean bboxDefaultOk() { + return bboxContinuity != null && bboxContinuity.bboxDefaultOk; + } + + public boolean bboxStrictOk() { + return bboxContinuity != null && bboxContinuity.bboxStrictOk; + } + + public boolean predictionOk() { + return bboxContinuity != null && bboxContinuity.predictionOk; + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/ReIDCoordinator.java b/android/robot/src/main/java/org/openbot/cartfollow/ReIDCoordinator.java new file mode 100644 index 000000000..5614f64d3 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/ReIDCoordinator.java @@ -0,0 +1,329 @@ +package org.openbot.cartfollow; + +import android.app.Activity; +import android.graphics.Bitmap; +import android.graphics.RectF; +import android.os.SystemClock; +import java.io.IOException; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import org.openbot.tflite.Detector.Recognition; + +public class ReIDCoordinator { + private static final int MAX_PENDING_GALLERY = 12; + private static final int CONFIRMED_GALLERY_K = 8; + private static final int MAX_CANDIDATES = 3; + private static final long FOLLOW_INTERVAL_MS = 1000; + private static final long HIGH_RISK_INTERVAL_MS = 300; + + private final ReIDFeatureExtractor extractor; + private final String disabledReason; + private final List pendingGallery = new ArrayList<>(); + private final List confirmedGallery = new ArrayList<>(); + + private ReIDMatchResult lastResult = ReIDMatchResult.unavailable("not_started", 0); + private Recognition lastBestCandidate = null; + private BboxContinuityEvidence lastBboxEvidence = + BboxContinuityEvidence.unavailable("not_started"); + private long lastRunTimeMs = 0L; + private int stableMatchCount = 0; + private int candidateSwitchCount = 0; + private int lastBestIndex = -1; + + public ReIDCoordinator(Activity activity, int numThreads) { + ReIDFeatureExtractor created = null; + String reason = null; + try { + created = + new TfliteReIDFeatureExtractor( + activity, TfliteReIDFeatureExtractor.DEFAULT_ASSET_PATH, numThreads); + } catch (IOException | IllegalArgumentException e) { + reason = "reid_model_unavailable"; + } + extractor = created; + disabledReason = reason; + } + + public void reset() { + pendingGallery.clear(); + confirmedGallery.clear(); + lastResult = ReIDMatchResult.unavailable(isAvailable() ? "reset" : disabledReason, 0); + lastBestCandidate = null; + lastBboxEvidence = BboxContinuityEvidence.unavailable("reset"); + lastRunTimeMs = 0L; + stableMatchCount = 0; + candidateSwitchCount = 0; + lastBestIndex = -1; + } + + public boolean isAvailable() { + return extractor != null; + } + + public int getGallerySize() { + return confirmedGallery.size(); + } + + public Recognition getLastBestCandidate() { + return lastBestCandidate; + } + + public BboxContinuityEvidence getLastBboxEvidence() { + return lastBboxEvidence; + } + + public ReIDMatchResult getLastResult() { + return lastResult; + } + + public void collectInitializationCandidate(Bitmap frame, Recognition candidate) { + if (!isAvailable() || frame == null || candidate == null || candidate.getLocation() == null) { + return; + } + if (pendingGallery.size() >= MAX_PENDING_GALLERY) return; + Bitmap crop = cropPerson(frame, candidate.getLocation(), 0.08f); + float[] feature = extractor.extract(crop); + if (feature != null) pendingGallery.add(feature); + } + + public void confirmGallery() { + confirmedGallery.clear(); + if (pendingGallery.isEmpty()) return; + for (float[] feature : selectDiverse(pendingGallery, CONFIRMED_GALLERY_K)) { + confirmedGallery.add(feature); + } + lastResult = + ReIDMatchResult.unavailable( + isAvailable() ? "gallery_confirmed" : disabledReason, confirmedGallery.size()); + } + + public IdentityEvidence evaluate( + List persons, + Bitmap frame, + TargetMemory memory, + FollowState state, + int frameW, + int frameH, + float legacyScore, + boolean legacyMatched, + Recognition legacyBest) { + ReIDMatchResult result = + maybeRunReID(persons, frame, memory, state, frameW, frameH, legacyBest); + Recognition best = lastBestCandidate != null ? lastBestCandidate : legacyBest; + boolean matched = legacyMatched; + float confidence = legacyScore; + if (result.reidAvailable && result.weakOk) { + matched = true; + confidence = result.bestScore; + } + return new IdentityEvidence( + matched ? confidence : 0f, + confidence, + matched, + result.reason, + result, + lastBboxEvidence, + stableMatchCount, + candidateSwitchCount, + best); + } + + private ReIDMatchResult maybeRunReID( + List persons, + Bitmap frame, + TargetMemory memory, + FollowState state, + int frameW, + int frameH, + Recognition legacyBest) { + if (!isAvailable()) { + lastResult = ReIDMatchResult.unavailable(disabledReason, 0); + lastBestCandidate = legacyBest; + lastBboxEvidence = bboxEvidence(legacyBest, memory, frameW, frameH); + return lastResult; + } + if (confirmedGallery.isEmpty()) { + lastResult = ReIDMatchResult.unavailable("gallery_empty", 0); + lastBestCandidate = legacyBest; + lastBboxEvidence = bboxEvidence(legacyBest, memory, frameW, frameH); + return lastResult; + } + if (persons == null || persons.isEmpty() || frame == null) { + lastResult = ReIDMatchResult.unavailable("no_candidates", confirmedGallery.size()); + lastBestCandidate = null; + lastBboxEvidence = BboxContinuityEvidence.unavailable("no_candidates"); + return lastResult; + } + long now = SystemClock.elapsedRealtime(); + boolean highRisk = + state != FollowState.FOLLOW || persons.size() > 1 || legacyBest == null || !same(lastBestCandidate, legacyBest); + long interval = highRisk ? HIGH_RISK_INTERVAL_MS : FOLLOW_INTERVAL_MS; + if (now - lastRunTimeMs < interval && lastResult.reidAvailable) { + return lastResult; + } + + long start = SystemClock.elapsedRealtime(); + List scores = new ArrayList<>(); + for (CandidateRef ref : candidateRefs(persons, legacyBest)) { + Bitmap crop = cropPerson(frame, ref.recognition.getLocation(), 0.08f); + float[] feature = extractor.extract(crop); + if (feature == null) continue; + float score = maxSimilarity(feature, confirmedGallery); + scores.add(new CandidateScore(ref.index, ref.recognition, score)); + } + if (scores.isEmpty()) { + lastResult = ReIDMatchResult.unavailable("feature_extract_failed", confirmedGallery.size()); + lastBestCandidate = legacyBest; + lastBboxEvidence = bboxEvidence(legacyBest, memory, frameW, frameH); + return lastResult; + } + scores.sort((a, b) -> Float.compare(b.score, a.score)); + CandidateScore best = scores.get(0); + float second = scores.size() > 1 ? scores.get(1).score : 0f; + if (best.index == lastBestIndex) { + stableMatchCount++; + } else { + if (lastBestIndex >= 0) candidateSwitchCount++; + stableMatchCount = 1; + lastBestIndex = best.index; + } + lastBestCandidate = best.recognition; + lastBboxEvidence = bboxEvidence(best.recognition, memory, frameW, frameH); + lastRunTimeMs = now; + lastResult = + new ReIDMatchResult( + best.score, + second, + best.index, + confirmedGallery.size(), + true, + SystemClock.elapsedRealtime() - start, + "fresh"); + return lastResult; + } + + private static BboxContinuityEvidence bboxEvidence( + Recognition recognition, TargetMemory memory, int frameW, int frameH) { + if (recognition == null || memory == null || recognition.getLocation() == null) { + return BboxContinuityEvidence.unavailable("candidate_not_available"); + } + return BboxContinuityEvidence.from( + recognition.getLocation(), memory.getLastBbox(), memory.getPreviousBbox(), frameW, frameH); + } + + private static List candidateRefs(List persons, Recognition legacyBest) { + List refs = new ArrayList<>(); + if (legacyBest != null) { + int idx = persons.indexOf(legacyBest); + if (idx >= 0) refs.add(new CandidateRef(idx, legacyBest)); + } + List all = new ArrayList<>(); + for (int i = 0; i < persons.size(); i++) { + Recognition r = persons.get(i); + if (r != null && r.getLocation() != null) all.add(new CandidateRef(i, r)); + } + all.sort( + Comparator.comparingDouble( + (CandidateRef ref) -> -area(ref.recognition.getLocation())) + .thenComparingInt(ref -> ref.index)); + for (CandidateRef ref : all) { + boolean exists = false; + for (CandidateRef current : refs) { + if (current.index == ref.index) { + exists = true; + break; + } + } + if (!exists) refs.add(ref); + if (refs.size() >= MAX_CANDIDATES) break; + } + return refs; + } + + private static Bitmap cropPerson(Bitmap frame, RectF bbox, float paddingRatio) { + int fw = frame.getWidth(); + int fh = frame.getHeight(); + float padX = bbox.width() * paddingRatio; + float padY = bbox.height() * paddingRatio; + int left = clamp((int) (bbox.left - padX), 0, fw - 1); + int top = clamp((int) (bbox.top - padY), 0, fh - 1); + int right = clamp((int) (bbox.right + padX), 1, fw); + int bottom = clamp((int) (bbox.bottom + padY), 1, fh); + int width = Math.max(1, right - left); + int height = Math.max(1, bottom - top); + return Bitmap.createBitmap(frame, left, top, width, height); + } + + private static List selectDiverse(List features, int k) { + List selected = new ArrayList<>(); + if (features.isEmpty()) return selected; + selected.add(features.get(0)); + while (selected.size() < k && selected.size() < features.size()) { + float bestDistance = -1f; + float[] best = null; + for (float[] candidate : features) { + if (selected.contains(candidate)) continue; + float nearestSimilarity = -1f; + for (float[] existing : selected) { + nearestSimilarity = Math.max(nearestSimilarity, dot(candidate, existing)); + } + float distance = 1f - nearestSimilarity; + if (distance > bestDistance) { + bestDistance = distance; + best = candidate; + } + } + if (best == null) break; + selected.add(best); + } + return selected; + } + + private static float maxSimilarity(float[] feature, List gallery) { + float best = -1f; + for (float[] g : gallery) best = Math.max(best, dot(feature, g)); + return Math.max(0f, best); + } + + private static float dot(float[] a, float[] b) { + int n = Math.min(a.length, b.length); + float sum = 0f; + for (int i = 0; i < n; i++) sum += a[i] * b[i]; + return sum; + } + + private static boolean same(Recognition a, Recognition b) { + return a != null && b != null && a == b; + } + + private static float area(RectF b) { + return Math.max(0f, b.width()) * Math.max(0f, b.height()); + } + + private static int clamp(int v, int lo, int hi) { + return v < lo ? lo : (v > hi ? hi : v); + } + + private static class CandidateRef { + final int index; + final Recognition recognition; + + CandidateRef(int index, Recognition recognition) { + this.index = index; + this.recognition = recognition; + } + } + + private static class CandidateScore { + final int index; + final Recognition recognition; + final float score; + + CandidateScore(int index, Recognition recognition, float score) { + this.index = index; + this.recognition = recognition; + this.score = score; + } + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/ReIDMatchResult.java b/android/robot/src/main/java/org/openbot/cartfollow/ReIDMatchResult.java new file mode 100644 index 000000000..a1f835331 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/ReIDMatchResult.java @@ -0,0 +1,47 @@ +package org.openbot.cartfollow; + +public class ReIDMatchResult { + public static final float BEST_WEAK = 0.75f; + public static final float MARGIN_WEAK = 0.03f; + public static final float BEST_MID = 0.80f; + public static final float MARGIN_MID = 0.05f; + public static final float BEST_STRONG = 0.85f; + public static final float MARGIN_STRONG = 0.05f; + + public final float bestScore; + public final float secondScore; + public final float margin; + public final int bestCandidateIndex; + public final int gallerySize; + public final boolean reidAvailable; + public final boolean weakOk; + public final boolean midOk; + public final boolean strongOk; + public final long latencyMs; + public final String reason; + + public ReIDMatchResult( + float bestScore, + float secondScore, + int bestCandidateIndex, + int gallerySize, + boolean reidAvailable, + long latencyMs, + String reason) { + this.bestScore = bestScore; + this.secondScore = secondScore; + this.margin = bestScore - secondScore; + this.bestCandidateIndex = bestCandidateIndex; + this.gallerySize = gallerySize; + this.reidAvailable = reidAvailable; + this.weakOk = reidAvailable && bestScore >= BEST_WEAK && margin >= MARGIN_WEAK; + this.midOk = reidAvailable && bestScore >= BEST_MID && margin >= MARGIN_MID; + this.strongOk = reidAvailable && bestScore >= BEST_STRONG && margin >= MARGIN_STRONG; + this.latencyMs = latencyMs; + this.reason = reason; + } + + public static ReIDMatchResult unavailable(String reason, int gallerySize) { + return new ReIDMatchResult(0f, 0f, -1, gallerySize, false, 0L, reason); + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/SystemSafetyEvidence.java b/android/robot/src/main/java/org/openbot/cartfollow/SystemSafetyEvidence.java new file mode 100644 index 000000000..c790fea2e --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/SystemSafetyEvidence.java @@ -0,0 +1,16 @@ +package org.openbot.cartfollow; + +public class SystemSafetyEvidence { + public final boolean emergencyStop; + public final boolean communicationOk; + public final boolean detectorOk; + public final String reason; + + public SystemSafetyEvidence( + boolean emergencyStop, boolean communicationOk, boolean detectorOk, String reason) { + this.emergencyStop = emergencyStop; + this.communicationOk = communicationOk; + this.detectorOk = detectorOk; + this.reason = reason; + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/TargetMemory.java b/android/robot/src/main/java/org/openbot/cartfollow/TargetMemory.java index 2953b609e..0207d6b20 100644 --- a/android/robot/src/main/java/org/openbot/cartfollow/TargetMemory.java +++ b/android/robot/src/main/java/org/openbot/cartfollow/TargetMemory.java @@ -21,6 +21,7 @@ public class TargetMemory { private float desiredBottomRatio; private RectF lastBbox; + private RectF previousBbox; private float lastCenterX; private float lastCenterY; private float lastArea; @@ -38,6 +39,7 @@ public void captureFromBitmap( upperColorHist = computeHsvHist(bitmap, upperHalf(bbox)); lowerColorHist = computeHsvHist(bitmap, lowerHalf(bbox)); lastBbox = new RectF(bbox); + previousBbox = null; lastCenterX = bbox.centerX(); lastCenterY = bbox.centerY(); lastArea = confirmedArea; @@ -83,6 +85,7 @@ public ImageSetpointDistanceEstimator.Setpoint getDistanceSetpoint() { public void updateDynamic(Recognition r) { if (r == null || r.getLocation() == null) return; RectF b = r.getLocation(); + if (lastBbox != null) previousBbox = new RectF(lastBbox); lastBbox = new RectF(b); lastCenterX = b.centerX(); lastCenterY = b.centerY(); @@ -100,6 +103,7 @@ public void clear() { desiredAreaRatio = 0f; desiredBottomRatio = 0f; lastBbox = null; + previousBbox = null; lastCenterX = 0f; lastCenterY = 0f; lastArea = 0f; @@ -114,6 +118,10 @@ public RectF getLastBbox() { return lastBbox != null ? new RectF(lastBbox) : (confirmedBbox != null ? new RectF(confirmedBbox) : null); } + public RectF getPreviousBbox() { + return previousBbox != null ? new RectF(previousBbox) : null; + } + public float getLastArea() { return lastArea > 0 ? lastArea : confirmedArea; } diff --git a/android/robot/src/main/java/org/openbot/cartfollow/TargetTrack.java b/android/robot/src/main/java/org/openbot/cartfollow/TargetTrack.java new file mode 100644 index 000000000..04a85e6e8 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/TargetTrack.java @@ -0,0 +1,56 @@ +package org.openbot.cartfollow; + +import android.graphics.RectF; +import org.openbot.tflite.Detector.Recognition; + +public class TargetTrack { + public final int trackId; + public RectF lastBbox; + public RectF previousBbox; + public Recognition recognition; + public int ageFrames; + public int missedFrames; + public int stableFrames; + public long lastSeenTimestampMs; + public String matchReason; + + TargetTrack(int trackId, Recognition recognition, long timestampMs) { + this.trackId = trackId; + this.recognition = recognition; + this.lastBbox = recognition == null ? null : new RectF(recognition.getLocation()); + this.previousBbox = null; + this.ageFrames = 1; + this.missedFrames = 0; + this.stableFrames = 1; + this.lastSeenTimestampMs = timestampMs; + this.matchReason = "new_track"; + } + + void update(Recognition recognition, long timestampMs, String reason) { + if (recognition == null || recognition.getLocation() == null) return; + if (lastBbox != null) previousBbox = new RectF(lastBbox); + lastBbox = new RectF(recognition.getLocation()); + this.recognition = recognition; + ageFrames++; + missedFrames = 0; + stableFrames++; + lastSeenTimestampMs = timestampMs; + matchReason = reason; + } + + void markMissed() { + ageFrames++; + missedFrames++; + stableFrames = 0; + recognition = null; + matchReason = "missed"; + } + + public boolean isVisible() { + return recognition != null && missedFrames == 0 && lastBbox != null; + } + + public float area() { + return lastBbox == null ? 0f : Math.max(0f, lastBbox.width()) * Math.max(0f, lastBbox.height()); + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/TargetTrackManager.java b/android/robot/src/main/java/org/openbot/cartfollow/TargetTrackManager.java new file mode 100644 index 000000000..cd4a5edd8 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/TargetTrackManager.java @@ -0,0 +1,177 @@ +package org.openbot.cartfollow; + +import android.graphics.RectF; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import org.openbot.tflite.Detector.Recognition; + +public class TargetTrackManager { + private static final float CENTER_GATE_RATIO = 0.25f; + private static final float AREA_RATIO_MIN = 0.50f; + private static final float AREA_RATIO_MAX = 2.00f; + private static final float IOU_SOFT_GATE = 0.10f; + private static final int MAX_MISSED_FRAMES = 12; + + private final List tracks = new ArrayList<>(); + private int nextTrackId = 1; + private int lockedTrackId = -1; + private int suspectedTrackId = -1; + + public void reset() { + tracks.clear(); + nextTrackId = 1; + lockedTrackId = -1; + suspectedTrackId = -1; + } + + public void update(List detections, int frameW, int frameH, long timestampMs) { + List safeDetections = detections == null ? new ArrayList<>() : detections; + Set matchedTracks = new HashSet<>(); + Set matchedDetections = new HashSet<>(); + + for (Recognition detection : safeDetections) { + if (detection == null || detection.getLocation() == null) continue; + Match best = null; + for (TargetTrack track : tracks) { + if (track.lastBbox == null || matchedTracks.contains(track)) continue; + Match match = score(track, detection, frameW, frameH); + if (!match.accepted) continue; + if (best == null || match.score > best.score) best = match; + } + if (best != null) { + best.track.update(detection, timestampMs, best.reason); + matchedTracks.add(best.track); + matchedDetections.add(detection); + } + } + + for (TargetTrack track : new ArrayList<>(tracks)) { + if (!matchedTracks.contains(track)) track.markMissed(); + if (track.missedFrames > MAX_MISSED_FRAMES) tracks.remove(track); + } + + for (Recognition detection : safeDetections) { + if (detection == null || detection.getLocation() == null || matchedDetections.contains(detection)) { + continue; + } + tracks.add(new TargetTrack(nextTrackId++, detection, timestampMs)); + } + } + + public List getTracks() { + return new ArrayList<>(tracks); + } + + public int getActiveTrackCount() { + int count = 0; + for (TargetTrack track : tracks) { + if (track.isVisible()) count++; + } + return count; + } + + public TargetTrack getTrackForRecognition(Recognition recognition) { + if (recognition == null) return null; + for (TargetTrack track : tracks) { + if (track.recognition == recognition) return track; + } + return null; + } + + public TargetTrack getLockedTrack() { + return getTrackById(lockedTrackId); + } + + public TargetTrack getTrackById(int trackId) { + if (trackId < 0) return null; + for (TargetTrack track : tracks) { + if (track.trackId == trackId) return track; + } + return null; + } + + public int getLockedTrackId() { + return lockedTrackId; + } + + public int getSuspectedTrackId() { + return suspectedTrackId; + } + + public void setSuspectedTrackId(int suspectedTrackId) { + this.suspectedTrackId = suspectedTrackId; + } + + public int lockClosest(RectF reference) { + TargetTrack best = null; + float bestScore = Float.MAX_VALUE; + for (TargetTrack track : tracks) { + if (!track.isVisible()) continue; + float score = reference == null ? -track.area() : centerDistance(track.lastBbox, reference); + if (best == null || score < bestScore) { + best = track; + bestScore = score; + } + } + lockedTrackId = best == null ? -1 : best.trackId; + suspectedTrackId = -1; + return lockedTrackId; + } + + public boolean isLockedTrack(TargetTrack track) { + return track != null && track.trackId == lockedTrackId; + } + + private static Match score(TargetTrack track, Recognition detection, int frameW, int frameH) { + RectF current = detection.getLocation(); + RectF last = track.lastBbox; + float diag = (float) Math.hypot(Math.max(1, frameW), Math.max(1, frameH)); + float centerJump = centerDistance(current, last) / Math.max(1f, diag); + float areaRatio = area(current) / Math.max(1f, area(last)); + float iou = iou(current, last); + boolean accepted = + (centerJump <= CENTER_GATE_RATIO && areaRatio >= AREA_RATIO_MIN && areaRatio <= AREA_RATIO_MAX) + || iou >= IOU_SOFT_GATE; + float score = iou * 2.0f - centerJump - Math.abs(1f - areaRatio) * 0.25f; + String reason = + String.format( + java.util.Locale.US, "iou=%.2f center=%.3f area=%.2f", iou, centerJump, areaRatio); + return new Match(track, accepted, score, reason); + } + + private static float centerDistance(RectF a, RectF b) { + if (a == null || b == null) return Float.MAX_VALUE; + return (float) Math.hypot(a.centerX() - b.centerX(), a.centerY() - b.centerY()); + } + + private static float area(RectF b) { + return b == null ? 0f : Math.max(0f, b.width()) * Math.max(0f, b.height()); + } + + private static float iou(RectF a, RectF b) { + if (a == null || b == null) return 0f; + float left = Math.max(a.left, b.left); + float top = Math.max(a.top, b.top); + float right = Math.min(a.right, b.right); + float bottom = Math.min(a.bottom, b.bottom); + float inter = Math.max(0f, right - left) * Math.max(0f, bottom - top); + float union = area(a) + area(b) - inter; + return union <= 0f ? 0f : inter / union; + } + + private static class Match { + final TargetTrack track; + final boolean accepted; + final float score; + final String reason; + + Match(TargetTrack track, boolean accepted, float score, String reason) { + this.track = track; + this.accepted = accepted; + this.score = score; + this.reason = reason; + } + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/TfliteReIDFeatureExtractor.java b/android/robot/src/main/java/org/openbot/cartfollow/TfliteReIDFeatureExtractor.java new file mode 100644 index 000000000..bb1f060f5 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/TfliteReIDFeatureExtractor.java @@ -0,0 +1,118 @@ +package org.openbot.cartfollow; + +import android.app.Activity; +import android.content.res.AssetFileDescriptor; +import android.graphics.Bitmap; +import java.io.FileInputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.MappedByteBuffer; +import java.nio.channels.FileChannel; +import org.tensorflow.lite.Interpreter; +import org.tensorflow.lite.Tensor; + +public class TfliteReIDFeatureExtractor implements ReIDFeatureExtractor { + public static final String DEFAULT_ASSET_PATH = "networks/reid/osnet_x0_25_market1501.tflite"; + + private static final float[] MEAN = {0.485f, 0.456f, 0.406f}; + private static final float[] STD = {0.229f, 0.224f, 0.225f}; + + private final Interpreter interpreter; + private final int inputWidth; + private final int inputHeight; + private final boolean nchw; + private final int embeddingSize; + private final int[] pixels; + private final ByteBuffer inputBuffer; + + public TfliteReIDFeatureExtractor(Activity activity, String assetPath, int numThreads) + throws IOException { + Interpreter.Options options = new Interpreter.Options(); + options.setNumThreads(numThreads > 0 ? numThreads : 2); + interpreter = new Interpreter(loadModelFile(activity, assetPath), options); + + Tensor inputTensor = interpreter.getInputTensor(0); + int[] inShape = inputTensor.shape(); + if (inShape.length != 4) { + throw new IOException("Unsupported ReID input shape length: " + inShape.length); + } + nchw = inShape[1] == 3; + inputHeight = nchw ? inShape[2] : inShape[1]; + inputWidth = nchw ? inShape[3] : inShape[2]; + if (inputWidth <= 0 || inputHeight <= 0) { + throw new IOException("Invalid ReID input shape."); + } + + int[] outShape = interpreter.getOutputTensor(0).shape(); + int size = 1; + for (int i = 1; i < outShape.length; i++) { + size *= Math.max(1, outShape[i]); + } + embeddingSize = Math.max(1, size); + pixels = new int[inputWidth * inputHeight]; + inputBuffer = ByteBuffer.allocateDirect(4 * inputWidth * inputHeight * 3); + inputBuffer.order(ByteOrder.nativeOrder()); + } + + @Override + public synchronized float[] extract(Bitmap personCrop) { + if (personCrop == null) return null; + Bitmap resized = Bitmap.createScaledBitmap(personCrop, inputWidth, inputHeight, true); + resized.getPixels(pixels, 0, inputWidth, 0, 0, inputWidth, inputHeight); + inputBuffer.rewind(); + if (nchw) { + for (int c = 0; c < 3; c++) { + for (int y = 0; y < inputHeight; y++) { + for (int x = 0; x < inputWidth; x++) { + inputBuffer.putFloat(normalizedChannel(pixels[y * inputWidth + x], c)); + } + } + } + } else { + for (int y = 0; y < inputHeight; y++) { + for (int x = 0; x < inputWidth; x++) { + int px = pixels[y * inputWidth + x]; + inputBuffer.putFloat(normalizedChannel(px, 0)); + inputBuffer.putFloat(normalizedChannel(px, 1)); + inputBuffer.putFloat(normalizedChannel(px, 2)); + } + } + } + float[][] output = new float[1][embeddingSize]; + inputBuffer.rewind(); + interpreter.run(inputBuffer, output); + return l2Normalize(output[0]); + } + + public void close() { + interpreter.close(); + } + + private static float normalizedChannel(int px, int channel) { + int v; + if (channel == 0) v = (px >> 16) & 0xFF; + else if (channel == 1) v = (px >> 8) & 0xFF; + else v = px & 0xFF; + return (v / 255f - MEAN[channel]) / STD[channel]; + } + + private static float[] l2Normalize(float[] values) { + float sum = 0f; + for (float v : values) sum += v * v; + float norm = (float) Math.sqrt(Math.max(sum, 1e-12f)); + for (int i = 0; i < values.length; i++) values[i] /= norm; + return values; + } + + private static MappedByteBuffer loadModelFile(Activity activity, String assetPath) + throws IOException { + AssetFileDescriptor fileDescriptor = activity.getAssets().openFd(assetPath); + FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor()); + FileChannel fileChannel = inputStream.getChannel(); + return fileChannel.map( + FileChannel.MapMode.READ_ONLY, + fileDescriptor.getStartOffset(), + fileDescriptor.getDeclaredLength()); + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/TraversabilityEvidence.java b/android/robot/src/main/java/org/openbot/cartfollow/TraversabilityEvidence.java new file mode 100644 index 000000000..7d9eb312e --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/TraversabilityEvidence.java @@ -0,0 +1,22 @@ +package org.openbot.cartfollow; + +public class TraversabilityEvidence { + public final float leftFreeScore; + public final float centerFreeScore; + public final float rightFreeScore; + public final boolean centerBlocked; + public final String reason; + + public TraversabilityEvidence( + float leftFreeScore, + float centerFreeScore, + float rightFreeScore, + boolean centerBlocked, + String reason) { + this.leftFreeScore = leftFreeScore; + this.centerFreeScore = centerFreeScore; + this.rightFreeScore = rightFreeScore; + this.centerBlocked = centerBlocked; + this.reason = reason; + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticConfig.java b/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticConfig.java new file mode 100644 index 000000000..3c0c6a56a --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticConfig.java @@ -0,0 +1,11 @@ +package org.openbot.cartfollow.diagnostics; + +public class CartFollowDiagnosticConfig { + public long frameLogIntervalMs = 200; + public long cropIntervalMs = 500; + public long overlayIntervalMs = 1000; + public boolean saveCrops = true; + public boolean saveOverlays = false; + public float paddingRatio = 0.08f; + public int jpegQuality = 90; +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticSaver.java b/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticSaver.java new file mode 100644 index 000000000..4d26a21d6 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticSaver.java @@ -0,0 +1,354 @@ +package org.openbot.cartfollow.diagnostics; + +import android.graphics.Bitmap; +import android.graphics.Matrix; +import android.graphics.RectF; +import java.io.File; +import java.io.FileOutputStream; +import java.io.FileWriter; +import java.io.IOException; +import java.util.Locale; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import org.openbot.cartfollow.BehaviorDecisionResult; +import org.openbot.cartfollow.BboxContinuityEvidence; +import org.openbot.cartfollow.IdentityEvidence; +import org.openbot.cartfollow.ReIDMatchResult; +import org.openbot.tflite.Detector.Recognition; +import timber.log.Timber; + +public class CartFollowDiagnosticSaver { + private final ExecutorService executor = Executors.newSingleThreadExecutor(); + + public void saveFrameAsync( + Bitmap frame, + CartFollowDiagnosticSession session, + CartFollowDiagnosticConfig config, + long frameNum, + int frameW, + int frameH, + int sensorOrientation, + float fps, + int numPersons, + String followState, + BehaviorDecisionResult decision, + String commandText, + IdentityEvidence identity, + Recognition locked, + Recognition suspected, + Recognition bestReid, + boolean saveCropsForThisFrame) { + if (session == null) return; + final long timestampMs = System.currentTimeMillis(); + final long elapsedMs = timestampMs - session.startedAtMs; + final String action = decision == null ? "" : decision.selectedAction.name(); + final String actionReason = decision == null ? "" : safe(decision.actionReason); + final String safetyBlock = + decision == null || decision.safetyBlockReason == null ? "" : safe(decision.safetyBlockReason); + final String safeCommand = safe(commandText); + final IdentitySnapshot identitySnapshot = new IdentitySnapshot(identity); + final RecognitionSnapshot lockedSnapshot = RecognitionSnapshot.from(locked); + final RecognitionSnapshot suspectedSnapshot = RecognitionSnapshot.from(suspected); + final RecognitionSnapshot bestSnapshot = RecognitionSnapshot.from(bestReid); + Bitmap.Config bitmapConfig = frame == null || frame.getConfig() == null ? Bitmap.Config.ARGB_8888 : frame.getConfig(); + final Bitmap frameCopy = + frame != null && config.saveCrops && saveCropsForThisFrame ? frame.copy(bitmapConfig, false) : null; + + executor.execute( + () -> { + String lockedPath = ""; + String suspectedPath = ""; + String bestPath = ""; + if (frameCopy != null) { + lockedPath = + saveCrop(frameCopy, lockedSnapshot, session, config, frameNum, "locked", sensorOrientation); + suspectedPath = + saveCrop(frameCopy, suspectedSnapshot, session, config, frameNum, "suspected", sensorOrientation); + bestPath = + saveCrop(frameCopy, bestSnapshot, session, config, frameNum, "best_reid", sensorOrientation); + frameCopy.recycle(); + } + appendFrameLog( + session, + frameNum, + timestampMs, + elapsedMs, + fps, + numPersons, + followState, + action, + actionReason, + safetyBlock, + safeCommand); + appendIdentityLog( + session, + frameNum, + timestampMs, + identitySnapshot, + lockedPath, + suspectedPath, + bestPath); + }); + } + + public void saveEventAsync( + CartFollowDiagnosticSession session, long frameNum, String eventType, String note) { + if (session == null) return; + final long timestampMs = System.currentTimeMillis(); + final String safeType = safe(eventType); + final String safeNote = safe(note); + executor.execute( + () -> { + String row = + String.format( + Locale.US, + "%s,%d,%d,%s,%s\n", + csv(session.sessionId), timestampMs, frameNum, csv(safeType), csv(safeNote)); + append(session.eventsCsv, row, "events.csv"); + session.eventRows++; + }); + } + + public void saveGallerySnapshotAsync( + Bitmap bitmap, CartFollowDiagnosticSession session, String label) { + if (bitmap == null || session == null) return; + Bitmap.Config bitmapConfig = bitmap.getConfig() == null ? Bitmap.Config.ARGB_8888 : bitmap.getConfig(); + final Bitmap copy = bitmap.copy(bitmapConfig, false); + final String safeLabel = sanitize(label == null ? "gallery" : label); + executor.execute( + () -> { + String filename = + String.format(Locale.US, "%06d_%s.jpg", session.galleryCount, safeLabel); + File file = new File(session.galleryDir, filename); + try (FileOutputStream fos = new FileOutputStream(file)) { + copy.compress(Bitmap.CompressFormat.JPEG, 90, fos); + session.galleryCount++; + } catch (IOException e) { + Timber.e(e, "Failed to save diagnostic gallery snapshot"); + } finally { + copy.recycle(); + } + }); + } + + public void shutdown() { + executor.shutdown(); + } + + private void appendFrameLog( + CartFollowDiagnosticSession session, + long frameNum, + long timestampMs, + long elapsedMs, + float fps, + int numPersons, + String followState, + String action, + String actionReason, + String safetyBlock, + String commandText) { + String row = + String.format( + Locale.US, + "%s,%d,%d,%d,%.2f,%d,%s,%s,%s,%s,%s\n", + csv(session.sessionId), + frameNum, + timestampMs, + elapsedMs, + fps, + numPersons, + csv(followState), + csv(action), + csv(actionReason), + csv(safetyBlock), + csv(commandText)); + append(session.frameLogCsv, row, "frame_log.csv"); + session.frameRows++; + } + + private void appendIdentityLog( + CartFollowDiagnosticSession session, + long frameNum, + long timestampMs, + IdentitySnapshot id, + String lockedPath, + String suspectedPath, + String bestPath) { + String row = + String.format( + Locale.US, + "%s,%d,%d,%d,%d,%d,%d,%d,%d,%.4f,%.4f,%.4f,%d,%s,%s,%s,%s,%s,%s,%.4f,%d,%d,%d,%s,%s,%s,%s,%s\n", + csv(session.sessionId), + frameNum, + timestampMs, + id.trackId, + id.lockedTrackId, + id.suspectedTrackId, + id.activeTrackCount, + id.trackAge, + id.missedFrames, + id.bestScore, + id.secondScore, + id.margin, + id.gallerySize, + id.weakOk ? "1" : "0", + id.midOk ? "1" : "0", + id.strongOk ? "1" : "0", + id.bboxDefaultOk ? "1" : "0", + id.bboxStrictOk ? "1" : "0", + id.predictionOk ? "1" : "0", + id.targetBelief, + id.beliefStableFrames, + id.beliefUncertainFrames, + id.candidateSwitchCount, + csv(id.beliefReason), + csv(id.reidReason), + csv(lockedPath), + csv(suspectedPath), + csv(bestPath)); + append(session.identityLogCsv, row, "identity_log.csv"); + session.identityRows++; + } + + private String saveCrop( + Bitmap frame, + RecognitionSnapshot snapshot, + CartFollowDiagnosticSession session, + CartFollowDiagnosticConfig config, + long frameNum, + String role, + int sensorOrientation) { + if (snapshot == null || snapshot.bbox == null) return ""; + RectF bbox = snapshot.bbox; + float padX = bbox.width() * config.paddingRatio; + float padY = bbox.height() * config.paddingRatio; + int left = clamp((int) (bbox.left - padX), 0, frame.getWidth() - 1); + int top = clamp((int) (bbox.top - padY), 0, frame.getHeight() - 1); + int right = clamp((int) (bbox.right + padX), left + 1, frame.getWidth()); + int bottom = clamp((int) (bbox.bottom + padY), top + 1, frame.getHeight()); + int w = right - left; + int h = bottom - top; + if (w <= 0 || h <= 0) return ""; + + Bitmap rawCrop; + try { + rawCrop = Bitmap.createBitmap(frame, left, top, w, h); + } catch (Exception e) { + return ""; + } + + Bitmap uprightCrop; + if (sensorOrientation % 360 != 0) { + Matrix matrix = new Matrix(); + matrix.postRotate(sensorOrientation); + uprightCrop = + Bitmap.createBitmap(rawCrop, 0, 0, rawCrop.getWidth(), rawCrop.getHeight(), matrix, true); + rawCrop.recycle(); + } else { + uprightCrop = rawCrop; + } + + String filename = String.format(Locale.US, "%06d_%s.jpg", frameNum, sanitize(role)); + File cropFile = new File(session.cropsDir, filename); + try (FileOutputStream fos = new FileOutputStream(cropFile)) { + uprightCrop.compress(Bitmap.CompressFormat.JPEG, config.jpegQuality, fos); + session.cropCount++; + return "crops/" + filename; + } catch (IOException e) { + Timber.e(e, "Failed to save diagnostic crop"); + return ""; + } finally { + uprightCrop.recycle(); + } + } + + private void append(File file, String row, String label) { + try (FileWriter writer = new FileWriter(file, true)) { + writer.append(row); + } catch (IOException e) { + Timber.e(e, "Failed to append %s", label); + } + } + + private static int clamp(int value, int min, int max) { + return Math.max(min, Math.min(max, value)); + } + + private static String csv(String value) { + if (value == null) return ""; + return "\"" + value.replace("\"", "\"\"") + "\""; + } + + private static String safe(String value) { + return value == null ? "" : value; + } + + private static String sanitize(String value) { + return safe(value).replaceAll("[^a-zA-Z0-9_\\-]", "_"); + } + + private static class RecognitionSnapshot { + final RectF bbox; + + RecognitionSnapshot(RectF bbox) { + this.bbox = bbox; + } + + static RecognitionSnapshot from(Recognition recognition) { + if (recognition == null || recognition.getLocation() == null) return null; + return new RecognitionSnapshot(new RectF(recognition.getLocation())); + } + } + + private static class IdentitySnapshot { + final int trackId; + final int lockedTrackId; + final int suspectedTrackId; + final int activeTrackCount; + final int trackAge; + final int missedFrames; + final float bestScore; + final float secondScore; + final float margin; + final int gallerySize; + final boolean weakOk; + final boolean midOk; + final boolean strongOk; + final boolean bboxDefaultOk; + final boolean bboxStrictOk; + final boolean predictionOk; + final float targetBelief; + final int beliefStableFrames; + final int beliefUncertainFrames; + final int candidateSwitchCount; + final String beliefReason; + final String reidReason; + + IdentitySnapshot(IdentityEvidence identity) { + ReIDMatchResult reid = identity == null ? null : identity.reidMatch; + BboxContinuityEvidence bbox = identity == null ? null : identity.bboxContinuity; + trackId = identity == null ? -1 : identity.trackId; + lockedTrackId = identity == null ? -1 : identity.lockedTrackId; + suspectedTrackId = identity == null ? -1 : identity.suspectedTrackId; + activeTrackCount = identity == null ? 0 : identity.activeTrackCount; + trackAge = identity == null ? 0 : identity.trackAge; + missedFrames = identity == null ? 0 : identity.missedFrames; + bestScore = reid == null ? 0f : reid.bestScore; + secondScore = reid == null ? 0f : reid.secondScore; + margin = reid == null ? 0f : reid.margin; + gallerySize = reid == null ? 0 : reid.gallerySize; + weakOk = identity != null && identity.weakOk(); + midOk = identity != null && identity.midOk(); + strongOk = identity != null && identity.strongOk(); + bboxDefaultOk = bbox != null && bbox.bboxDefaultOk; + bboxStrictOk = bbox != null && bbox.bboxStrictOk; + predictionOk = bbox != null && bbox.predictionOk; + targetBelief = identity == null ? 0f : identity.targetBelief; + beliefStableFrames = identity == null ? 0 : identity.beliefStableFrames; + beliefUncertainFrames = identity == null ? 0 : identity.beliefUncertainFrames; + candidateSwitchCount = identity == null ? 0 : identity.candidateSwitchCount; + beliefReason = identity == null || identity.beliefReason == null ? "" : identity.beliefReason; + reidReason = reid == null || reid.reason == null ? "" : reid.reason; + } + } +} diff --git a/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticSession.java b/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticSession.java new file mode 100644 index 000000000..eb0ee3b89 --- /dev/null +++ b/android/robot/src/main/java/org/openbot/cartfollow/diagnostics/CartFollowDiagnosticSession.java @@ -0,0 +1,118 @@ +package org.openbot.cartfollow.diagnostics; + +import android.content.Context; +import android.os.Build; +import android.os.Environment; +import android.util.Log; +import java.io.File; +import java.io.FileWriter; +import java.io.IOException; +import java.text.SimpleDateFormat; +import java.util.Date; +import java.util.Locale; +import org.json.JSONException; +import org.json.JSONObject; + +public class CartFollowDiagnosticSession { + private static final String TAG = "CartFollowDiagnostic"; + + private static final String FRAME_LOG_HEADER = + "session_id,frame_id,timestamp_ms,elapsed_ms,fps,num_persons,follow_state," + + "selected_action,action_reason,safety_block_reason,command_text"; + private static final String IDENTITY_LOG_HEADER = + "session_id,frame_id,timestamp_ms,track_id,locked_track_id,suspected_track_id," + + "active_track_count,track_age,missed_frames,best_score,second_score,margin," + + "gallery_size,weak_ok,mid_ok,strong_ok,bbox_default_ok,bbox_strict_ok," + + "prediction_ok,target_belief,belief_stable_frames,belief_uncertain_frames," + + "candidate_switch_count,belief_reason,reid_reason,locked_crop_path," + + "suspected_crop_path,best_reid_crop_path"; + private static final String EVENTS_HEADER = "session_id,timestamp_ms,frame_id,event_type,note"; + + public final String sessionId; + public final File sessionDir; + public final File cropsDir; + public final File galleryDir; + public final File overlaysDir; + public final File frameLogCsv; + public final File identityLogCsv; + public final File eventsCsv; + public final long startedAtMs; + + public int frameRows = 0; + public int identityRows = 0; + public int eventRows = 0; + public int cropCount = 0; + public int galleryCount = 0; + + public CartFollowDiagnosticSession(Context context) { + this.startedAtMs = System.currentTimeMillis(); + String timestamp = new SimpleDateFormat("yyyyMMdd_HHmmss", Locale.US).format(new Date()); + this.sessionId = "cart_diag_" + timestamp; + File baseDir = + new File( + context.getExternalFilesDir(Environment.DIRECTORY_PICTURES), + "cartfollow_diagnostics"); + this.sessionDir = new File(baseDir, sessionId); + this.cropsDir = new File(sessionDir, "crops"); + this.galleryDir = new File(sessionDir, "gallery"); + this.overlaysDir = new File(sessionDir, "overlays"); + this.cropsDir.mkdirs(); + this.galleryDir.mkdirs(); + this.overlaysDir.mkdirs(); + this.frameLogCsv = new File(sessionDir, "frame_log.csv"); + this.identityLogCsv = new File(sessionDir, "identity_log.csv"); + this.eventsCsv = new File(sessionDir, "events.csv"); + } + + public void initCsvFiles() { + writeHeader(frameLogCsv, FRAME_LOG_HEADER); + writeHeader(identityLogCsv, IDENTITY_LOG_HEADER); + writeHeader(eventsCsv, EVENTS_HEADER); + } + + public void writeSessionInfo( + CartFollowDiagnosticConfig config, + String detectorModelName, + float minConfidence, + boolean reidAvailable, + int gallerySize) { + JSONObject json = new JSONObject(); + try { + json.put("session_id", sessionId); + json.put("created_at", new SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US).format(new Date())); + json.put("collector", "CartFollowDiagnostic"); + json.put("app_mode", "HumanCartSimulator"); + json.put("phase", "phase_c_diagnostic"); + json.put("frame_log_interval_ms", config.frameLogIntervalMs); + json.put("crop_interval_ms", config.cropIntervalMs); + json.put("overlay_interval_ms", config.overlayIntervalMs); + json.put("save_crops", config.saveCrops); + json.put("save_overlays", config.saveOverlays); + json.put("bbox_padding_ratio", config.paddingRatio); + json.put("jpeg_quality", config.jpegQuality); + json.put("detector", detectorModelName == null ? "" : detectorModelName); + json.put("min_confidence", minConfidence); + json.put("reid_available", reidAvailable); + json.put("gallery_size", gallerySize); + json.put("device_model", Build.MODEL == null ? "" : Build.MODEL); + json.put("sdk_int", Build.VERSION.SDK_INT); + } catch (JSONException e) { + Log.e(TAG, "Failed to build session_info.json", e); + return; + } + + try (FileWriter writer = new FileWriter(new File(sessionDir, "session_info.json"), false)) { + writer.write(json.toString(2)); + } catch (IOException | JSONException e) { + Log.e(TAG, "Failed to write session_info.json", e); + } + } + + private void writeHeader(File file, String header) { + try (FileWriter writer = new FileWriter(file, false)) { + writer.append(header).append('\n'); + } catch (IOException e) { + Log.e(TAG, "Failed to init csv: " + file.getName(), e); + } + } +} diff --git a/android/robot/src/main/res/layout/fragment_human_cart_simulator.xml b/android/robot/src/main/res/layout/fragment_human_cart_simulator.xml index 1ad1b22f4..3d4c16ecb 100644 --- a/android/robot/src/main/res/layout/fragment_human_cart_simulator.xml +++ b/android/robot/src/main/res/layout/fragment_human_cart_simulator.xml @@ -31,20 +31,56 @@ app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent" /> - + app:layout_constraintTop_toTopOf="parent"> + + + + + +