[WIP] Improve data integrity and reliability of ingestion

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>Improve data integrity and reliability of ingestion</issue_title>
<issue_description>### Problem

• The ingestion job re‑queue endpoint uses a complex conditional WHERE clause to avoid resetting jobs that are still locked. If a worker crashes without clearing its lock, the lock expires based on WORKER_STALE_AFTER_MINUTES. Under heavy load, race conditions or stale locks could leave jobs stuck indefinitely.
• The import script caches existing documents by content hash and warns if more than 5 000 documents match. It only compares names and hashes in memory; identical file names with different content may slip through or legitimate differences may be flagged.
• The Supabase schema uses vector(1536) columns and warns that any change to EMBEDDING_DIMENSIONS or the OpenAI embedding model must be synchronised across migrations and worker environment variables. Misconfiguration will cause runtime failures.

Suggestions

• Use Postgres advisory locks or a heartbeat mechanism so workers can renew their locks periodically. This provides stronger guarantees that stale locks are cleaned up.
• Create a unique constraint on (owner_id, content_hash) to enforce deduplication at the database level. Remove reliance on in‑memory caches for duplicate detection.
• Add automated CI checks that validate embedding‑dimension consistency between the schema, migrations and worker configuration. Provide a migration script to alter vector dimensions safely when upgrading models.
  </issue_description>
• Closes Improve data integrity and reliability of ingestion #55

<agent_instructions>Please implement logical and efficient fixes to these issues. </agent_instructions>

Comments on the Issue (you are @copilot in this section)