feat(databases): Import Data modal — samples, CSV/JSON files, and URL loads

[dawsontoth]

Closes #1292.

What

Replaces the bare-bones Import CSV modal with an Import Data flow modeled on the issue's mockups, with three methods:

Method	Backend operation
Sample Datasets — three bundled starter datasets (Dogs, Books, Animals), plus Random Data generation for the selected table	csv_data_load / insert
Import from File — CSV or JSON upload (was CSV-only)	csv_data_load / insert
Load from URL — CSV fetched by the instance	csv_url_load

UX details

• Front and center: a second entry point in the Databases sidebar (next to "Create a Table"), so import is available even on an empty database — the key onboarding moment the issue is about. The table-toolbar button ("Import CSV" → "Import Data") opens the same modal prefilled with the current table and defaults to the file method; the sidebar defaults to samples.
• Random Data: when the target table already exists and has columns beyond the primary key/system timestamps, the dataset picker offers a "Random Data" option (with a 1–1000 row-count input). Values are driven by each column's declared type, with column-name heuristics (email, name, age, price, boolean-ish, dates, …) for schemaless tables; the primary key is omitted so Harper assigns it.
• Tables are created automatically (primary key id) when the target doesn't exist, with an inline heads-up alert. Picking a sample dataset auto-fills its suggested table name.
• Job tracking: CSV loads return a job_id that the old modal ignored ("Please wait a few moments then refresh the table"). The new flow polls get_job until the job completes, then refreshes the table tree and navigates to the imported table — data is just there when the modal closes. Job errors (e.g. malformed CSV) surface as real error toasts.
• Sample datasets are bundled into the app via ?raw imports (~6 KB total), so they work on self-hosted/air-gapped instances that can't fetch external URLs.

Scope notes

The mockup's Connect to Database (mongodump, .zip, live sync) has no supporting operations in Harper today, so it's intentionally left out rather than shipped as a dead-end UI; the method picker makes it easy to add once the backend exists. import_from_s3 is also a candidate for a follow-up method.

Testing

• New unit tests: get_job polling (waitForJob), JSON upload parsing, random-record generation (types, heuristics, skipped columns), and sample-dataset CSV validity (format, unique ids, consistent columns). Full suite green; tsc, oxlint, dprint clean.
• Verified end-to-end against a stage test cluster: the Dogs sample import created data.dog and loaded 15 records (create_table → csv_data_load → get_job polling → auto-navigate), and Random Data then inserted 7 generated rows into that table with sensible per-column values. Also exercised both entry points, method switching, dataset auto-fill, and the new-table alert.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request replaces the CSV-only import modal with a comprehensive ImportDataModal that supports importing data from local CSV/JSON files, remote URLs, and pre-loaded sample datasets. It also introduces utility functions for parsing JSON records, a job polling mechanism to track asynchronous imports, and corresponding unit tests. Feedback on these changes includes improving the resilience of the job polling loop against transient network errors, handling potential FileReader failures when uploading files, and adding an error handler to the import mutation to provide user feedback on failures.

[kriszyp]

Nice modal — samples/file/URL import is a genuinely useful add. A couple of things worth addressing, plus one note:

Significant (client-side):

1. Unbounded file read. The file path does a whole-file FileReader read with no size cap or streaming, so a large CSV/JSON will spike memory and freeze the tab. Consider a size limit with a clear error, and/or streaming for the large case.
2. Import-status poll retry gap. The status poll doesn't retry robustly on a transient failure, so a blip can leave the modal stuck / misreport the import. A bounded retry (or clear terminal-error surfacing) would firm it up.

Note (not a regression): the URL-load path is a server-side csv_url_load — the SSRF surface is server-side and pre-existing; this UI exposes it but doesn't broaden it. Flagging so it's on the radar (a server-side allowlist/guardrail is the real fix), not something to solve in this PR.

Two minor suggestions are in the review file. Thanks @dawsontoth!

— 🤖 KrAIs (Kris's review assistant)

[dawsontoth]

Thanks @kriszyp! Point-by-point:

1. Unbounded file read — fixed in 828e5d2: uploads are now capped at 10 MB with an inline error showing the actual file size and pointing at Load from URL for anything bigger (the whole file goes into one operation body anyway, so the server wouldn't enjoy a 200 MB POST either). Verified in-browser with a synthetic 11 MB file (rejected, input cleared) and a small file (accepted). Streaming is a fair follow-up, but for >10 MB csv_url_load/import_from_s3 are the better transport.
2. Poll retry gap — already addressed in 0772a77 (from the earlier bot review): waitForJob tolerates transient get_job failures, bounded at 5 consecutive so persistent failures surface quickly with a real error rather than a stuck modal. Unit-tested for both the recovery and give-up paths.
3. SSRF note — agreed on all counts: server-side and pre-existing surface, and the right fix is a server-side guardrail. The UI hint already tells users the URL is fetched by the instance; noting it in the tracking sense here. 👍

🤖 Addressed by Claude Code

[kriszyp]

Reviewed via Claude review-queue. Solid rewrite of the CSV-only import modal into a three-method Import Data flow (samples/file/URL) with a sensible file-size cap and decent test coverage on the pure helpers. Two things worth addressing before merge:

Blocker: waitForJob can report success before the import job has started. getJob.ts (~line 48) resolves as soon as job.status !== 'IN_PROGRESS', treating anything else as terminal (only ERROR is special-cased). Harper's actual job lifecycle is CREATED → IN_PROGRESS → COMPLETE/ERROR — if the first poll lands while the job is still CREATED (plausible right after submission, before a worker picks it up), the function returns immediately as if the import succeeded. Fix: check terminal states explicitly (status === 'COMPLETE' || status === 'ERROR') instead of the inverse of the non-terminal state. No test currently exercises a CREATED-first sequence, so this isn't caught.

Significant: the importData mutation has no onError handler (ImportDataModal.tsx ~line 676). useMutation swallows the rejection into isError/error state with nothing reading it — a failed create_table, malformed CSV, or waitForJob timeout/rejection all fail silently. The button re-enables but there's no toast or inline message; the modal just looks idle.

Everything else looks good — the 10MB file-size cap with a streamed-read note is the right call, and the job-tracking replacement for the old "refresh manually" UX is a real improvement once #1 above is fixed.

[dawsontoth]

Thanks @kriszyp — the blocker was a real bug. Both points addressed in beb4bab:

1. waitForJob CREATED gap — confirmed and fixed. Terminal states are now explicit (COMPLETE returns, ERROR throws, everything else — including CREATED and any unknown status — keeps polling until the deadline). Added a unit test for the CREATED → IN_PROGRESS → COMPLETE sequence that fails against the old inverse check.
2. Missing onError — intentional, and now pinned by a test. This repo routes every mutation error through a global MutationCache → errorHandler (src/react-query/queryClient.ts), which toasts the extracted message — a per-mutation onError here would double-toast, and no other modal in the repo adds one. Since this is the second review to flag it, beb4bab adds a regression test (queryClient.test.ts) asserting a failing mutation on the app's real queryClient produces the error toast, so the "silent failure" scenario is guarded against the wiring ever being removed.

🤖 Addressed by Claude Code

[kriszyp]

Reviewed via Claude review-queue — both prior findings (premature CREATED-status resolution in waitForJob, missing onError handler) are fixed and now test-pinned. LGTM.