Reframe post opening: several years of Iceberg, last year at data-heavy org#71
Conversation
…ta-heavy org Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
'the wrong choices interact' - rephrase pls
There was a problem hiding this comment.
is this on any catalog? if not, specify.
|
|
||
| **Phase 5 — Plan.** The winning scenario becomes a concrete plan: engine-specific commands with exact parameters, an execution order (ingestion tuning first, then layout, then maintenance), a schedule cadence, and monitoring thresholds that tell you when the next optimization cycle is due. | ||
|
|
||
| The skill handles Spark, Trino, AWS Glue/EMR, Snowflake, and Flink/Kafka Connect — each with engine-specific syntax, because the same compaction operation looks very different across these engines. |
There was a problem hiding this comment.
perhaps use duckdb as well, good for local work and has new iceberg compatabilities
There was a problem hiding this comment.
Great for the upcoming versions.
There was a problem hiding this comment.
| The benchmark scores each plan with an LLM judge evaluating correctness, specificity, and safety. **All 22 passed with a perfect 5/5 average.** |
There was a problem hiding this comment.
a bit worrying that it passed all tests with flying colours. would check for over/under fit to skill code. did you generate this from a different session / model?
There was a problem hiding this comment.
The benchmark runner spins a fresh session for each test.
There was a problem hiding this comment.
it never connects directly to your warehouse
then how does it fetch the metadata?
There was a problem hiding this comment.
Theoretically, you could pull out the metadata and query log and give it to the model. But you are right that ideally it connects directly to your table, ingestion pipeline and query engine and pulls the metadata independently.
- Rephrase 'wrong choices interact' → 'mistakes don't fail independently' - Clarify metadata access: you export it yourself; direct connectivity is roadmap - Note skill is available on GitHub - We→I (personal blog voice) in benchmarks section - 5.0/5 → 5/5 per reviewer suggestion - Add DuckDB + direct connectivity to v0.1 roadmap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Clarify that the skill supports both direct connectivity to the table/pipeline/engine and manual metadata export - Remove direct connectivity from the "what's missing" list (it's already available) - Drop DuckDB-specific callout; generalize to "other query engines such as DuckDB" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014sy3CvoMeptEkgif3MM7Jh
3 participants
One-line opening edit on the Iceberg Optimizer Skill post.
Before:
After:
This separates the broader experience claim (several years with Iceberg) from the specific recent engagement (last year, one org), which is more accurate and sets up the post's authority more clearly.
🤖 Generated with Claude Code
https://claude.ai/code/session_014sy3CvoMeptEkgif3MM7Jh
Generated by Claude Code