feat(cdc) Set up structure to load snapshot into Clickhouse#398
Conversation
| """ | ||
| return column_name | ||
|
|
||
| def get_bulk_loader(self, source, dest_table): |
There was a problem hiding this comment.
Passing the source in input allows us to abstract the source away from the process to insert data into clickhouse. It also allows us to reuse the same instance if we decide to load multiple datasets at the same time.
| help='The dataset to bulk load') | ||
| @click.option('--source', | ||
| help='Source of the dump. Depending on the dataset it may have different meaning.') | ||
| @click.option('--dest-table', |
There was a problem hiding this comment.
I'd like to make this more abstract but I believe we need to see how we will use this in prod before deciding wether we can hardcode the temporary table name in the dataset like the other table names. Same if we decide to put it into config. I would address that once we tried this in prod.
| """ | ||
|
|
||
| @classmethod | ||
| def load(cls, path: str, product: str) -> BulkLoadSource: |
There was a problem hiding this comment.
hey @tkaemming look! type hints in Snuba !
(will take care of mypy separately)
tkaemming
left a comment
tkaemming
left a comment
There was a problem hiding this comment.
Added some initial thoughts, generally this looks pretty good.
| 'pylint==1.9.2', | ||
| 'pyparsing==2.2.0', | ||
| 'pytest==3.6.1', | ||
| 'pytest==3.9.3', |
There was a problem hiding this comment.
Was this upgraded for a specific reason, or just trying to stay current?
There was a problem hiding this comment.
it is needed in order to use the "tmp_path" fixture in tests. It was introduced in 3.9
| import sentry_sdk | ||
|
|
||
| sentry_sdk.init(dsn=settings.SENTRY_DSN) | ||
| logging.basicConfig(level=getattr(logging, log_level.upper()), format='%(asctime)s %(message)s') |
There was a problem hiding this comment.
We should refactor this later so that the SDK doesn't need to be installed in each cli script independently.
| """ | ||
| return column_name | ||
|
|
||
| def get_bulk_loader(self, source, dest_table): |
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class PostgresSnapshotDescriptor(SnapshotDescriptor): |
## Bump sentry-conventions to 0.11.0 Updates `sentry-conventions` from 0.10.0 to [0.11.0](https://github.com/getsentry/sentry-conventions/releases/tag/0.11.0). ### Changes in 0.11.0 **New Features** - Add `session.id` attribute (#412) - Update several attributes to latest OTel representation (#418) - Add `db.operation.batch.size` (#407) - Add `app.vitals.start.prewarmed` attribute (#379) - Add and deprecate runtime context attributes (#383) - Add Android Runtime (ART) GC and memory attributes (#382) - Add `db.stored_procedure.name` and ensure span name attribute consistency (#398) - Add Cloudflare SDK attributes (#392) - Add missing GCP and FaaS attributes (#403) **Bug Fixes** - Set `pii: 'maybe'` on `faas` string attributes (#415) - Add missing Cloudflare visibility attributes (#408) **Internal** - Remove `sdks` field from attribute schema and definitions (#410)
2 participants
This is the first step needed to be able to bulk load data into clickhouse.
The basic principle is that we will load data from files (one per table) into Snuba datasets, which, at present, are represented by one Clickhouse table. We are going to load data into a temp table and then do the swap.
This introduces: