Skip to content

fix: get_partitions_to_optimize should return a Sequence, not Iterable #405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tkaemming merged 4 commits into from
Aug 9, 2019
Merged

fix: get_partitions_to_optimize should return a Sequence, not Iterable #405

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
205bcf4
fix: `get_partitions_to_optimize` should return a Sequence, not Iterable
tkaemming Aug 9, 2019
beec922
annotate
tkaemming Aug 9, 2019
dfde7de
Merge branch 'master' into python3-filter
tkaemming Aug 9, 2019
36e209f
Merge branch 'master' into python3-filter
tkaemming Aug 9, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions snuba/optimize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from datetime import timedelta
from typing import Sequence
import logging

from snuba import util
Expand All @@ -13,7 +14,7 @@ def run_optimize(clickhouse, database, table, before=None):
return len(parts)


def get_partitions_to_optimize(clickhouse, database, table, before=None):
def get_partitions_to_optimize(clickhouse, database, table, before=None) -> Sequence[util.Part]:
engine = clickhouse.execute("""
SELECT engine
FROM system.tables
Expand Down Expand Up @@ -69,7 +70,7 @@ def get_partitions_to_optimize(clickhouse, database, table, before=None):
parts = [util.decode_part_str(part) for part, count in active_parts]

if before:
parts = filter(lambda p: (p[0] + timedelta(days=6 - p[0].weekday())) < before, parts)
parts = [p for p in parts if (p[0] + timedelta(days=6 - p[0].weekday())) < before]

@tkaemming tkaemming Aug 9, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this change, but preserving the annotations::

(snuba) ted@veneno % python -m mypy --follow-imports skip snuba/optimize.py
snuba/optimize.py:73: error: Incompatible types in assignment (expression has type "Iterator[Part]", variable has type "List[Part]")


return parts

Expand Down
10 changes: 8 additions & 2 deletions snuba/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from functools import wraps
from hashlib import md5
from itertools import chain, groupby
from typing import NamedTuple
import jsonschema
import logging
import numbers
Expand Down Expand Up @@ -636,15 +637,20 @@ def wrapper(*args, **kwargs):
return decorator


def decode_part_str(part_str):
class Part(NamedTuple):
date: datetime

@tkaemming tkaemming Aug 9, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a date and not a datetime, but I haven't checked to see if that would break anything that expects it to be one and not the other.

retention_days: int


def decode_part_str(part_str: str) -> Part:
match = PART_RE.match(part_str)
if not match:
raise ValueError("Unknown part name/format: " + str(part_str))

date_str, retention_days = match.groups()
date = datetime.strptime(date_str, '%Y-%m-%d')

return (date, int(retention_days))
return Part(date, int(retention_days))


def force_bytes(s):
Expand Down