feat(dataqualityrule): added data quality rule support#66
Open
jacopocinaark wants to merge 8 commits into
Open
feat(dataqualityrule): added data quality rule support#66jacopocinaark wants to merge 8 commits into
jacopocinaark wants to merge 8 commits into
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…Klab/Artesian.SDK-Python into feature/22668-DataQualityRule
There was a problem hiding this comment.
Pull request overview
Adds Data Quality Rule (DQR) support to the Python SDK’s MarketData surface, including DTOs/enums, MarketDataService endpoints, tests, and usage docs/samples.
Changes:
- Added
MarketDataServiceCRUD APIs for data quality rules and rule assignments, plus an assignment events feed endpoint. - Introduced Data Quality DTOs/enums (rule types, schedules, outlier models, paged results, assignments, events, status summary).
- Extended tests, README documentation, and added runnable samples for rule/assignment flows.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/TestMarketDataService.py | Adds unit tests for DQR CRUD and assignment CRUD. |
| src/Artesian/MarketData/MarketDataService.py | Adds DQR and assignment endpoints to the service, including an events feed method. |
| src/Artesian/MarketData/_Enum/ScheduleDefinitionType.py | Adds schedule definition discriminator enum. |
| src/Artesian/MarketData/_Enum/RuleType.py | Adds DQ rule type enum (CompletenessAndFreshness/Outlier). |
| src/Artesian/MarketData/_Enum/PeriodPrecision.py | Adds precision enum used by period-based configs. |
| src/Artesian/MarketData/_Enum/OutlierModel.py | Adds outlier model discriminator enum. |
| src/Artesian/MarketData/_Enum/MarketDataTypeV2.py | Adds “v2” market data type enum for DQ configs. |
| src/Artesian/MarketData/_Enum/CheckAggregatedStatus.py | Adds aggregated status enum (OK/KO). |
| src/Artesian/MarketData/_Enum/init.py | Updates enum package exports (currently incomplete for new public enums). |
| src/Artesian/MarketData/_Dto/VersionedCompletenessAndFreshnessConfigDto.py | Adds versioned completeness/freshness config DTO. |
| src/Artesian/MarketData/_Dto/ScheduleDefinitionDto.py | Adds base schedule definition DTO abstraction. |
| src/Artesian/MarketData/_Dto/ScheduleConfigDto.py | Adds schedule config DTO (definition + maxDelay). |
| src/Artesian/MarketData/_Dto/RecordValidationConfigDto.py | Adds record validation window DTO. |
| src/Artesian/MarketData/_Dto/PagedResult.py | Adds paged result wrappers for DQRs and assignments. |
| src/Artesian/MarketData/_Dto/OutlierRefCurveConfigDto.py | Adds reference-curve outlier model config DTO. |
| src/Artesian/MarketData/_Dto/OutlierModelConfigDto.py | Adds base outlier model config DTO (currently has constructor issue). |
| src/Artesian/MarketData/_Dto/OutlierConfigDto.py | Adds outlier rule configuration DTO. |
| src/Artesian/MarketData/_Dto/OutlierAbsoluteBoundConfigDto.py | Adds absolute-bounds outlier model config DTO. |
| src/Artesian/MarketData/_Dto/MarketDataQualityRuleAssignmentDto.py | Adds rule assignment DTOs (input/output). |
| src/Artesian/MarketData/_Dto/DqCheckChangeEventDto.py | Adds DQ change-event DTOs for assignment event feed. |
| src/Artesian/MarketData/_Dto/DataQualityStatusSummaryDto.py | Adds status summary DTO (currently has keyword/serialization mapping risk). |
| src/Artesian/MarketData/_Dto/DataQualityRuleDtoOutput.py | Adds rule output DTO (adds aggregatedStatus). |
| src/Artesian/MarketData/_Dto/DataQualityRuleDtoInput.py | Adds rule input DTO. |
| src/Artesian/MarketData/_Dto/DataQualityRuleConfigDto.py | Adds base config DTO with type discriminator. |
| src/Artesian/MarketData/_Dto/CronScheduleDefinitionDto.py | Adds cron-based schedule definition DTO. |
| src/Artesian/MarketData/_Dto/CompletenessAndFreshnessConfigDto.py | Adds completeness/freshness base config DTO. |
| src/Artesian/MarketData/_Dto/ActualCompletenessAndFreshnessConfigDto.py | Adds “actual time series” completeness/freshness config DTO. |
| src/Artesian/MarketData/_Dto/init.py | Exposes new DTOs via the DTO package exports. |
| samples/TestDataQualityAssignment.py | Adds a manual end-to-end sample for rule assignment lifecycle. |
| samples/TestDataQuality.py | Adds a manual sample for rule CRUD lifecycle. |
| README.md | Documents Data Quality Rules usage and updates formatting in other sections. |
Comments suppressed due to low confidence (2)
src/Artesian/MarketData/MarketDataService.py:802
marketDataIdandruleIdare always added to query params even when None. This risks sendingmarketDataId=None/ruleId=None; omit them when not provided.
params["page"] = page
params["pageSize"] = pageSize
params["marketDataId"] = marketDataId
params["ruleId"] = ruleId
if ruleName:
src/Artesian/MarketData/MarketDataService.py:795
- Pagination validation error messages contain grammatical errors ("must to be") and report constraints inconsistently (code enforces 1-based pages). Consider clearer, structured messages.
if pageSize < 1:
raise ValueError(
"PageSize must to be greater than 0. Page Size:" + str(pageSize)
)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+7
to
+17
| @dataclass | ||
| class OutlierModelConfigDto(DataQualityRuleConfigDto): | ||
| """ | ||
| Base configuration for outlier detection rules. | ||
| """ | ||
|
|
||
| @property | ||
| def model(self: "OutlierModelConfigDto") -> OutlierModel: | ||
| raise NotImplementedError( | ||
| "OutlierModelConfigDto.model must be implemented by subclasses" | ||
| ) |
Comment on lines
+30
to
+35
| lastCheckTime: Optional[datetime] = None | ||
| overallStatus: Optional[CheckAggregatedStatus] = None | ||
| activeRulesCount: int = 0 | ||
| failedRulesCount: int = 0 | ||
| from_: Optional[date] = None | ||
| to: Optional[date] = None |
Comment on lines
+951
to
+955
| async def readDataQualityRuleAssignmentEventsFeedAsync( | ||
| self: MarketDataService, | ||
| id: int, | ||
| afterTimestamp: Optional[datetime] = None, | ||
| ) -> List[DqCheckChangeEventDtoOutput]: |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 31 out of 31 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (4)
src/Artesian/MarketData/MarketDataService.py:527
marketDataIdis always added to query params, even when it is None. Withrequests, this can result inmarketDataId=Nonebeing sent, which changes the meaning of the request. Only include this filter when a value is provided.
if type is not None:
params["type"] = type.name
params["marketDataId"] = marketDataId
if name:
src/Artesian/MarketData/MarketDataService.py:793
- The validation error messages here are ungrammatical/inconsistent ("must to be") and differ from the style used elsewhere in this file. Prefer the same
page must be >= 1 (got X)format used inreadDataQualityRuleAsync.
if page < 1:
raise ValueError("Page must to be greater than 0. Page:" + str(page))
if pageSize < 1:
raise ValueError(
"PageSize must to be greater than 0. Page Size:" + str(pageSize)
)
src/Artesian/MarketData/MarketDataService.py:953
- New
readDataQualityRuleAssignmentEventsFeedAsyncbehavior is not covered by unit tests (endpoint path andafterTimestampquery serialization). This file already has extensive request-matching tests, so this looks like an accidental gap.
async def readDataQualityRuleAssignmentEventsFeedAsync(
self: MarketDataService,
id: int,
afterTimestamp: Optional[datetime] = None,
) -> List[DqCheckChangeEventDtoOutput]:
src/Artesian/MarketData/_Dto/DataQualityStatusSummaryDto.py:35
- Field name
from_will serialize to JSON keyFrom_with the current global key transformer (__camelToPascalonly uppercases the first letter). The docstring says the API field name isFrom, so this DTO likely won't round-trip correctly unless the serializer strips the trailing underscore or a per-field rename is configured.
from_: Optional[date] = None
to: Optional[date] = None
Comment on lines
+796
to
+799
| params["page"] = page | ||
| params["pageSize"] = pageSize | ||
| params["marketDataId"] = marketDataId | ||
| params["ruleId"] = ruleId |
| self: MarketDataService, | ||
| page: int, | ||
| pageSize: int, | ||
| type: Optional[RuleType], |
Comment on lines
+469
to
+472
| recordValidationConfig=RecordValidationConfigDto( | ||
| recordRangeFrom="P-1D", | ||
| recordRangeTo="P0D", | ||
| ), |
Comment on lines
+503
to
+507
| recordRangeTo="P7D", | ||
| ), | ||
| versionToleranceFrom="PT-1H", | ||
| versionToleranceTo="PT1H", | ||
| versionPrecision=PeriodPrecision.Hour, |
…Klab/Artesian.SDK-Python into feature/22668-DataQualityRule
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (4)
src/Artesian/MarketData/MarketDataService.py:496
typeis documented as an optional filter, but it's a required positional argument in the signature. This forces callers to always pass a value (or explicitly passNone), which is inconsistent with the docstring and other optional query filters.
self: MarketDataService,
page: int,
pageSize: int,
type: Optional[RuleType],
marketDataId: Optional[int] = None,
src/Artesian/MarketData/MarketDataService.py:800
marketDataIdandruleIdare optional filters, but they are always added to the query params even whenNone. Withrequests, this can end up sendingmarketDataId=None/ruleId=Noneon the wire, which changes server-side filtering semantics.
params = {}
params["page"] = page
params["pageSize"] = pageSize
params["marketDataId"] = marketDataId
params["ruleId"] = ruleId
src/Artesian/MarketData/MarketDataService.py:794
- The validation error messages have grammatical issues ("must to be") and are inconsistent with the clearer f-string format used elsewhere in this file (e.g.,
readDataQualityRuleAsync). This is a public-facing exception message.
raise ValueError("Page must to be greater than 0. Page:" + str(page))
if pageSize < 1:
raise ValueError(
"PageSize must to be greater than 0. Page Size:" + str(pageSize)
)
src/Artesian/MarketData/MarketDataService.py:954
- This new API surface (
readDataQualityRuleAssignmentEventsFeed*) has no unit test coverage intests/TestMarketDataService.py, unlike the other newly added Data Quality Rule endpoints. Add aresponses-based test to lock down the query param serialization (especiallyafterTimestamp) and the list deserialization behavior.
async def readDataQualityRuleAssignmentEventsFeedAsync(
self: MarketDataService,
id: int,
afterTimestamp: Optional[datetime] = None,
) -> List[DqCheckChangeEventDtoOutput]:
Comment on lines
+552
to
+557
| self: MarketDataService, | ||
| page: int, | ||
| pageSize: int, | ||
| type: Optional[RuleType], | ||
| marketDataId: Optional[int] = None, | ||
| name: Optional[str] = None, |
Comment on lines
+683
to
+684
| Rules and assignments expose aggregated check status through the | ||
| `aggregatedStatus` property. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ref: #22668