Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixup tests and views for the 2025 rollover #715

Merged
merged 5 commits into from
Jan 29, 2025

Conversation

jeancochrane
Copy link
Contributor

@jeancochrane jeancochrane commented Jan 27, 2025

Now that we have 2025 data in Athena, some of our data integrity tests are failing due to views that aren't properly equipped to handle data following a rollover. This PR updates those views and bumps the data_test_iasworld_year_start variable to 2025 so that all of our tests are looking at the latest year of data.

Data integrity test output: https://github.com/ccao-data/data-architecture/actions/runs/13000380643/job/36257605843

iasWorld data test output: https://github.com/ccao-data/data-architecture/actions/runs/13000382805/job/36257611283

@jeancochrane jeancochrane changed the title Fixup tests for the 2025 rollover Fixup tests and views for the 2025 rollover Jan 27, 2025
@jeancochrane jeancochrane force-pushed the jeancochrane/fixup-tests-post-2025-rollover branch from 4b7b839 to 3a1302c Compare January 27, 2025 23:26
@jeancochrane jeancochrane force-pushed the jeancochrane/fixup-tests-post-2025-rollover branch from 3a1302c to 83826a1 Compare January 27, 2025 23:28
dbt/macros/pre_stage_filters.sql Show resolved Hide resolved
dbt/macros/pre_stage_filters.sql Show resolved Hide resolved
dbt/macros/tests/test_all.sql Show resolved Hide resolved
dbt/models/default/default.vw_pin_history.sql Show resolved Hide resolved
dbt/models/default/default.vw_pin_value.sql Show resolved Hide resolved
dbt/models/reporting/schema.yml Show resolved Hide resolved
dbt/models/ccao/ccao.vw_time_util.sql Show resolved Hide resolved
@jeancochrane jeancochrane marked this pull request as ready for review January 27, 2025 23:38
@jeancochrane jeancochrane requested a review from a team as a code owner January 27, 2025 23:38
@jeancochrane jeancochrane requested a review from dfsnow January 27, 2025 23:38
Copy link
Member

@dfsnow dfsnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work here @jeancochrane. These are definitely some tricky issues. Glad we'll have tests to catch them going forward.


{% macro pre_mailed_filters(tablename) %}
{{- pre_stage_filters(tablename, "CCAOVALUE") -}}
and cardinality(stages.procnames) = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: I would maybe add a brief comment here explaining what cardinality() is doing. I had to look it up and then take a moment to grok how it relates to procnames. Copying and pasting your PR comment below about the logic is fine IMO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, done in 28b78f8.

dbt/models/default/default.vw_pin_value.sql Show resolved Hide resolved
@jeancochrane jeancochrane force-pushed the jeancochrane/fixup-tests-post-2025-rollover branch from 28b78f8 to 276c1ee Compare January 29, 2025 16:25
@jeancochrane jeancochrane merged commit e7b1979 into master Jan 29, 2025
8 checks passed
@jeancochrane jeancochrane deleted the jeancochrane/fixup-tests-post-2025-rollover branch January 29, 2025 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants