Skip to content

RDS Instance System Tests #26733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Oct 10, 2022
Merged

RDS Instance System Tests #26733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions airflow/providers/amazon/aws/sensors/rds.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,12 @@ class RdsDbSensor(RdsBaseSensor):
:param target_statuses: Target status of DB
"""

template_fields: Sequence[str] = (
'db_identifier',
'db_type',
'target_statuses',
)

def __init__(
self,
*,
Expand Down
6 changes: 3 additions & 3 deletions docs/apache-airflow-providers-amazon/operators/rds.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ Create a database instance
To create a AWS DB instance you can use
:class:`~airflow.providers.amazon.aws.operators.rds.RdsCreateDbInstanceOperator`.

.. exampleinclude:: /../../tests/system/providers/amazon/aws/rds/example_rds_instance.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_rds_instance.py
:language: python
:dedent: 4
:start-after: [START howto_operator_rds_create_db_instance]
Expand All @@ -160,7 +160,7 @@ Delete a database instance
To delete a AWS DB instance you can use
:class:`~airflow.providers.amazon.aws.operators.rds.RDSDeleteDbInstanceOperator`.

.. exampleinclude:: /../../tests/system/providers/amazon/aws/rds/example_rds_instance.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_rds_instance.py
:language: python
:dedent: 4
:start-after: [START howto_operator_rds_delete_db_instance]
Expand All @@ -178,7 +178,7 @@ To wait for an Amazon RDS instance or cluster to reach a specific status you can
:class:`~airflow.providers.amazon.aws.sensors.rds.RdsDbSensor`.
By default, the sensor waits for a database instance to reach the ``available`` state.

.. exampleinclude:: /../../tests/system/providers/amazon/aws/rds/example_rds_instance.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_rds_instance.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_rds_instance]
Expand Down
57 changes: 39 additions & 18 deletions ...rs/amazon/aws/rds/example_rds_instance.py → ...viders/amazon/aws/example_rds_instance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down Expand Up @@ -26,56 +25,78 @@
RdsDeleteDbInstanceOperator,
)
from airflow.providers.amazon.aws.sensors.rds import RdsDbSensor
from tests.system.providers.amazon.aws.utils import set_env_id
from airflow.utils.trigger_rule import TriggerRule
from tests.system.providers.amazon.aws.utils import ENV_ID_KEY, SystemTestContextBuilder

sys_test_context_task = SystemTestContextBuilder().build()

ENV_ID = set_env_id()
DAG_ID = "example_rds_instance"
DAG_ID = 'example_rds_instance'

RDS_DB_IDENTIFIER = f'{ENV_ID}-database'
RDS_USERNAME = 'database_username'
# NEVER store your production password in plaintext in a DAG like this.
# Use Airflow Secrets or a secret manager for this in production.
RDS_PASSWORD = 'database_password'

with DAG(
dag_id=DAG_ID,
schedule=None,
schedule='@once',
start_date=datetime(2021, 1, 1),
tags=['example'],
catchup=False,
) as dag:
test_context = sys_test_context_task()
rds_db_identifier = f'{test_context[ENV_ID_KEY]}-database'

# [START howto_operator_rds_create_db_instance]
create_db_instance = RdsCreateDbInstanceOperator(
task_id='create_db_instance',
db_instance_identifier=RDS_DB_IDENTIFIER,
db_instance_class="db.t4g.micro",
engine="postgres",
db_instance_identifier=rds_db_identifier,
db_instance_class='db.t4g.micro',
engine='postgres',
rds_kwargs={
"MasterUsername": RDS_USERNAME,
"MasterUserPassword": RDS_PASSWORD,
"AllocatedStorage": 20,
'MasterUsername': RDS_USERNAME,
'MasterUserPassword': RDS_PASSWORD,
'AllocatedStorage': 20,
},
)
# [END howto_operator_rds_create_db_instance]

# RdsCreateDbInstanceOperator waits by default, setting as False to test the Sensor below.
create_db_instance.wait_for_completion = False

# [START howto_sensor_rds_instance]
db_instance_available = RdsDbSensor(
task_id="db_instance_available",
db_identifier=RDS_DB_IDENTIFIER,
await_db_instance = RdsDbSensor(
task_id='await_db_instance',
db_identifier=rds_db_identifier,
)
# [END howto_sensor_rds_instance]

# [START howto_operator_rds_delete_db_instance]
delete_db_instance = RdsDeleteDbInstanceOperator(
task_id='delete_db_instance',
db_instance_identifier=RDS_DB_IDENTIFIER,
db_instance_identifier=rds_db_identifier,
rds_kwargs={
"SkipFinalSnapshot": True,
'SkipFinalSnapshot': True,
},
)
# [END howto_operator_rds_delete_db_instance]
delete_db_instance.trigger_rule = TriggerRule.ALL_DONE

chain(
# TEST SETUP
test_context,
# TEST BODY
create_db_instance,
await_db_instance,
delete_db_instance,
)

from tests.system.utils.watcher import watcher

# This test needs watcher in order to properly mark success/failure
# when "tearDown" task with trigger rule is part of the DAG
list(dag.tasks) >> watcher()

@uranusjr uranusjr Sep 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This additional list calls is unnecessary?

@ferruzzi ferruzzi Sep 28, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed for system tests which have trigger_rule='ALL_DONE' on one of the tasks. Even if either of the first two tasks fail, the third (delete_db_instance) will trigger, and the test will still mark as a pass without the watcher.

@uranusjr uranusjr Sep 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird, I thought dag.tasks already returns a (new) list

airflow/airflow/models/dag.py

Lines 1163 to 1165 in 685f523

@property
def tasks(self) -> list[Operator]:
return list(self.task_dict.values())

@ferruzzi ferruzzi Sep 28, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My apologies, I misunderstood and thought you were saying the whole line was not needed.

Yeah, interesting. That block is a copy/paste from the AIP-47 specification, and all of the previous system tests do it this way. But it does look like maybe it isn't required.

@ferruzzi ferruzzi Sep 30, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to keep it standardized like all of the others and fix them all later, unless you feel strongly about changing this one.

@potiuk potiuk Oct 10, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a good reason we added it first and possibly it has changed since by modifying "tasks" property to return list. As @ferruzzi wrote - better to keep it consistent and we might want to remove the list as a final refactor.


chain(create_db_instance, db_instance_available, delete_db_instance)

from tests.system.utils import get_test_run # noqa: E402

Expand Down
17 changes: 0 additions & 17 deletions tests/system/providers/amazon/aws/rds/__init__.py
View file
Open in desktop

This file was deleted.