AWS Glue Job runs indefinitely after exception occurs when adding Checks to the VerificationSuite

[mattwaismann]

Describe the bug
When running PyDeequ Checks on an AWS Glue job, if an exception occurs (e.g. accidentally calling hasCompletenss(...) instead of hasCompleteness(...)), the Glue job will keep running indefinitely.

To Reproduce
Glue 4.0
deequ-2.0.4-spark-3.3.jar
python-deequ 1.1.0

check_result = (
        VerificationSuite(spark)
        .onData(df)
        .addCheck(
            check.isComplete("primary_key")
            .hasCompletenss("product_name", lambda x: x > 0.999) #notice the typo
        )
        .run()

Expected behavior
The expectation is that when errors such as this occur, the Glue job stops running or an exception is thrown so the callback server and spark session can be shut down:

Screenshots
In the CloudWatch Glue error logs you see that the job continues polling for executor status:

[mattwaismann]

I experienced this issue again when I missed a positional argument when incorrectly calling check.hasCompleteness("col_name")

Expected behavior: Glue job stops with Exception: TypeError: hasCompleteness() missing 1 required positional argument: 'assertion'

Actual behavior: Glue job continued to run indefinitely

[github-actions]

Thanks for reporting this. This is a legitimate bug — when an exception occurs during Check construction (e.g., a typo like hasCompletenss or a missing argument), the Python callback server that PyDeequ starts for passing lambda functions to the JVM gets into a hung state, preventing the Spark/Glue job from terminating.

The root cause is in the PythonCallback class in scala_utils.py — once the callback server is started, if an unhandled Python exception occurs before run() completes, there's no cleanup mechanism to shut down the callback server, leaving the JVM waiting indefinitely for a response.

We'll look into adding proper error handling/cleanup around the callback server lifecycle to ensure exceptions propagate correctly and the job terminates. In the meantime, as a workaround, you can wrap your verification suite in a try/except and explicitly call:

try:
    # your verification code
except Exception as e:
    spark.sparkContext._gateway.shutdown_callback_server()
    spark.stop()
    raise e

This ensures the callback server and Spark session are properly shut down even when an error occurs.

This has also been flagged for our maintainer team to review.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.