Unable to resolve column names with a '.' in the name

[Tom-Hudson]

Describe the bug
When attempting to use any check on a field that has a '.' in the name such as:

.isComplete("column.name")

You will get an exception like:

Exception: Verification checks failed: [{'check_status': 'Error', 'check_level': 'Error', 'constraint_status': 'Failure', 'check': 'Verification checks', 'constraint_message': "org.apache.spark.sql.AnalysisException: cannot resolve '`column.name`' given input columns: [column.name];

To Reproduce
Steps to reproduce the behavior:

1. Create a data source with a column name that has a '.' in such as 'column.name'
2. Create a verification suite that includes the check .isComplete("column.name")
3. Run the code
4. See the error

Expected behavior
The field name should be resolved, allowing checks to be performed

[github-actions]

This is a known issue that stems from how Spark resolves column names — a dot (.) in a column name is interpreted as a nested field accessor (e.g., struct.field). This happens at the Spark/Deequ level rather than in PyDeequ's Python code specifically.

Workaround: Wrap the column name in backticks to escape it:

.isComplete("`column.name`")

This tells Spark to treat column.name as a literal column name rather than a nested struct reference. This applies to all checks and analyzers that accept a column parameter.

If backtick-escaping doesn't work with Deequ's internal processing, the alternative is to rename your columns before running PyDeequ checks:

df = df.withColumnRenamed("column.name", "column_name")

This is ultimately a limitation in how Deequ passes column names to Spark SQL expressions. A proper fix would require Deequ (the Scala library) to escape column names containing dots when building expressions.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.