CLI >= v247 Pipelines no longer support any wildcards in path reference

[anjaen]

Describe the issue

When yml config contains a reference to a file or notebook, where a wildcard for, for example python or sql files, is used, this is no longer working in CLI version 247.0 or higher. Release notes contains no indicators this should stop working.

This example will fail validate, but worked in previous versions:

resources:
  pipelines:
    pipeline_name:
      name: example_pipeline
      <<: *permissions
      serverless: true
      catalog: ${var.catalog}
      target: ${var.example_schema}
      libraries:
        - file:
            path: ../src/example/pipelines/*.py
      configuration:
        bundle.source_path: ${workspace.file_path}/src
        bundle.catalog: ${var.catalog}

This example will pass validate and be possible to deploy, but still fail to find any files for the pipeline to run:

resources:
  pipelines:
    example_pipeline:
      name: example_pipeline
      <<: *permissions
      serverless: true
      catalog: ${var.catalog}
      target: ${var.example_schema}
      libraries:
        - file:
            path: ${workspace.file_path}/src/example/pipelines/*.py
      configuration:
        bundle.source_path: ${workspace.file_path}/src
        bundle.catalog: ${var.catalog}

Configuration

Please provide a minimal reproducible configuration for the issue

Steps to reproduce the behavior

Bad behavior #1

1. Have a pipeline config with a relative path containing wildcards for libraries -file or -notebook
2. Run databricks bundle validate
3. See error similar to the following
   databricks bundle validate --profile dev Error: unable to determine if C:\Users\REDACTED\databricks\Repos\REDACTED\src\example\pipelines*.py is not a notebook: open src/example/pipelines/*.py: The filename, directory name, or volume label syntax is incorrect.

Bad behavior #2

1. Have a pipeline config with an absolute path, but where the path is containing wildcards for libraries -file or -notebook
2. Run databricks bundle deploy
3. Open the pipeline in Databricks
4. see error 'File or notebook not found' for files referenced with wildcard, e.g. REDACTED/src/example/pipelines/*.py

Expected Behavior

Expect to be able to still reference code files in a folder using wildcards.

Actual Behavior

Deployment validation or pipeline run (with config change as mentioned above) fails.

OS and CLI version

CLI version >=247.0

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?
Yes.
Worked in production up to v245.0, according to my tests locally on Windows, still works in v246.

Debug Logs

Nothing more of interest than the above error for alternative #1, in alternative #2 nothing particular in debug logs.

[pietern]

Thanks for reporting this issue. This looks like a bug. We'll investigate.

[shreyas-goenka]

@anjaen Thanks for reporting this issue. We are working on a fix for issue (1) here: #2723

For issue (2), I don't think we ever supported glob patterns for absolute paths in the workspace file system. path: ${workspace.file_path}/src/example/pipelines/*.py does not work even on 0.246. Please let me know if I am missing something here.

[anjaen]

@anjaen Thanks for reporting this issue. We are working on a fix for issue (1) here: #2723

For issue (2), I don't think we ever supported glob patterns for absolute paths in the workspace file system. path: ${workspace.file_path}/src/example/pipelines/*.py does not work even on 0.246. Please let me know if I am missing something here.

That's fine, thanks for working on issue (1), issue (2) was an attempt to workaround the validation error in (1) by replacing the .../src with an alternative reference to the folder containing src, not something I need fixed as long as (1) is fixed.