Skip to content

Run the examples in the new format #18946

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 14 commits into from
Dec 7, 2025
Merged

Run the examples in the new format #18946

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
50f9345
Run the examples in the new format
Nov 26, 2025
60c2a9e
Fix issues causing GitHub checks to fail
Nov 26, 2025
ec321cc
Trigger CI
Nov 27, 2025
ec77fba
fix typo in custom_data_source and update rust_example.sh wih passing…
Nov 27, 2025
14a91c1
fix regexp example using content from regex.csv file
Nov 27, 2025
ca8300f
fix type in udf examples and use csv data from cars.csv file
Nov 27, 2025
a4cac2f
Track generated file so rustfmt matches CI
Nov 28, 2025
5110e4c
move logic to run()
Nov 28, 2025
f8c18f6
cleanup test folders in script and use temporary directory
Dec 5, 2025
33f17d4
Merge branch 'main' into cj-zhukov/run-examples-in-the-new-format
alamb Dec 5, 2025
db30d51
Merge branch 'main' into cj-zhukov/run-examples-in-the-new-format
alamb Dec 5, 2025
2f5c63f
remove cleanup test folders in script
Dec 6, 2025
383993a
Fix example
alamb Dec 6, 2025
2c98086
Merge remote-tracking branch 'apache/main' into cj-zhukov/run-example…
alamb Dec 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix typo in custom_data_source and update rust_example.sh wih passing… 
… correct arg
  • Loading branch information
Sergey Zhukov committed Nov 27, 2025
commit ec77fba66754758202248ad0b5a98f8c70431d8a
2 changes: 1 addition & 1 deletion ci/scripts/rust_example.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,5 @@ for dir in */; do
fi

echo "Running example group: $example_name"
Copy link
Contributor

@alamb alamb Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I ran this script twice, I got an error the second time around:

./ci/scripts/rust_example.sh
./ci/scripts/rust_example.sh

The second run made this:

Running example: deserialize_to_struct
Running example group: datafusion-examples
error: no example target named `datafusion-examples` in default-run packages
help: available example targets:
    builtin_functions
    custom_data_source
    data_io
    dataframe
    execution_monitoring
    external_dependency
    flight
    proto
    query_planning
    sql_ops
    udf

Copy link
Contributor

@alamb alamb Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the issue is that some of the examples leave files around:

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion2$ find datafusion-examples/examples/datafusion-examples
datafusion-examples/examples/datafusion-examples
datafusion-examples/examples/datafusion-examples/test_json
datafusion-examples/examples/datafusion-examples/test_json/lQqU6IGDpwHJyGQB_0.json
datafusion-examples/examples/datafusion-examples/test_json/KKsELYFJ4st3GUOa_0.json
datafusion-examples/examples/datafusion-examples/test_csv
datafusion-examples/examples/datafusion-examples/test_csv/9Alj5dF7w72vLpp0_0.csv.gz
datafusion-examples/examples/datafusion-examples/test_csv/4Bh8tCQllXXE43A9_0.csv.gz
datafusion-examples/examples/datafusion-examples/test_table
datafusion-examples/examples/datafusion-examples/test_table/Y2CVyUXOXRPhDhFA_0.parquet
datafusion-examples/examples/datafusion-examples/test_table/2nskzCtDZnEJR851_0.parquet
datafusion-examples/examples/datafusion-examples/test_parquet
datafusion-examples/examples/datafusion-examples/test_parquet/4CIKeX6U9Ik1YYgY_0.parquet
datafusion-examples/examples/datafusion-examples/test_parquet/ooUJiB7QMqY0BRdN_0.parquet

Maybe we could change the examples to use a temporary directory or else have this script clean up before running

Copy link
Contributor Author

@cj-zhukov cj-zhukov Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good point! Let's do it.

cargo run --profile ci --example "$example_name" all
cargo run --profile ci --example "$example_name" -- all
done
12 changes: 6 additions & 6 deletions datafusion-examples/examples/custom_data_source/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ enum ExampleKind {
CustomFileCasts,
CustomFileFormat,
DefaultColumnValues,
FileFtreamProvider,
FileStreamProvider,
}

impl AsRef<str> for ExampleKind {
Expand All @@ -67,7 +67,7 @@ impl AsRef<str> for ExampleKind {
Self::CustomFileCasts => "custom_file_casts",
Self::CustomFileFormat => "custom_file_format",
Self::DefaultColumnValues => "default_column_values",
Self::FileFtreamProvider => "file_stream_provider",
Self::FileStreamProvider => "file_stream_provider",
}
}
}
Expand All @@ -84,7 +84,7 @@ impl FromStr for ExampleKind {
"custom_file_casts" => Ok(Self::CustomFileCasts),
"custom_file_format" => Ok(Self::CustomFileFormat),
"default_column_values" => Ok(Self::DefaultColumnValues),
"file_stream_provider" => Ok(Self::FileFtreamProvider),
"file_stream_provider" => Ok(Self::FileStreamProvider),
_ => Err(DataFusionError::Execution(format!("Unknown example: {s}"))),
}
}
Expand All @@ -99,7 +99,7 @@ impl ExampleKind {
Self::CustomFileCasts,
Self::CustomFileFormat,
Self::DefaultColumnValues,
Self::FileFtreamProvider,
Self::FileStreamProvider,
];

const RUNNABLE_VARIANTS: [Self; 7] = [
Expand All @@ -109,7 +109,7 @@ impl ExampleKind {
Self::CustomFileCasts,
Self::CustomFileFormat,
Self::DefaultColumnValues,
Self::FileFtreamProvider,
Self::FileStreamProvider,
];

const EXAMPLE_NAME: &str = "custom_data_source";
Expand Down Expand Up @@ -139,7 +139,7 @@ impl ExampleKind {
ExampleKind::DefaultColumnValues => {
default_column_values::default_column_values().await?
}
ExampleKind::FileFtreamProvider => {
ExampleKind::FileStreamProvider => {
file_stream_provider::file_stream_provider().await?
}
ExampleKind::All => unreachable!("`All` should be handled in main"),
Expand Down
Loading