Skip to content

Feat: Implement hf:// / "hugging face" integration in datafusion-cli #10792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xinlifoobar wants to merge 20 commits into from
+1,191 −0
Closed

Feat: Implement hf:// / "hugging face" integration in datafusion-cli #10792

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
4979491
Implement parse hf url
xinlifoobar Jun 4, 2024
426fde9
Merge remote-tracking branch 'origin' into dev/xinli/hfstore
xinlifoobar Jun 4, 2024
b47cca1
Draft hf_store impl
xinlifoobar Jun 4, 2024
6efddcc
Merge remote-tracking branch 'origin' into dev/xinli/hfstore
xinlifoobar Jun 6, 2024
cc16287
Complete integrations code
xinlifoobar Jun 6, 2024
d324c68
Implement get eventually
xinlifoobar Jun 6, 2024
e97a5a0
implement for list table api
xinlifoobar Jun 7, 2024
faac3bc
Fix CI
xinlifoobar Jun 7, 2024
4591734
Add e2e ci test
xinlifoobar Jun 7, 2024
f59889d
stage path change
xinlifoobar Jun 9, 2024
ce2bb24
complete path refactoring
xinlifoobar Jun 10, 2024
645fe80
refining builder code
xinlifoobar Jun 10, 2024
b8476e8
Refining list code
xinlifoobar Jun 10, 2024
0a536fe
Merge branch 'main' of github.com:apache/datafusion into dev/xinli/hf…
xinlifoobar Jun 10, 2024
31835ba
Add document
xinlifoobar Jun 10, 2024
d1f1809
Merge branch 'main' of github.com:apache/datafusion into dev/xinli/hf…
xinlifoobar Jun 11, 2024
7f21b3c
Fix ci
xinlifoobar Jun 11, 2024
bc7345b
Fix doc
xinlifoobar Jun 11, 2024
1f659b3
disable win-x64 test
xinlifoobar Jun 13, 2024
ff7f701
Merge branch 'main' of github.com:apache/datafusion into dev/xinli/hf…
xinlifoobar Jun 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix doc
  • Loading branch information
@xinlifoobar
xinlifoobar committed Jun 11, 2024
commit bc7345b4a78565c2bc5d877a0b4f72b452b85918
8 changes: 4 additions & 4 deletions docs/source/user-guide/cli/datasources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,7 @@ STORED AS parquet
LOCATION "hf://datasets/cais/mmlu/astronomy/";
```
Copy link
Contributor Author

@xinlifoobar xinlifoobar Jun 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently not working due to:

  1. The hugging face user access token is case-sensitive.
  2. a previous change enforces every option value in lower case. https://github.com/apache/datafusion/pull/9723/files.

I will figure out the history to see whether this will be feasible.


or
or

```bash
$ export HF_USER_ACCESS_TOKEN=******
Expand All @@ -400,7 +400,7 @@ LOCATION "hf://datasets/cais/mmlu/astronomy/";

Supported configuration options are:

| Environment Variable | Configuration Option | Description |
| -------------------- | -------------------- | ----------- |
| `HF_ENDPOINT` | `hf.endpoint` | Hugging Face endpoint |
| Environment Variable | Configuration Option | Description |
| ---------------------- | ---------------------- | ------------------------------ |
| `HF_ENDPOINT` | `hf.endpoint` | Hugging Face endpoint |
| `HF_USER_ACCESS_TOKEN` | `hf.user_access_token` | Hugging Face user access token |