Add SSH/SFTP support for ObjectStoragePath#60757
Conversation
2517925 to
b092dbc
Compare
b092dbc to
71e7d1e
Compare
fritz-astronomer
left a comment
There was a problem hiding this comment.
lgtm.
I see lots of tests for connection parsing, and I was going to comment about having an integration test - but I suppose as long as the connection is passed as expected, sshfs is going to handle all that.
Were you able to do a full test with an SSH/SFTP system?
Thanks for the review, @fritz-astronomer! Yes, I spun up a local SFTP server with Docker and ran through the |
|
Hi @bolkedebruin, hope you're doing well. I would like to request your review on this PR when you have a moment. Please take your time, and I truly appreciate any feedback you might have. Thank you so much - it means a lot! 🙏 |
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
* 60558 (feature) : added ssh sftp object store * fix depreciation warning * documentation clean ups * clean ups
Summary
Adds SSH/SFTP protocol support for Airflow's
ObjectStoragePathabstraction, enabling file operations over SFTP using existing Airflow
connections.
get_fs()function using thesshfslibrary(fsspec-compatible)
sftp://andssh://URI schemescredentials, key files)
Usage
Changes
implementation
Installation
pip install apache-airflow-providers-sftp[sshfs]
closes: #60558
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.