SSH/SFTP Object Store

[fritz-astronomer]

Description

Would be awesome to have a SSH / SFTP ObjectStoragePath implementation

• https://github.com/fsspec/sshfs
• https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.sftp.SFTPFileSystem

It looks like it can be added as an Alternative Backend, so maybe not too difficult to implement?

Use case/motivation

Be able to use an Airflow ssh_conn_id to perform standard operations via ssh across operating systems.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sjyangkevin]

I am interested in looking into it.

[anishgirianish]

Hi! I've opened a PR to implement this: #60757

Adds ObjectStoragePath support for SFTP/SSH using the sshfs library.   
 Integrates with existing Airflow SFTP connections.                     
                                                                        
 from airflow.sdk import ObjectStoragePath                              
 path = ObjectStoragePath("sftp://my_conn@server/path/file.txt")        
 content = path.read_text()  

Would appreciate a review. Thanks!

[fritz-astronomer]

@anishgirianish - I think that looks good, though I don't have recent knowledge of what existing FileSystem implementations looked like to confirm it matches others super closely.

(I commented this on the PR)
I see lots of tests for connection parsing, and I was going to comment about having an integration test - but I suppose as long as the connection is passed as expected, sshfs is going to handle all that.

Were you able to do a full test with an SSH/SFTP system?

[anishgirianish]

@anishgirianish - I think that looks good, though I don't have recent knowledge of what existing FileSystem implementations looked like to confirm it matches others super closely.

(I commented this on the PR) I see lots of tests for connection parsing, and I was going to comment about having an integration test - but I suppose as long as the connection is passed as expected, sshfs is going to handle all that.

Were you able to do a full test with an SSH/SFTP system?

Thanks for the review, @fritz-astronomer! Yes, I spun up a local SFTP server with Docker and ran through the
basic operations - reads, writes, listing, deletes all worked fine.