You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are plenty of tools with seemingly similar functionality at first glance.
Some of them mount PVCs locally, others transfer data between PVCs, perform backups, etc.
However, I was surprised to find that none of them can efficiently transfer files to and from a PVC—especially in a fast and concurrent manner.
Of course, there's always the manual approach: running a debug Pod or building a custom image with the necessary tools—but that's a long and cumbersome way to get things done.
Currently, I'm developing a tool for PostgreSQL WAL archiving, and I ran into a roadblock when I needed to transfer a large amount of data inside a PVC.
If the volume is hostPath-based, it's relatively straightforward—you can simply copy the required files onto the target node. But with CSI-backed volumes, where the PVC is mounted as a block device, things become more complex.
In my case, I have three primary storage classes: hostPath, Ceph RBD, and NFS.
Most of my PostgreSQL instances use RBD volumes, so without a lot of shell magic, you can’t just "copy something in" directly.
I've tested this tool with Ceph RBD, distroless-based images, and hostPath PVCs.
It performs much faster than kubectl cp or kubectl exec. With those, I gave up waiting for even a 100 GiB copy to finish—and that's a relatively small amount of data.
Perhaps someone will find it useful too. Best regards!
🤖 Beep beep! I’m a robot speaking on behalf of @ahmetb. 🤖
Thanks for submitting your kubectl plugin to Krew!
One of the krew-index maintainers will review it soon. Note that the reviews for new plugin submissions may take a few days.
In the meanwhile, here are a few tips to make your plugin manifest better:
This pull-request has been approved by: hashmap-kz Once this PR has been reviewed and has the lgtm label, please assign chriskim06 for approval. For more information see the Code Review Process.
The full list of commands accepted by this bot can be found here.
Details
Needs approval from an approver in each of these files:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cncf-cla: yesIndicates the PR's author has signed the CNCF CLA.kind/new-pluginsize/MDenotes a PR that changes 30-99 lines, ignoring generated files.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello! The new plugin page: https://github.com/hashmap-kz/kubectl-syncpod/tree/master
Motivation
There are plenty of tools with seemingly similar functionality at first glance.
Some of them mount PVCs locally, others transfer data between PVCs, perform backups, etc.
However, I was surprised to find that none of them can efficiently transfer files to and from a PVC—especially in a fast and concurrent manner.
Of course, there's always the manual approach: running a debug Pod or building a custom image with the necessary tools—but that's a long and cumbersome way to get things done.
Currently, I'm developing a tool for PostgreSQL WAL archiving, and I ran into a roadblock when I needed to transfer a large amount of data inside a PVC.
If the volume is hostPath-based, it's relatively straightforward—you can simply copy the required files onto the target node. But with CSI-backed volumes, where the PVC is mounted as a block device, things become more complex.
In my case, I have three primary storage classes: hostPath, Ceph RBD, and NFS.
Most of my PostgreSQL instances use RBD volumes, so without a lot of shell magic, you can’t just "copy something in" directly.
I've tested this tool with
Ceph RBD,distroless-basedimages, and hostPath PVCs.It performs much faster than
kubectl cporkubectl exec. With those, I gave up waiting for even a 100 GiB copy to finish—and that's a relatively small amount of data.Perhaps someone will find it useful too.
Best regards!