Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After deploying Velero 1.16 in our OpenStack clusters with
item-block-worker-count=5, the plugin panicked when taking a batch of volume snapshots.message: /Error getting volume snapshotter for volume snapshot location error: /rpc error: code = Aborted desc = plugin panicked: runtime error: invalid memory address or nil pointer dereference, stack trace: goroutine 90 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:26 +0x5e github.com/vmware-tanzu/velero/pkg/plugin/framework/common.HandlePanic({0x147b1e0?, 0x2380950}) /go/pkg/mod/github.com/vmware-tanzu/velero@v1.11.1/pkg/plugin/framework/common/handle_panic.go:43 +0x13b github.com/vmware-tanzu/velero/pkg/plugin/framework.(*VolumeSnapshotterGRPCServer).Init.func1() /go/pkg/mod/github.com/vmware-tanzu/velero@v1.11.1/pkg/plugin/framework/volume_snapshotter_server.go:56 +0x2a panic({0x147b1e0?, 0x2380950?}) /usr/local/go/src/runtime/panic.go:792 +0x132What did you expect to happen:
The Init() function of plugin should be called only once per process.
**Fix Implementation **:
This appears to be a race condition where the volume snapshot plugin's Init() function is called concurrently. I add a sync.once to protect the Init() function in this plugin repo and it works fine.