[Feature Request] Content Integrity Verification

[shizhMSFT]

Proposal

Artifacts are stored in various types of storage systems, such as remote registries or local file systems. Since these systems are vulnerable to issues like accidental deletion, file system corruption, or physical damage to the storage medium, it is essential for ORAS to implement a dedicated mechanism for verifying the integrity of stored artifacts.

Note

Unlike Proof of Data Possession (PDP) and Proof of Retrievability (PoR), the integrity verification method described here requires retrieving the entire content for validation rather than relying on sampling techniques.

Content integrity verification should be performed at 4 distinct levels:

1. Blob content
2. DAG content (i.e. artifact)
3. Artifact with referrers
4. Storage medium (e.g., OCI image layout)

Blob Content Verification

Blob content verification can be done using the VerifyReader in the package content.

package content

import (
	"context"
	"io"

	ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)

// Verify checks the integrity of the content against the provided descriptor.
func Verify(ctx context.Context, fetcher Fetcher, desc ocispec.Descriptor) error {
	rc, err := fetcher.Fetch(ctx, desc)
	if err != nil {
		return err
	}
	defer rc.Close()

	vr := NewVerifyReader(rc, desc)
	if _, err = io.Copy(io.Discard, vr); err != nil {
		return err
	}
	return vr.Verify()
}

DAG Content Verification

Artifacts a.k.a. Directed Acyclic Graph (DAG) verification should be in the package oras. The APIs are Verify/VerifyGraph. Just like Copy/CopyGraph, Verify/VerifyGraph copies the artifact to nowhere (discard them) under the mode of "on error resume next".

The APIs are subject to change and can be

func Verify(ctx context.Context, src ReadOnlyTarget, ref string, opts VerifyOptions) (ocispec.Descriptor, *VerifyReport, error)
func VerifyGraph(ctx context.Context, src content.ReadOnlyStorage, root ocispec.Descriptor, opts VerifyGraphOptions) (*VerifyGraphReport, error)

It is a common scenario that we verify a repository or even a registry with layers are shared. Therefore, we should design a way to re-use the verification report to avoid re-verifying the verified blobs.

Artifact with referrers

By extending "DAG Content Verification", we can have ExtendedVerify/ExtendedVerifyGraph just like ExtendedCopy/ExtendedCopyGraph.

Storage medium

In certain scenarios, providing access to the low-level storage medium can significantly improve the efficiency and depth of integrity verification.

For example, when verifying the entire repository in an OCI image layout, direct access to the storage path structure (e.g., blobs/<alg>/<digest>) enables straightforward validation of individual blobs. Additionally, this approach allows identification of unreferenced blobs, offering further insights into the repository's content and structure.