md5 checksum not checked on server?

[mwan-movideo]

Should the md5 checksum be checked for file upload?

Situation 1:
The blob is being overwritten.
My steps:

1. Get a blob reference
2. calculate the md5 checksum on the original file
3. upload the md5 checksum using upload properties
4. upload another file which is slightly modified than the original file

My modified file is uploaded successfully, and I can download the file from azure cloud and the downloaded file is the modified file.

Situation 2:
The blob is new.
How should I submit a md5 to be checked when the file is uploaded?

1. I can not upload properties before the file upload because blob is not created.
2. If I submit the md5 after the file upload, how can the server know my md5?

[joeg]

This is not a bug.

The md5 it is only checked by the server when it is sent with the entire blob or it is set via uploadProperties / uploadBlockList where it is not validated, but will be returned on subsequent downloads. Any subsequent upload will overwrite the Content Md5 of the pre-existing blob.

For example if you upload a blob in a single put request (i.e. <64MB) with a content md5 header it is validated prior to storing the blob, however if you upload the blob, then do a uploadProperties with a md5 this value is simply stored as metadata. This is due to scale reasons, i.e. if you blob was 250 gb the service would have to retrieve all of the data in order to validate the md5 which is not feasible.

The Java library provided has a few conveniences to help for your scenario. Specifically BlobRequestOptions provides a setting UseTransactionalMd5 (default to false) and StoreBlobContentMd5(default to false) which will calculate and store the values you are seeking automatically. As explained in the blog the Blobs content md5 and the md5 of the data in a single request ( referred to as transactional md5) are different.

See http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/18/windows-azure-blob-md5-overview.aspx for more. Specifically the REST section.

[mwan-movideo]

Thanks for the request options. Now I can work around to upload the full blob and get md5 checksum from blob properties after the upload to check it.

However, the following Java code should fail when upload but it went through no problem. I tried to set the md5 just before upload, and the md5 is from a different file than the file being uploaded. The upload process didn't complain.

package movideo.azuretest;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URISyntaxException;
import java.security.InvalidKeyException;

import org.apache.commons.codec.digest.DigestUtils;

import com.microsoft.windowsazure.services.blob.client.BlobContainerPermissions;
import com.microsoft.windowsazure.services.blob.client.BlobContainerPublicAccessType;
import com.microsoft.windowsazure.services.blob.client.CloudBlob;
import com.microsoft.windowsazure.services.blob.client.CloudBlobClient;
import com.microsoft.windowsazure.services.blob.client.CloudBlobContainer;
import com.microsoft.windowsazure.services.blob.client.CloudBlockBlob;
import com.microsoft.windowsazure.services.blob.client.ListBlobItem;
import com.microsoft.windowsazure.services.core.storage.CloudStorageAccount;
import com.microsoft.windowsazure.services.core.storage.StorageException;
import com.microsoft.windowsazure.services.core.storage.utils.Base64;

public class Test1
{

public static final String storageConnectionString = "<<Clould connection string>>";

/**
 * @param args
 * @throws URISyntaxException
 * @throws InvalidKeyException
 * @throws StorageException
 * @throws IOException
 * @throws FileNotFoundException
 */
public static void main(String[] args) throws InvalidKeyException, URISyntaxException, StorageException, FileNotFoundException, IOException
{
    CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
    CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
    CloudBlobContainer container = blobClient.getContainerReference("mycontainer1");
    container.createIfNotExist();

    /* 
     * Test point 1:
     * I have tried to make the public access CONTAINER, BLOB and PRIVATE.
     */
    BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
    containerPermissions.setPublicAccess(BlobContainerPublicAccessType.BLOB);
    container.uploadPermissions(containerPermissions);

    CloudBlockBlob blob = container.getBlockBlobReference("playlist/7/100x20.jpg");

    /*
     * Test point 2:
     * Here are two files with only one byte difference (simulate a network transmit error).
     * Tried upload srcFile's md5 with modSrcFile, which should fail, but not. 
     */
    File srcFile = new File("/home/mwan/workspace/files/azure/launch.jpg");
    File modSrcFile = new File("/home/mwan/workspace/files/azure/launch-mod.jpg");
    FileInputStream inputStream = new FileInputStream(srcFile);

    /*
     * Test point 3:
     * Tried to set the md5 and not set the md5. 
     * According to {@link http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/18/windows-azure-blob-md5-overview.aspx}
     * set the md5 should make the server to check it.
     */
    blob.getProperties().setContentMD5(Base64.encode(DigestUtils.md5(inputStream)));

    /*
     * Test point 4:
     * Tried to upload the correct file and the incorrect file.
     */
    //      blob.upload(inputStream, srcFile.length());
    blob.upload(new FileInputStream(modSrcFile), srcFile.length());

    /*
     * Test point 5:
     * Get the md5 back from server to verify the file is uploaded successfully, but always getting null.
     */
    for(ListBlobItem blobItem : container.listBlobs("playlist/7/"))
    {
        CloudBlob ablob = new CloudBlockBlob(blobItem.getUri(), blobClient);
        System.out.println(ablob.getName() + " md5 => " + ablob.getProperties().getContentMD5());
    }
}

}

[joeg]

There are two ways to upload a block blob, either in a single put ( up to 64 mb) or as individual blocks with a trailing put block list. The single put will validate the md5 as all data is at the server when the header is retrieved, the put block list will not validate it as all the data is not part of the request itself, instead it will simply store it on the bog. We expose the latter via the BlobOutputStream which is accessible via openOutputStream, or used internally for uploads that meet certain criteria. If the blob is over the SinglePutThreshold (configurable on the client) or if the InputStream is non markable the upload methods will revert to the OutputStream, In your case I would recommend using the BufferedInputStream to work around this.

We will do better in documenting this more fully in the future, however the BlobRequestOptions properties are designed to offer the most seamless way of working with md5.

joe

[tomlee]

I came across the same problem. The single block blob put you suggested doesn't validate MD5 when I use the Java SDK. I set the MD5 with blob.getProperties().setContentMD5(). I have already set setStoreBlobContentMD5 and setUseTransactionalContentMD5 of the BlobRequestOptions object to true and use blob.upload(inpustream, length, null, blobRequestOptions, null) to supply the options.

I believe this is a bug. It's either the MD5 validation is ignored at the server side or the MD5 is not sent to the server when the blob upload request is composed.

I suggest you try it out the code and you'll know it isn't working. Otherwise, please include the code you've tested is working.