You are currently viewing Copying Azure blobs from one storage account to another

Copying Azure blobs from one storage account to another

Azure does not provide any ability to move the data. You have to copy the data and then delete form the original container if you want to move the data.

There are different options available to copy data from one blob container to another. Let’s have look at those options.

Azure CLI

One obvious option is to use Azure CLI. Azure CLI provides az storage commands to create storage account, blob container, upload and download the data objects.

If you use upload / download data objects commands, they are synchronous. So, if you are moving very large data objects, it may take hours as it would first download the data to your client machine, then it would try to upload the data. As this operation is synchronous you can do nothing but to watch this operation to complete. Also, if this operation fails after some of the data objects are transferred, it does not maintain any checkpoint. So you have no option but to start this operation again.

Alternatively, Azure CLI also provides an asynchronous blob copy option – az storage blob copy command. This command runs asynchronously and Azure storage service manages the progress of the operation. So, it allows you to track the progress and cancel the operation if required.

Below Azure CLI showx example command. It tries to copy “myblob” from source container to destination container. This is an asynchronous command. You can view status of the command using az storage blob show and if you want you can cancel the operation using az storage blob cancel commands.

Do not forget to specify $HOT_STORAGE_NAME (source storage account), $COOL_STORAGE_NAME (destination storage account), and their account keys ($HOT_KEY and $COOL_KEY) variables.

# To copy blob from source container to destination container
az storage blob copy start-batch \
  --destination-container archived-specifications \
  --account-name $COOL_STORAGE_NAME \
  --account-key $COOL_KEY \
  --source-account-name "$HOT_STORAGE_NAME" \
  --source-account-key $HOT_KEY \
  --source-container specifications

# To view status of operation
az storage blob show \
  --container-name destContainer \
  --name myBlob

AzCopy

This utility has been specifically written to copy data from one blob container to another OR to copy data from on premise to Azure storage.

All the operations executed by AzCopy are asynchronous and they can be recovered if any failure occurs. It can also be restarted from point of failure.

AzCopy does provide comprehensive support for hierarchical containers and blob selection by pattern matching – these two features are not available with Azure CLI.

Below are some examples of AzCopy command. You can include certain flag like –recursive=true, to include all the files under nested directories, or you can also specify the –include=”*.txt”, which will just include the text file blobs in the operation.

# To upload file to the container specified 
azcopy copy myfile.txt https://myaccount.blob.core.windows.net/mycontainer/

# To copy all contents of myfolder to container
azcopy copy myfolder https://myaccount.blob.core.windows.net/mycontainer/?<sas token> --recursive=true

# To download the blob data
azcopy copy https://myaccount.blob.core.windows.net/mycontainer/myblob?<sas token> myblobdata

# To copy data from source account to destination account using sas tokens
azcopy copy https://sourceaccount.blob.core.windows.net/sourcecontainer/*?<source sas token> https://destaccount.blob.core.windows.net/destcontainer/*?<dest sas token>

.NET Storage Client Library

This library provides you API and objects which can be used to manage the data objects in Azure blob storage. You can create your own custom application to upload, download or move the data objects from one blob container to another.

The biggest advantage of using this library is you can read all the metadata of blob using the APIs provided in this library. You can read last modified by, last modified time, creation time, etc. If required, you can use this metadata to select only those objects which you really want to move.

Also, the APIs provided by this library are asynchronous. So you can take full advantage of multi tasking features of .NET framework.

The custom application created using this library can be deployed in variety of environments (e.g. Azure Functions) depending on need of your solution.

However you may not want to use this approach if your taks is one-off kind of task as the time taken for developing custom application may be higher than running just AzCopy or Azure CLI command.

If you have complex task which you want to repeat that task again and again, then it would be preferable to create a custom application using the .NET storage client library.

Below is a sample code which uses NuGet package: Windows.Azure.Storage, and demonstrates how to upload / download the blob contents.

CloudStorageAccount sourceAccount = CloudStorageAccount.Parse(sourceConnection);
CloudBlobClient sourceClient = sourceAccount.CreateCloudBlobClient();


// To download the contents
CloudBlobContainer sourceBlobContainer = sourceClient.GetContainerReference(sourceContainer);
ICloudBlob sourceBlob = await sourceBlobContainer.GetBlobReferenceFromServerAsync("MyBlob.doc");
Console.WriteLine($"Last modified: {sourceBlob.Properties.LastModified}");
await sourceBlob.DownloadToFileAsync("MyFile.doc", System.IO.FileMode.Create);

// To Upload the blob contents to destination container
CloudBlobContainer destBlobContainer = destClient.GetContainerReference(destContainer);
CloudBlockBlob destBlob = destBlobContainer.GetBlockBlobReference("NewBlob.doc");
await destBlob.UploadFromFileAsync("MyFile.doc");

I hope this article is helpful to understand how can you copy blobs from one container to another, or even to a container within different storage account. Let me know your thoughts.

Leave a Reply

This Post Has One Comment

  1. Anonymous

    I used your azcopy command to copy a container between subscriptions, and it created a single folder in the target container with an asterisk for its name.