Merge pull request #64 from jofriedm-msft/dev

Revert "Revert changes for version 5.1.0"

Author: jofriedm-msft

BreakingChanges.txt

@@ -1,4 +1,4 @@
-Changes in 5.1.1
+Changes in 5.1.1
 OTHER
   * Reverted the code from 5.1.0 because it contained a regression and an accidental change.
 
@@ -9,9 +9,9 @@ BLOB
  * getQualifiedStorageUri() has been deprecated. Please use getSnapshotQualifiedStorageUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token.
  * Fixed a bug where copying from a blob that included a SAS token and a snapshot did not use the SAS token.
 
- FILE
+FILE
  * Fixed a bug where copying from a blob that included a SAS token and a snapshot did not use the SAS token.
- 
+
 QUEUE
  * For addMessage() the CloudQueueMessage message passed in will be populated with the pop receipt, insertion/expiration time, and message ID.
 
@@ -168,8 +168,8 @@ Changes in 0.5.0
 
  OTHER
  * The packaging structure has changed to
-	1. com.microsoft.windowsazure.storage � RetryPolicies, LocationMode, StorageException, Storage Credentials etc. Basically all public classes that are common across services.
-	2. com.microsoft.windowsazure.storage.core � These classes are reserved for internal use and users never have to include this namespace in their projects. 
+	1. com.microsoft.windowsazure.storage � RetryPolicies, LocationMode, StorageException, Storage Credentials etc. Basically all public classes that are common across services.
+	2. com.microsoft.windowsazure.storage.core � These classes are reserved for internal use and users never have to include this namespace in their projects. 
 	3. com.microsoft.windowsazure.storage.blob
 	4. com.microsoft.windowsazure.storage.queue
 	5. com.microsoft.windowsazure.storage.table

README.md

@@ -1,10 +1,10 @@
-#Microsoft Azure Storage SDK for Java
+# Microsoft Azure Storage SDK for Java
 
 This project provides a client library in Java that makes it easy to consume Microsoft Azure Storage services. For documentation please see the Microsoft Azure [Java Developer Center](http://azure.microsoft.com/en-us/develop/java/) and the [JavaDocs](http://azure.github.io/azure-storage-java/).
 
 > If you are looking for the Azure Storage Android SDK, please visit [https://github.com/Azure/azure-storage-android](https://github.com/Azure/azure-storage-android).
 
-#Features
+# Features
   * Blob
       * Create/Read/Update/Delete containers
       * Create/Read/Update/Delete blobs
@@ -19,10 +19,10 @@ This project provides a client library in Java that makes it easy to consume Mic
       * Batch operations
       * Advanced Table Operations
 
-#Getting Started
+# Getting Started
 
-##Download
-###Option 1: Via Maven
+## Download
+### Option 1: Via Maven
 
 To get the binaries of this library as distributed by Microsoft, ready for use within your project, you can use Maven.
 
@@ -34,19 +34,19 @@ To get the binaries of this library as distributed by Microsoft, ready for use w
 </dependency>
 ```
 
-###Option 2: Source Via Git
+### Option 2: Source Via Git
 
 To get the source code of the SDK via git just type:
 
     git clone git://github.com/Azure/azure-storage-java.git
-    cd ./azure-storage-java/microsoft-azure-storage
+    cd ./azure-storage-java
     mvn compile
 
-###Option 3: Source Zip
+### Option 3: Source Zip
 
 To download a copy of the source code, click "Download ZIP" on the right side of the page or click [here](https://github.com/Azure/azure-storage-java/archive/master.zip). Unzip and navigate to the microsoft-azure-storage folder.
 
-##Minimum Requirements
+## Minimum Requirements
 
 * Java 1.6+
 * [Jackson-Core](https://github.com/FasterXML/jackson-core) is used for JSON parsing. 
@@ -58,13 +58,13 @@ The two dependencies, [Jackson-Core](https://github.com/FasterXML/jackson-core)
 
 SLF4J is only needed if you enable logging through the OperationContext class. If you plan to use logging, please also download an [SLF4J binding](http://repo2.maven.org/maven2/org/slf4j/) which will link the SLF4J API with the logging implementation of your choice. Simple is a good default. See the [SLF4J user manual](http://www.slf4j.org/manual.html) for more information.
 
-##Usage
+## Usage
 
 To use this SDK to call Microsoft Azure storage services, you need to first [create an account](https://account.windowsazure.com/signup).
 
 Samples are provided in the microsoft-azure-storage-samples folder. The unit tests in microsoft-azure-storage-test can also be helpful.
 
-##Code Sample
+## Code Sample
 
 The following is a quick example on how to upload a file to azure blob and download it back. You may also download and view the samples in the microsoft-azure-storage-samples folder. For additional information on using the client libraries to access Azure services see the How To guides for [blobs](http://azure.microsoft.com/en-us/documentation/articles/storage-java-how-to-use-blob-storage/), [queues](http://azure.microsoft.com/en-us/documentation/articles/storage-java-how-to-use-queue-storage/), [tables](http://azure.microsoft.com/en-us/documentation/articles/storage-java-how-to-use-table-storage/) and the [general documentation](http://azure.microsoft.com/en-us/develop/java/).
 
@@ -117,17 +117,17 @@ public class BlobSample {
 }
 ```
 
-#Need Help?
+# Need Help?
 
 Be sure to check out the Microsoft Azure [Developer Forums on MSDN](http://social.msdn.microsoft.com/Forums/windowsazure/en-US/home?forum=windowsazuredata) or the [Developer Forums on Stack Overflow](http://stackoverflow.com/questions/tagged/azure+windows-azure-storage) if you have trouble with the provided code.
 
-#Contribute Code or Provide Feedback
+# Contribute Code or Provide Feedback
 
 If you would like to become an active contributor to this project please follow the instructions provided in [Azure Projects Contribution Guidelines](http://azure.github.io/guidelines/).
 
 If you encounter any bugs with the library please file an issue in the [Issues](https://github.com/Azure/azure-storage-java/issues) section of the project.
 
-#Learn More
+# Learn More
 
 * [Azure Developer Center](http://azure.microsoft.com/en-us/develop/java/)
 * [Azure Storage Service](http://azure.microsoft.com/en-us/documentation/services/storage/)

microsoft-azure-storage-test/src/com/microsoft/azure/storage/blob/CloudBlobClientEncryptionTests.java

@@ -23,6 +23,7 @@
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.net.URISyntaxException;
+import java.nio.charset.Charset;
 import java.security.InvalidAlgorithmParameterException;
 import java.security.InvalidKeyException;
 import java.security.NoSuchAlgorithmException;
@@ -143,7 +144,41 @@ else if (type == BlobType.APPEND_BLOB) {
         TestHelper.assertStreamsAreEqualAtIndex(stream, new ByteArrayInputStream(outputStream.toByteArray()), 0, 0, 
                 size, 2 * 1024);
     }
-    
+
+    @Test
+    public void testDownloadUnencryptedBlobWithEncryptionPolicy() throws StorageException, IOException, URISyntaxException, NoSuchAlgorithmException
+    {
+        String blobName = BlobTestHelper.generateRandomBlobNameWithPrefix("test");
+        CloudBlockBlob blob = container.getBlockBlobReference(blobName);
+        blob.deleteIfExists();
+
+        byte[] msg = "my message".getBytes();
+        // Upload data without encryption
+        blob.uploadFromByteArray(msg, 0, msg.length);
+
+        // Create options with encryption policy
+        BlobRequestOptions options = new BlobRequestOptions();
+        options.setEncryptionPolicy(new BlobEncryptionPolicy(new RsaKey("myKey", 1024), null));
+        options.setRequireEncryption(true);
+
+        try {
+            blob.downloadText(Charset.defaultCharset().name(), null, options, null);
+            fail("Expect exception");
+        }
+        catch (StorageException e) {
+            assertEquals(SR.ENCRYPTION_DATA_NOT_PRESENT_ERROR, e.getMessage());
+        }
+
+        byte[] buffer = new byte[msg.length];
+        try {
+            blob.downloadRangeToByteArray(0, (long) buffer.length, buffer, 0, null, options, null);
+            fail("Expect exception");
+        }
+        catch (StorageException e) {
+            assertEquals(SR.ENCRYPTION_DATA_NOT_PRESENT_ERROR, e.getMessage());
+        }
+    }
+
     @Test
     public void testBlobEncryptionWithFile() throws URISyntaxException, StorageException, IOException,
             InvalidKeyException, NoSuchAlgorithmException, NoSuchPaddingException {

microsoft-azure-storage-test/src/com/microsoft/azure/storage/blob/CloudBlobClientTests.java

@@ -247,11 +247,11 @@ public void testSingleBlobPutThresholdInBytes() throws URISyntaxException, Stora
         try {
             bClient.getDefaultRequestOptions().setSingleBlobPutThresholdInBytes(
                     BlobConstants.MAX_SINGLE_UPLOAD_BLOB_SIZE_IN_BYTES + 1);
-            fail("Cannot set upload blob threshold above 64 MB");
+            fail("Cannot set upload blob threshold above 256 MB");
         }
         catch (IllegalArgumentException e) {
             assertEquals(
-                    "The argument is out of range. Argument name: singleBlobPutThresholdInBytes, Value passed: 67108865.",
+                    "The argument is out of range. Argument name: singleBlobPutThresholdInBytes, Value passed: 268435457.",
                     e.getMessage());
         }
 

microsoft-azure-storage-test/src/com/microsoft/azure/storage/blob/CloudBlockBlobTests.java

@@ -33,8 +33,6 @@
 import com.microsoft.azure.storage.file.SharedAccessFilePermissions;
 import com.microsoft.azure.storage.file.SharedAccessFilePolicy;
 
-import junit.framework.Assert;
-
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Test;
@@ -60,33 +58,11 @@
 import java.util.Random;
 import java.util.TimeZone;
 
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.experimental.categories.Category;
-
-import com.microsoft.azure.storage.AccessCondition;
-import com.microsoft.azure.storage.Constants;
-import com.microsoft.azure.storage.NameValidator;
-import com.microsoft.azure.storage.OperationContext;
-import com.microsoft.azure.storage.RetryNoRetry;
-import com.microsoft.azure.storage.SendingRequestEvent;
-import com.microsoft.azure.storage.StorageCredentialsAnonymous;
-import com.microsoft.azure.storage.StorageCredentialsSharedAccessSignature;
 import com.microsoft.azure.storage.StorageErrorCodeStrings;
-import com.microsoft.azure.storage.StorageEvent;
-import com.microsoft.azure.storage.StorageException;
 import com.microsoft.azure.storage.TestRunners.CloudTests;
 import com.microsoft.azure.storage.TestRunners.DevFabricTests;
 import com.microsoft.azure.storage.TestRunners.DevStoreTests;
 import com.microsoft.azure.storage.TestRunners.SlowTests;
-import com.microsoft.azure.storage.core.Utility;
-import com.microsoft.azure.storage.file.CloudFile;
-import com.microsoft.azure.storage.file.CloudFileShare;
-import com.microsoft.azure.storage.file.FileProperties;
-import com.microsoft.azure.storage.file.FileTestHelper;
-import com.microsoft.azure.storage.file.SharedAccessFilePermissions;
-import com.microsoft.azure.storage.file.SharedAccessFilePolicy;
 
 import static org.junit.Assert.*;
 
@@ -1018,6 +994,44 @@ public void testLargeBlobUploadFromStreamTest() throws URISyntaxException, Stora
         blockBlobRef.download(dstStream);
         BlobTestHelper.assertStreamsAreEqual(srcStream, new ByteArrayInputStream(dstStream.toByteArray()));
     }
+    
+    @Test
+    @Category({ DevFabricTests.class, DevStoreTests.class, SlowTests.class })
+    public void testLargeSinglePutBlobTest() throws URISyntaxException, StorageException, IOException {
+        final String blockBlobName = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final String blockBlobName2 = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final String blockBlobName3 = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final CloudBlockBlob blob = this.container.getBlockBlobReference(blockBlobName);
+        final CloudBlockBlob blob2 = this.container.getBlockBlobReference(blockBlobName2);
+        final CloudBlockBlob blob3 = this.container.getBlockBlobReference(blockBlobName3);
+        BlobRequestOptions options = new BlobRequestOptions();
+        options.setStoreBlobContentMD5(false);
+        options.setEncryptionPolicy(null);
+        try
+        {
+            byte[] buffer = BlobTestHelper.getRandomBuffer(256 * Constants.MB);
+
+            OperationContext operationContext = new OperationContext();
+
+            blob.uploadFromByteArray(buffer, 0, 128 * Constants.MB, null, null, operationContext);
+            assertEquals(1, operationContext.getRequestResults().size());
+
+            options.setSingleBlobPutThresholdInBytes(256 * Constants.MB);
+            blob2.uploadFromByteArray(buffer, 0, 256 * Constants.MB, null, options, operationContext);
+            assertEquals(1, operationContext.getRequestResults().size());
+
+            // Reduce threshold and upload data greater than the single put blob upload threshold
+            options.setSingleBlobPutThresholdInBytes(Constants.MB);
+            blob3.uploadFromByteArray(buffer, 0, 3 * Constants.MB, null, options, operationContext);
+            assertTrue(operationContext.getRequestResults().size() > 1);
+        }
+        finally
+        {
+            blob.delete();
+            blob2.delete();
+            blob3.delete();
+        }
+    }
 
     @Test
     @Category({ DevFabricTests.class, DevStoreTests.class })
@@ -1159,6 +1173,29 @@ public void testBlobUploadWithoutMD5Validation() throws URISyntaxException, Stor
         assertEquals("MDAwMDAwMDA=", blockBlobRef2.properties.getContentMD5());
     }
 
+    @Test
+    @Category({ DevFabricTests.class, DevStoreTests.class })
+    public void testVerifyTransactionalMD5ValidationMissingOverallMD5() throws URISyntaxException, StorageException, IOException {
+        final String blockBlobName = BlobTestHelper.generateRandomBlobNameWithPrefix("testBlockBlob");
+        final CloudBlockBlob blockBlobRef = this.container.getBlockBlobReference(blockBlobName);
+
+        final int length = 2 * 1024 * 1024;
+        ByteArrayInputStream srcStream = BlobTestHelper.getRandomDataStream(length);
+        BlobRequestOptions options = new BlobRequestOptions();
+        options.setSingleBlobPutThresholdInBytes(1024*1024);
+        options.setDisableContentMD5Validation(true);
+        options.setStoreBlobContentMD5(false);
+
+        blockBlobRef.upload(srcStream, -1, null, options, null);
+
+        options.setDisableContentMD5Validation(false);
+        options.setStoreBlobContentMD5(true);
+        options.setUseTransactionalContentMD5(true);
+        final CloudBlockBlob blockBlobRef2 = this.container.getBlockBlobReference(blockBlobName);
+        blockBlobRef2.downloadRange(1024, (long)1024, new ByteArrayOutputStream(), null, options, null);
+        assertNull(blockBlobRef2.getProperties().getContentMD5());
+    }
+
     @Test
     @Category({ DevFabricTests.class, DevStoreTests.class })
     public void testBlockBlobUploadContentMD5() throws URISyntaxException, StorageException, IOException {
@@ -1198,7 +1235,7 @@ public void eventOccurred(SendingRequestEvent eventArg) {
             }
         };
 
-        length = 33 * Constants.MB;
+        length = BlobConstants.DEFAULT_SINGLE_BLOB_PUT_THRESHOLD_IN_BYTES + 1;
         srcStream = BlobTestHelper.getRandomDataStream(length);
 
         sendingRequestEventContext.getSendingRequestEventHandler().addListener(event);
@@ -1709,6 +1746,23 @@ public void testBlobConditionalAccess() throws StorageException, IOException, UR
         newETag = blob.getProperties().getEtag();
         assertFalse("ETage should be modified on write metadata", newETag.equals(currentETag));
     }
+    
+    @Test
+    public void testBlobExceedMaxRange() throws URISyntaxException, StorageException, IOException
+    {
+        CloudBlockBlob blob = container.getBlockBlobReference("blockblob4");
+        blob.deleteIfExists();
+
+        byte[] msg = "my message".getBytes("UTF-8");
+        blob.uploadFromByteArray(msg, 0, msg.length);
+
+        byte[] buffer = new byte[msg.length + 5];
+
+        blob.downloadRangeToByteArray(0, (long) buffer.length, buffer, 0, null, null, null);
+        String expected = new String (msg, "UTF-8");
+        String actual = new String(buffer, "UTF-8").substring(0, 10);
+        assertEquals(expected, actual);
+    }
 
     @Test
     @Category({ DevFabricTests.class, DevStoreTests.class })

microsoft-azure-storage-test/src/com/microsoft/azure/storage/blob/CloudPageBlobTests.java

@@ -211,6 +211,32 @@ public void testPageBlobDownloadRangeValidationTest() throws StorageException, U
         assertEquals(100, downloadLength);
     }
 
+    /**
+     * Test requesting stored content MD5 with OpenWriteExisting().
+     * 
+     * @throws URISyntaxException
+     * @throws StorageException
+     */
+    @Test
+    public void testPageOpenWriteExistingWithMD5() throws URISyntaxException, StorageException, IOException {
+        final String pageBlobName = BlobTestHelper.generateRandomBlobNameWithPrefix("testPageBlob");
+        final CloudPageBlob pageBlobRef = this.container.getPageBlobReference(pageBlobName);
+        pageBlobRef.create(512);
+
+        BlobRequestOptions options = new BlobRequestOptions();
+        options.setStoreBlobContentMD5(true);
+        options.setDisableContentMD5Validation(false);
+
+        try
+        {
+            pageBlobRef.openWriteExisting(null, options, null);
+            fail("Expect failure due to requesting MD5 calculation");
+        }
+        catch (IllegalArgumentException e)
+        {
+        }
+    }
+
     @Test
     public void testPageBlobUploadFromStreamTest() throws URISyntaxException, StorageException, IOException {
         final String pageBlobName = BlobTestHelper.generateRandomBlobNameWithPrefix("testPageBlob");