OLAP DB documentation (bulk of it)

Author: jodebrui

samples/databases/wide-world-importers/documentation/wwi-olap-catalog.md

@@ -1,7 +1,47 @@
 # WideWorldImportersDW OLAP Database Catalog
 
-This folder contains documentation for the sample.
+The WideWorldImportersDW database is used for data warehousing and analytical processing. The transactional data about sales and purchases is generated in the WideWorldImporters database, and loaded into the WideWorldImportersDW database using a [daily ETL process](wwi-etl.md).
 
-Start with [root.md](root.md)
+The data in WideWorldImportersDW thus mirrors the data in WideWorldImporters, but the tables are organized differently. WideWorldImportersDW uses the [star schema](https://wikipedia.org/wiki/Star_schema) approach for its table design. Besides the fact and dimension tables, the database includes a number of staging tables that are used in the ETL process.
 
-Note that these contents will most likely be migrated to MSDN.
+## Schemas
+
+The different types of tables are organized in three schemas.
+
+|Schema|Description|
+|-----------------------------|---------------------|
+|Dimension|Dimension tables.|
+|Fact|Fact tables.|  
+|Integration|Staging tables and other objects needed for ETL.|  
+
+## Tables
+
+The dimension and fact tables are listed below. The tables in the Integration schema are used only for the ETL process, and are not listed.
+
+### Dimension tables
+
+WideWorldImportersDW has the following dimension tables. The description includes the relationship with the source tables in the WideWorldImporters database.
+
+|Table|Source tables|
+|-----------------------------|---------------------|
+|City|`Application.Cities`, `Application.StateProvinces`, `Application.Countries`.|
+|Customer|`Sales.Customers`, `Sales.BuyingGroups`, `Sales.CustomerCategories`.|
+|Date|New table with information about dates, including financial year (based on November 1st start for financial year).|
+|Employee|`Application.People`.|
+|StockItem|`Warehouse.StockItems`, `Warehouse.Colors`, `Warehouse.PackageType`.|
+|Supplier|`Purchasing.Suppliers`, `Purchasing.SupplierCategories`.|
+|PaymentMethod|`Application.PaymentMethods`.|
+|TransactionType|`Application.TransactionTypes`.|
+
+### Fact tables
+
+WideWorldImportersDW has the following dimension tables. The description includes the relationship with the source tables in the WideWorldImporters database, as well as the classes of analytics/reporting queries each fact table is typically used with.
+
+|Table|Source tables|Sample Analytics|
+|-----------------------------|---------------------|
+|Order|`Sales.Orders` and `Sales.OrderLines`|Sales people, picker/packer productivity, and on time to pick orders. In addition, low stock situations leading to back orders.|
+|Sale|`Sales.Invoices` and `Sales.InvoiceLines`|Sales dates, delivery dates, profitability over time, profitability by sales person.|
+|Purchase|`Purchasing.PurchaseOrderLines`|Expected vs actual lead times|
+|Transaction|`Sales.CustomerTransactions` and `Purchasing.SupplierTransactions`|Measuring issue dates vs finalization dates, and amounts.|
+|Movement|`Warehouse.StockTransactions`|Movements over time.|
+|Stock Holding|`Warehouse.StockItemHoldings`|On-hand stock levels and value|

samples/databases/wide-world-importers/documentation/wwi-olap-installation.md

@@ -1,7 +1,50 @@
 # WideWorldImportersDW Installation and Configuration
 
-This folder contains documentation for the sample.
 
-Start with [root.md](root.md)
+- [SQL Server 2016](https://www.microsoft.com/en-us/evalcenter/evaluate-sql-server-2016) (or higher) or [Azure SQL Database](https://azure.microsoft.com/services/sql-database/). To use the Full version of the sample, use SQL Server Evaluation/Developer/Enterprise Edition.
+- [SQL Server Management Studio](https://msdn.microsoft.com/library/mt238290.aspx). For the best results use the April 2016 preview or later.
 
-Note that these contents will most likely be migrated to MSDN.
+## Download
+
+The latest release of the sample:
+
+[wide-world-importers-v0.1](https://github.com/Microsoft/sql-server-samples/releases/tag/wide-world-importers-v0.1)
+
+Download the sample WideWorldImportersDW database backup/bacpac that corresponds to your edition of SQL Server or Azure SQL Database.
+
+Source code to recreate the sample database is available from the following location. Note that data population is based on ETL from the OLTP database (WideWorldImporters):
+
+[wide-world-importers](https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers/wwi-dw-database-scripts)
+
+## Install
+
+
+### SQL Server
+
+To restore a backup to a SQL Server instance, you can use Management Studio.
+1. Open SQL Server Management Studio and connect to the target SQL Server instance.
+2. Right-click on the **Databases** node, and select **Restore Database**.
+3. Select **Device** and click on the button **...**
+4. In the dialog **Select backup devices**, click **Add**, navigate to the database backup in the filesystem of the server, and select the backup. Click **OK**.
+5. If needed, change the target location for the data and log files, in the **Files** pane. Note that it is best practice to place data and log files on different drives.
+6. Click **OK**. This will initiate the database restore. After it completes, you will have the database WideWorldImporters installed on your SQL Server instance.
+
+### Azure SQL Database
+
+To import a bacpac into a new SQL Database, you can use Management Studio.
+1. (optional) If you do not yet have a SQL Server in Azure, navigate to the [Azure portal](https://portal.azure.com/) and create a new SQL Database. In the process of create a database, you will create a server. Make note of the server.
+  - See [this tutorial](https://azure.microsoft.com/documentation/articles/sql-database-get-started/) to create a database in minutes
+2. Open SQL Server Management Studio and connect to your server in Azure.
+3. Right-click on the **Databases** node, and select **Import Data-Tier Application**.
+4. In the **Import Settings** select **Import from local disk** and select the bacpac of the sample database from your file system.
+5. Under **Database Settings** change the database name to *WideWorldImportersDW* and select the target edition and service objective to use.
+6. Click **Next** and **Finish** to kick off deployment. It will take a few minutes to complete. When specifying a service objective lower than S2 it may take longer.
+
+## Configuration
+
+The sample database can make use of PolyBase to query files in Hadoop or Azure blob storage. However, that feature is not installed by default with SQL Server - you need to select it during SQL Server setup. Therefore, a post-installation step is required.
+
+1. In SQL Server Management Studio, connect to the WideWorldImportersDW database and open a new query window.
+2. Run the following T-SQL command to enable the use of PolyBase in the database:
+
+  EXECUTE [Application].[Configuration_ApplyPolyBase]

samples/databases/wide-world-importers/documentation/wwi-olap-sql-features.md

@@ -1,7 +1,87 @@
 # WideWorldImportersDW Use of SQL Server Features and Capabilities
 
-This folder contains documentation for the sample.
+WideWorldImportersDW is designed to showcase many of the key features of SQL Server that are suitable for data warehousing and analytics. The following is a list of SQL Server features and capabilities, and a description of how they are used in WideWorldImportersDW.
 
-Start with [root.md](root.md)
+## PolyBase
 
-Note that these contents will most likely be migrated to MSDN.
+[Applies to SQL Server (2016 and later)]
+
+PolyBase is used to combine sales information from WideWorldImportersDW with a public data set about demographics to understand which cities might be of interest for further expansion of sales.
+
+To enable the use of PolyBase in the sample database, make sure it is installed, and run the following statement in the database:
+
+    EXEC [Application].[Configuration_ApplyPolybase]
+
+This will create an external table `dbo.CityPopulationStatistics` that references a public data set that contains population data for cities in the United States, hosted in Azure blob storage. The following query returns the data from that external data set:
+
+    SELECT CityID, StateProvinceCode, CityName, YearNumber, LatestRecordedPopulation FROM dbo.CityPopulationStatistics;
+
+To understand which cities might be of interest for further expansion, the following query looks at the growth rate of cities, and returns the top 100 largest cities with significant growth, and where Wide World Importers does not have a sales presence. The query involves a join between the remote table `dbo.CityPopulationStatistics` and the local table `Dimension.City`, and a filter involving the local table `Fact.Sales`.
+
+    WITH PotentialCities
+    AS
+    (
+    	SELECT cps.CityName,
+    	       cps.StateProvinceCode,
+    		   MAX(cps.LatestRecordedPopulation) AS PopulationIn2016,
+    		   (MAX(cps.LatestRecordedPopulation) - MIN(cps.LatestRecordedPopulation)) * 100.0
+    		       / MIN(cps.LatestRecordedPopulation) AS GrowthRate
+    	FROM dbo.CityPopulationStatistics AS cps
+    	WHERE cps.LatestRecordedPopulation IS NOT NULL
+    	AND cps.LatestRecordedPopulation <> 0
+    	GROUP BY cps.CityName, cps.StateProvinceCode
+    ),
+    InterestingCities
+    AS
+    (
+    	SELECT DISTINCT pc.CityName,
+    					pc.StateProvinceCode,
+    				    pc.PopulationIn2016,
+    					FLOOR(pc.GrowthRate) AS GrowthRate
+    	FROM PotentialCities AS pc
+    	INNER JOIN Dimension.City AS c
+    	ON pc.CityName = c.City
+    	WHERE GrowthRate > 2.0
+    	AND NOT EXISTS (SELECT 1 FROM Fact.Sale AS s WHERE s.[City Key] = c.[City Key])
+    )
+    SELECT TOP(100) CityName, StateProvinceCode, PopulationIn2016, GrowthRate
+    FROM InterestingCities
+    ORDER BY PopulationIn2016 DESC;
+
+## Clustered Columnstore Indexes
+
+(Full version of the sample)
+
+Clustered Columnstore Indexes (CCI) are used with all the fact tables, to reduce storage footprint and improve query performance. With the use of CCI, the base storage for the fact tables uses column compression.
+
+Nonclustered indexes are used on top of the clustered columnstore index, to facilitate primary key and foreign key constraints. These constraints were added out of an abundance of caution - the ETL process sources the data from the WideWorldImporters database, which has constraints to enforce integrity. Removing primary and foreign key constraints, and their supporting indexes, would reduce the storage footprint of the fact tables.
+
+**Data size**
+
+The sample database has limited data size, to make it easy to download and install the sample. However, to see the real performance benefits of columnstore indexes, you would want to use a larger data set.
+
+You can run the following statement to increase the size of the `Fact.Sales` table by inserting another 12 million rows of sample data. These rows are all inserted for the year 2012, such that there is no interference with the ETL process.
+
+    EXECUTE [Application].[Configuration_PopulateLargeSaleTable]
+
+This statement will take around 5 minutes to run. To insert more than 12 million rows, pass the desired number of rows to insert as a parameter to this stored procedure.
+
+To compare query performance with and without columnstore, you can drop and/or recreate the clustered columnstore index.
+
+To drop the index:
+
+    DROP INDEX [CCX_Fact_Order] ON [Fact].[Order]
+
+To recreate:
+
+    CREATE CLUSTERED COLUMNSTORE INDEX [CCX_Fact_Order] ON [Fact].[Order]
+
+## Partitioning
+
+(Full version of the sample)
+
+Data size in a Data Warehouse can grow very large. Therefore it is best practice to use partitioning to manage the storage of the large tables in the database.
+
+All of the larger fact tables are partitioned by year. The only exception is `Fact.Stock Holdings`, which is not date-based and has limited data size compared with the other fact tables.
+
+The partition function used for all partitioned tables is `PF_Date`, and the partition scheme being used is `PS_Date`.

samples/databases/wide-world-importers/documentation/wwi-oltp-htap-catalog.md

@@ -1,6 +1,6 @@
 # WideWorldImporters Database Catalog
 
-The WideWorldImporters database contains all the transaction information and daily data for sales and purchases.
+The WideWorldImporters database contains all the transaction information and daily data for sales and purchases, as well as sensor data for vehicles and cold rooms.
 
 ## Schemas
 

samples/databases/wide-world-importers/documentation/wwi-oltp-htap-installation.md

@@ -11,7 +11,7 @@ The latest release of the sample:
 
 [wide-world-importers-v0.1](https://github.com/Microsoft/sql-server-samples/releases/tag/wide-world-importers-v0.1)
 
-Download the sample database backup/bacpac that corresponds to your edition of SQL Server or Azure SQL Database.
+Download the sample WideWorldImporters database backup/bacpac that corresponds to your edition of SQL Server or Azure SQL Database.
 
 Source code to recreate the sample database is available from the following location. Note that recreating the sample will result in slight differences in the data, since there is a random factor in the data generation:
 

samples/databases/wide-world-importers/documentation/wwi-oltp-htap-sample-queries.md

@@ -1,7 +1,5 @@
 # WideWorldImporters Sample Queries
 
-This folder contains documentation for the sample.
+Refer to the sample-scripts.zip file that is included with the release of the sample, or refer to the source code:
 
-Start with [root.md](root.md)
-
-Note that these contents will most likely be migrated to MSDN.
+[wide-world-importers/sample-scripts](https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers/sample-scripts)

samples/databases/wide-world-importers/sample-scripts/polybase/DemonstratePolybase.sql

@@ -4,8 +4,8 @@
 USE WideWorldImportersDW;
 GO
 
--- WideWorldImporters have customers in a variety of cities but feel they are likely missing 
--- other important cities. They have decided to try to find other cities have a growth rate of more 
+-- WideWorldImporters have customers in a variety of cities but feel they are likely missing
+-- other important cities. They have decided to try to find other cities have a growth rate of more
 -- than 20% over the last 3 years, and where they do not have existing customers.
 -- They have obtained census data (a CSV file) and have loaded it into an Azure storage account.
 -- They want to combine that data with other data in their main OLTP database to work out where
@@ -22,28 +22,28 @@ GO
 -- Expand the dbo.CityPopulationStatistics table, expand the list of columns and note the
 -- values that are contained. Let's look at the data:
 
-SELECT * FROM dbo.CityPopulationStatistics;
+SELECT CityID, StateProvinceCode, CityName, YearNumber, LatestRecordedPopulation FROM dbo.CityPopulationStatistics;
 GO
 
 -- How did that work? First the procedure created an external data source like this:
 /*
 
-CREATE EXTERNAL DATA SOURCE AzureStorage 
-WITH 
+CREATE EXTERNAL DATA SOURCE AzureStorage
+WITH
 (
 	TYPE=HADOOP, LOCATION = 'wasbs://[email protected]'
 );
 
 */
--- This shows how to connect to AzureStorage. Next the procedure created an 
+-- This shows how to connect to AzureStorage. Next the procedure created an
 -- external file format to describe the layout of the CSV file:
 /*
 
-CREATE EXTERNAL FILE FORMAT CommaDelimitedTextFileFormat 
-WITH 
+CREATE EXTERNAL FILE FORMAT CommaDelimitedTextFileFormat
+WITH
 (
-	FORMAT_TYPE = DELIMITEDTEXT, 
-	FORMAT_OPTIONS 
+	FORMAT_TYPE = DELIMITEDTEXT,
+	FORMAT_OPTIONS
 	(
 		FIELD_TERMINATOR = ','
 	)
@@ -61,35 +61,35 @@ CREATE EXTERNAL TABLE dbo.CityPopulationStatistics
 	YearNumber int NOT NULL,
 	LatestRecordedPopulation bigint NULL
 )
-WITH 
-(	
-	LOCATION = '/', 
+WITH
+(
+	LOCATION = '/',
 	DATA_SOURCE = AzureStorage,
 	FILE_FORMAT = CommaDelimitedTextFileFormat,
 	REJECT_TYPE = VALUE,
 	REJECT_VALUE = 4 -- skipping 1 header row per file
 );
 
 */
--- From that point onwards, the external table can be used like a local table. Let's run that 
+-- From that point onwards, the external table can be used like a local table. Let's run that
 -- query that they wanted to use to find out which cities they should be finding new customers
 -- in. We'll start building the query by grouping the cities from the external table
 -- and finding those with more than a 20% growth rate for the period:
 
 WITH PotentialCities
 AS
 (
-	SELECT cps.CityName, 
+	SELECT cps.CityName,
 	       cps.StateProvinceCode,
 		   MAX(cps.LatestRecordedPopulation) AS PopulationIn2016,
-		   (MAX(cps.LatestRecordedPopulation) - MIN(cps.LatestRecordedPopulation)) * 100.0 
+		   (MAX(cps.LatestRecordedPopulation) - MIN(cps.LatestRecordedPopulation)) * 100.0
 		       / MIN(cps.LatestRecordedPopulation) AS GrowthRate
 	FROM dbo.CityPopulationStatistics AS cps
 	WHERE cps.LatestRecordedPopulation IS NOT NULL
-	AND cps.LatestRecordedPopulation <> 0 
+	AND cps.LatestRecordedPopulation <> 0
 	GROUP BY cps.CityName, cps.StateProvinceCode
 )
-SELECT * 
+SELECT CityName, StateProvinceCode, PopulationIn2016, GrowthRate
 FROM PotentialCities
 WHERE GrowthRate > 2.0;
 GO
@@ -100,31 +100,31 @@ GO
 WITH PotentialCities
 AS
 (
-	SELECT cps.CityName, 
+	SELECT cps.CityName,
 	       cps.StateProvinceCode,
 		   MAX(cps.LatestRecordedPopulation) AS PopulationIn2016,
-		   (MAX(cps.LatestRecordedPopulation) - MIN(cps.LatestRecordedPopulation)) * 100.0 
+		   (MAX(cps.LatestRecordedPopulation) - MIN(cps.LatestRecordedPopulation)) * 100.0
 		       / MIN(cps.LatestRecordedPopulation) AS GrowthRate
 	FROM dbo.CityPopulationStatistics AS cps
 	WHERE cps.LatestRecordedPopulation IS NOT NULL
-	AND cps.LatestRecordedPopulation <> 0 
+	AND cps.LatestRecordedPopulation <> 0
 	GROUP BY cps.CityName, cps.StateProvinceCode
 ),
 InterestingCities
 AS
 (
-	SELECT DISTINCT pc.CityName, 
-					pc.StateProvinceCode, 
+	SELECT DISTINCT pc.CityName,
+					pc.StateProvinceCode,
 				    pc.PopulationIn2016,
 					FLOOR(pc.GrowthRate) AS GrowthRate
 	FROM PotentialCities AS pc
 	INNER JOIN Dimension.City AS c
-	ON pc.CityName = c.City 
+	ON pc.CityName = c.City
 	WHERE GrowthRate > 2.0
 	AND NOT EXISTS (SELECT 1 FROM Fact.Sale AS s WHERE s.[City Key] = c.[City Key])
 )
-SELECT TOP(100) *
-FROM InterestingCities 
+SELECT TOP(100) CityName, StateProvinceCode, PopulationIn2016, GrowthRate
+FROM InterestingCities
 ORDER BY PopulationIn2016 DESC;
 GO
 
@@ -136,4 +136,4 @@ DROP EXTERNAL FILE FORMAT CommaDelimitedTextFileFormat;
 GO
 DROP EXTERNAL DATA SOURCE AzureStorage;
 GO
-*/
+*/