Merge pull request #141 from MV10/issue_69

Make Id optional, allow nonclustered PK, allow bigint Id

Author: MV10

README.md

@@ -84,7 +84,7 @@ var log = new LoggerConfiguration()
 
 ## Table definition
 
-You'll need to create a table like this in your database:
+You'll need to create a table like this in your database. Many other variations are possible. In particular, give careful consideration to whether you need the Id column (discussed in the next section). The table definition shown here is the default configuration.
 
 ```
 CREATE TABLE [Logs] (
@@ -95,8 +95,7 @@ CREATE TABLE [Logs] (
    [Level] nvarchar(128) NULL,
    [TimeStamp] datetimeoffset(7) NOT NULL,  -- use datetime for SQL Server pre-2008
    [Exception] nvarchar(max) NULL,
-   [Properties] xml NULL,
-   [LogEvent] nvarchar(max) NULL
+   [Properties] xml NULL
 
    CONSTRAINT [PK_Logs] 
      PRIMARY KEY CLUSTERED ([Id] ASC) 
@@ -107,23 +106,83 @@ CREATE TABLE [Logs] (
 ) ON [PRIMARY];
 ```
 
-**Remember to grant the necessary permissions for the sink to be able to write to the log table.**
+If you don't plan to use a column, you can specify which columns to exclude in the `columnOptions.Store` parameter (see below). 
 
-If you don't plan on using one or more columns, you can specify which columns to include in the *columnOptions.Store* parameter (see below). 
-
-The Level column should be defined as a TinyInt if the *columnOptions.Level.StoreAsEnum* is set to true.
+The Level column should be defined as a `tinyint` when `columnOptions.Level.StoreAsEnum` is set to `true`.
 
 
 ### Automatic table creation
 
-If you set the `autoCreateSqlTable` option to `true`, the sink will create a table for you in the database specified in the connection string.  Make sure that the user associated with this connection string has enough rights to make schema changes.
+If you set the `autoCreateSqlTable` option to `true`, the sink will create a table for you in the database specified in the connection string.  Make sure that the user associated with this connection string has enough rights to make schema changes; see below.
+
+
+### Permissions
+
+At a minimum, writing log entries requires SELECT and INSERT permissions for the log table. (SELECT is required because the sink's batching behavior uses bulk inserts which reads the schema before the write operations begin).
+
+SQL permissions are a very complex subject. Here is an example of one possible solution (valid for SQL 2012 or later):
+
+```
+CREATE ROLE [SerilogAutoCreate];
+GRANT SELECT ON sys.tables TO [SerilogAutoCreate];
+GRANT SELECT ON sys.schemas TO [SerilogAutoCreate];
+GRANT ALTER ON SCHEMA::[dbo] TO [SerilogAutoCreate]
+GRANT CREATE TABLE ON DATABASE::[SerilogTest] TO [SerilogAutoCreate];
+
+CREATE ROLE [SerilogWriter];
+GRANT SELECT TO [SerilogWriter];
+GRANT INSERT TO [SerilogWriter];
+
+CREATE LOGIN [Serilog] WITH PASSWORD = 'password';
+
+CREATE USER [Serilog] FOR LOGIN [Serilog] WITH DEFAULT_SCHEMA = dbo;
+GRANT CONNECT TO [Serilog];
+
+ALTER ROLE [SerilogAutoCreate] ADD MEMBER [Serilog];
+ALTER ROLE [SerilogWriter] ADD MEMBER [Serilog];
+```
+
+This creates a SQL login named `Serilog`, a database user named `Serilog`, and assigned to that user are the roles `SerilogAutoCreate` and `SerilogWriter`. As the name suggests, the SerilogAutoCreate role is not needed if you create the database ahead of time, which is the recommended course of action if you're concerned about security at this level.
+
+Also, ideally the SerilogWriter role would be restricted to the log table only, and that table has to already exist for table-specific GRANT statements to execute, so that's another reason that you probably don't want to use auto-create. Table-level restrictions would look like this (assuming you name your log table SecuredLog, of course):
+
+```
+GRANT SELECT ON [dbo].[SecuredLog] TO [SerilogWriter];
+GRANT SELECT ON [dbo].[SecuredLog] TO [SerilogWriter];
+```
+
+There are many possible variations. For example, you could also create a new schema that was specific to the log(s) and restrict access that way.
+
+
+## Id Column Options
+
+Previous versions of this sink assumed the Id column is always present as an `int` `IDENTITY` primary key with a clustered index. Other configurations are available, however this is still the default for backwards-compatibility reasons.
+
+You should consider your anticipated logging volume and query requirements carefully. The default setting is not especially useful in real-world query scenarios since a clustered index is primarily of use when the key is used for sorting or range searches, which will rarely be the case for the Id column.
+
+### No Id Column
+
+If you eliminate the Id column completely, the log table is stored as an unindexed heap. This is the ideal write-speed scenario for logging, however any non-clustered indexes you add will slightly degrade write performance. One way to mitigate this is to keep the non-clustered indexes offline and use batch reindexing on a scheduled basis. If you create your table ahead of time, simply omit the Id column and the constraint shown in the previous section.
+
+### Unclustered Id Column
+
+You can also retain the Id column as an `IDENTITY` primary key, but using a non-clustered index. The log is still stored as an unindexed heap, but writes with non-clustered indexes are slightly faster. Non-clustered indexes on other columns will reference the Id primary key. However, read performance will be slightly degraded since it requires two reads (the covering non-clustered index, then dereferencing the heap row from the Id). To create this type of table ahead of time, change the constraint in the previous section to `NONCLUSTERED` and leave out the `WITH` clause.
 
+### Bigint Data Type
+
+For very large log tables, you may wish to create the Id column with the `bigint` datatype. This 8-byte integer will permit a maximum identity value of 9,223,372,036,854,775,807. The only change to the table syntax in the previous section is the datatype where `[Id]` is defined. This will slightly degrade both read and write performance.
+
+## Batch Size and Performance
+
+This is a "periodic batching sink." This means the sink will queue a certain number of log events before they're actually written to SQL Server as a bulk insert operation. There is also a timeout so that the batch is always written even if it has not been filled. By default, the batch size is 50 and the timeout is 5 seconds. You can change these through configuration.
+
+Consider increasing the batch size in high-volume logging environments. In one test of a loop writing a single log entry to a local server instance (no network traffic), the default batch achieved around 14,000 rows per second. Increasing the batch size to 1000 rows increased write speed to nearly 43,000 rows per second. However, you should also consider the risk-factor. If the server crashes or the connection goes down, you may lose an entire batch of log entries. You can mitigate this by reducing the timeout. Run performance tests to find the optimal batch size for your production log content, network setup, and server configuration.
 
 ## Standard columns
 
-The "standard columns" used by this sink (apart from obvious required columns like Id) are described by the StandardColumn enumeration and controlled through code by the `columnOptions.Store` collection.
+The "standard columns" used by this sink are described by the `StandardColumn` enumeration and controlled through code by the `columnOptions.Store` collection. By default (and consistent with the SQL command to create a table, above) these columns are included:
 
-By default (and consistent with the SQL command to create a table, above) these columns are included:
+ - `StandardColumn.Id`
  - `StandardColumn.Message`
  - `StandardColumn.MessageTemplate`
  - `StandardColumn.Level`
@@ -141,10 +200,9 @@ columnOptions.Store.Remove(StandardColumn.Properties);
 columnOptions.Store.Add(StandardColumn.LogEvent);
 ```
 
-You can also store your own log event properties as additional columns; see below.
+You can also store your own log event properties in additional custom columns; see below.
 
-
-### Saving properties in additional columns
+### Saving properties in custom columns
 
 By default any log event properties you include in your log statements will be saved to the Properties column (and/or LogEvent column, per columnOption.Store).  But they can also be stored in their own columns via the AdditionalDataColumns setting.
 
@@ -153,25 +211,30 @@ var columnOptions = new ColumnOptions
 {
     AdditionalDataColumns = new Collection<DataColumn>
     {
-        new DataColumn {DataType = typeof (string), ColumnName = "User"},
-        new DataColumn {DataType = typeof (string), ColumnName = "Other"},
+        new DataColumn {DataType = typeof(string), ColumnName = "UserName", DataLength = 64},
+        new DataColumn {DataType = typeof(string), ColumnName = "RequestUri", DataLength = -1, AllowNull = false},
     }
 };
 
 var log = new LoggerConfiguration()
-    .WriteTo.MSSqlServer(@"Server=.\SQLEXPRESS;Database=LogEvents;Trusted_Connection=True;", "Logs", columnOptions: columnOptions)
+    .WriteTo.MSSqlServer(@"Server=...", "Logs", columnOptions: columnOptions)
     .CreateLogger();
 ```
 
-The log event properties `User` and `Other` will now be placed in the corresponding column upon logging. The property name must match a column name in your table. Be sure to include them in the table definition.
+The log event properties `UserName` and `RequestUri` will be written to the corresponding columns whenever those values (with the exact same property name) occur in a log entry. Be sure to include them in the table definition if you create your table ahead of time.
+
+When configuring through code, set the `DataType` property to a .NET type. When configuring through XML, JSON or other settings packages, specify a SQL data type. It will be internally converted to an equivalent .NET type. Variable-length data types like `string` and `varchar` require a `DataLength` property. Use -1 to specify SQL's `MAX` length.
 
+**Standard column names are reserved. Even if you exclude a standard column, never create a custom column by the same name.**
 
-#### Excluding redundant items from the Properties column
+
+#### Excluding redundant Properties or LogEvent data
 
 By default, additional properties will still be included in the data saved to the XML Properties or JSON LogEvent column (assuming one or both are enabled via the `columnOptions.Store` parameter). This is consistent with the idea behind structured logging, and makes it easier to convert the log data to another (e.g. NoSQL) storage platform later if desired. 
 
-However, if necessary, then the properties being saved in their own columns can be excluded from the data.  Use the `columnOptions.Properties.ExcludeAdditionalProperties` parameter in the sink configuration to exclude the redundant properties from the XML. 
+However, if necessary, the properties being saved in their own columns can be excluded from the data.  Use the `columnOptions.Properties.ExcludeAdditionalProperties` parameter in the sink configuration to exclude the redundant properties from the XML, or `columnOptions.LogEvent.ExcludeAdditionalProperties` if you've added the JSON LogEvent column. 
 
+The standard columns are always excluded from the Properties and LogEvent columns.
 
 ### Columns defined by AppSettings (.NET Framework)
 
@@ -219,7 +282,7 @@ The equivalent of adding custom columns as shown in the .NET Framework example a
 }
 ```
 
-As the name suggests, `columnOptionSection` is an entire configuration section in its own right. All possible entries and some sample values are shown below. All properties and subsections are optional.
+As the name suggests, `columnOptionSection` is an entire configuration section in its own right. All possible entries and some sample values are shown below. All properties and subsections are optional. It is not currently possible to specify a `PropertiesFilter` predicate in configuration.
 
 ```json
 "columnOptionsSection": {
@@ -230,7 +293,7 @@ As the name suggests, `columnOptionSection` is an entire configuration section i
         { "ColumnName": "Release", "DataType": "varchar", "DataLength": 32 }
     ],
     "disableTriggers": true,
-    "id": { "columnName": "Id" },
+    "id": { "columnName": "Id", "bigint": true, "nonClusteredIndex": true },
     "level": { "columnName": "Level", "storeAsEnum": false },
     "properties": { 
         "columnName": "Properties",
@@ -256,11 +319,13 @@ As the name suggests, `columnOptionSection` is an entire configuration section i
 ```
 
 
-### Options for serialization of the log event data
+### Options for serialization of event data
+
+Typically you will choose either XML or JSON serialization, but not both.
 
 #### JSON (LogEvent column)
 
-The log event JSON can be stored to the LogEvent column. This can be enabled by adding the LogEvent column to the `columnOptions.Store` collection. Use the `columnOptions.LogEvent.ExcludeAdditionalProperties` parameter to exclude redundant properties from the JSON. This is analogue to excluding redundant items from XML in the Properties column.
+Event data items can be stored to the LogEvent column. This can be enabled by adding the LogEvent column to the `columnOptions.Store` collection. Use the `columnOptions.LogEvent.ExcludeAdditionalProperties` parameter to exclude redundant properties from the JSON. This is analogue to excluding redundant items from XML in the Properties column.
 
 #### XML (Properties column)
 
@@ -276,7 +341,7 @@ If `OmitDictionaryContainerElement`, `OmitSequenceContainerElement` or `OmitStru
 
 If `OmitElementIfEmpty` is set then if a property is empty, it will not be serialized.
 
-##### Querying the properties XML
+##### Querying the Properties XML data
 
 Extracting and querying the properties data directly can be helpful when looking for specific log sequences.
 

src/Serilog.Sinks.MSSqlServer/Configuration/Microsoft.Extensions.Configuration/LoggerConfigurationMSSqlServerExtensions.cs

@@ -216,9 +216,15 @@ private static ColumnOptions ConfigureColumnOptions(ColumnOptions columnOptions,
 
             SetIfProvided<bool>((val) => { opts.DisableTriggers = val; }, config["disableTriggers"]);
 
-            SetIfProvided<string>((val) => { opts.Id.ColumnName = val; }, config.GetSection("id")["columnName"]);
+            var section = config.GetSection("id");
+            if (section.GetChildren().Any())
+            {
+                SetIfProvided<string>((val) => { opts.Id.ColumnName = val; }, section["columnName"]);
+                SetIfProvided<bool>((val) => { opts.Id.BigInt = val; }, section["bigInt"]);
+                SetIfProvided<bool>((val) => { opts.Id.NonClusteredIndex = val; }, section["nonClusteredIndex"]);
+            }
 
-            var section = config.GetSection("level");
+            section = config.GetSection("level");
             if(section.GetChildren().Any())
             {
                 SetIfProvided<string>((val) => { opts.Level.ColumnName = val; }, section["columnName"]);
@@ -258,11 +264,11 @@ private static ColumnOptions ConfigureColumnOptions(ColumnOptions columnOptions,
                 SetIfProvided<bool>((val) => { opts.LogEvent.ExcludeAdditionalProperties = val; }, section["excludeAdditionalProperties"]);
             }
 
-            SetIfProvided<string>((val) => { opts.Id.ColumnName = val; }, config.GetSection("message")["columnName"]);
+            SetIfProvided<string>((val) => { opts.Message.ColumnName = val; }, config.GetSection("message")["columnName"]);
 
-            SetIfProvided<string>((val) => { opts.Id.ColumnName = val; }, config.GetSection("exception")["columnName"]);
+            SetIfProvided<string>((val) => { opts.Exception.ColumnName = val; }, config.GetSection("exception")["columnName"]);
 
-            SetIfProvided<string>((val) => { opts.Id.ColumnName = val; }, config.GetSection("messageTemplate")["columnName"]);
+            SetIfProvided<string>((val) => { opts.MessageTemplate.ColumnName = val; }, config.GetSection("messageTemplate")["columnName"]);
 
             return opts;
 

src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/ColumnOptions.cs

@@ -17,14 +17,20 @@ public class ColumnOptions
         /// </summary>
         public ColumnOptions()
         {
-            Id = new IdColumnOptions();
+            Id = new IdColumnOptions
+            {
+                // Defaults for backwards compatibility, see docs
+                BigInt = false,
+                NonClusteredIndex = false
+            };
 
             Level = new LevelColumnOptions();
 
             Properties = new PropertiesColumnOptions();
 
             Store = new Collection<StandardColumn>
             {
+                StandardColumn.Id,
                 StandardColumn.Message,
                 StandardColumn.MessageTemplate,
                 StandardColumn.Level,
@@ -116,7 +122,13 @@ public ICollection<StandardColumn> Store
         /// <summary>
         ///     Options for the Id column.
         /// </summary>
-        public class IdColumnOptions : CommonColumnOptions { }
+        public class IdColumnOptions : CommonColumnOptions
+        {
+            /// <summary>
+            /// If true, stores as bigint (max value 2^64-1) instead of the default int.
+            /// </summary>
+            public bool BigInt { get; set; }
+        }
 
         /// <summary>
         ///     Options for the Level column.
@@ -223,6 +235,14 @@ public class CommonColumnOptions
             /// The name of the column in the database.
             /// </summary>
             public string ColumnName { get; set; }
+
+            /// <summary>
+            /// Currently only implemented for the optional Id IDENTITY PK column. Defaults to false.
+            /// If true, the log table will be stored as a heap structure. This can improve
+            /// write performance at the cost of query performance. See documentation for details.
+            /// </summary>
+            public bool NonClusteredIndex { get; set; }
+
         }
 
         /// <summary>