added ability to not infer schema

Author: johskristoffersen

README.md

@@ -6,6 +6,7 @@ https://www.audienceproject.com/blog/tech/sparkdynamodb-using-aws-dynamodb-data-
 
 ## News
 
+* 2021-01-28: Added option `inferSchema=false` which is useful when writing to a table with many columns
 * 2020-07-23: Releasing version 1.1.0 which supports Spark 3.0.0 and Scala 2.12. Future releases will no longer be compatible with Scala 2.11 and Spark 2.x.x.
 * 2020-04-28: Releasing version 1.0.4. Includes support for assuming AWS roles through custom STS endpoint (credits @jhulten).
 * 2020-04-09: We are releasing version 1.0.3 of the Spark+DynamoDB connector. Added option to `delete` records (thank you @rhelmstetter). Fixes (thank you @juanyunism for #46).
@@ -100,6 +101,7 @@ The following parameters can be set as options on the Spark writer object before
 - `targetCapacity` fraction of provisioned write capacity on the table to consume for writing or updating. Default 1 (i.e. 100% capacity).
 - `update` if true items will be written using UpdateItem on keys rather than BatchWriteItem. Default false.
 - `throughput` the desired write throughput to use. It overwrites any calculation used by the package. It is intended to be used with tables that are on-demand. Defaults to 100 for on-demand.
+- `inferSchema` if false will not automatically infer schema - this is useful when writing to a table with many columns
 
 ## System Properties
 The following Java system properties are available for configuration.

src/main/scala/com/audienceproject/spark/dynamodb/datasource/DynamoTable.scala

@@ -80,7 +80,7 @@ class DynamoTable(options: CaseInsensitiveStringMap,
 
     private def inferSchema(): StructType = {
         val inferenceItems =
-            if (dynamoConnector.nonEmpty) dynamoConnector.scan(0, Seq.empty, Seq.empty).firstPage().getLowLevelResult.getItems.asScala
+            if (dynamoConnector.nonEmpty && options.getBoolean("inferSchema",true)) dynamoConnector.scan(0, Seq.empty, Seq.empty).firstPage().getLowLevelResult.getItems.asScala
             else Seq.empty
 
         val typeMapping = inferenceItems.foldLeft(Map[String, DataType]())({