3.0.0

This major release is built against Apache Spark 4.1.0 instead of Spark 3, and thus requires Java 17. Please see below for a complete list of breaking changes, enhancements, and bug fixes.

Breaking Changes

1. When splitting text and creating chunks, the connector now defaults to creating one sidecar document per chunk as opposed to defaulting to adding all chunks to the source document.
2. Vector embeddings in XML documents now default to a QName of vector with a namespace of http://marklogic.com/vector, matching upcoming default index exclusions in the MarkLogic server.
3. Vector embeddings in JSON documents now default to a name of _vector, also matching upcoming default index exclusions in the MarkLogic server.
4. The deprecated spark.marklogic.write.fileRows.documentType option has been removed. This option was intended to be used with Spark's binaryFile data source, but binary files should instead be read with this connector's own support for reading files.
5. The com.marklogic.client, okhttp3, okio, and com.burgstaller.okhttp packages are no longer shaded in the connector jar. These had to be shaded when the connector depended on Spark 3 as Spark 3 included its own older version of OkHttp.

Enhancements

1. When a URI template has an expression that cannot be resolved for a given document, the new spark.marklogic.write.uriTemplate.warnOnMissingField option can be set to true to log a warning instead of failing. The expression will have its value replaced with UNRESOLVED- prepended to a random UUID.
2. When reading files, the connector now defaults a number of partitions equal to the value of spark.default.parallelism, helping avoid performance issues due to large numbers of very small partitions.
3. When classifying text via a Semaphore instance in Progress Data Cloud (PDC), the PDC token will be renewed if it expires during the course of a connector job.
4. When exporting documents to zip file, a warning will be logged once a zip file contains 500,000 entries. Writing multiple large zip files at once can lead to heap space exhaustion in the JVM; users can avoid this by increasing the number of partitions.
5. Added spark.marklogic.read.partitions.vars. as a prefix for defining variables to send to the custom code for reading partitions when reading items via custom code.

Bug Fixes

1. Fixed a bug with writing triples where datatype is only set if lang does not exist.
2. Fixed a bug where, when reading files, a partition could have zero files. Files are now evenly distributed across partitions.
3. Fixed a bug with exporting documents to zip files on Windows.

2.7.0

This minor release addresses the following items:

• Can now provide a secondary query when reading documents from MarkLogic. This is supported via the following new options:
  • spark.marklogic.read.secondaryUris.invoke
  • spark.marklogic.read.secondaryUris.javascript
  • spark.marklogic.read.secondaryUris.javascriptFile
  • spark.marklogic.read.secondaryUris.xquery
  • spark.marklogic.read.secondaryUris.xqueryFile
  • spark.marklogic.read.secondaryUris.vars.
• Can now provide a prompt when generating an embedding via the new spark.marklogic.write.embedder.prompt option.
• Can now encode vectors in documents when generating embeddings via the new spark.marklogic.write.embedder.base64encode option.
• Fixed a bug where classifying text and generating embeddings did not work when data was read from a structured data source such as JDBC or a delimited text file.
• Fixed a bug where a document with a URI containing multiple colons could not be read from MarkLogic and written to a file.
• Fixed a bug where URIs were incorrectly modified when documents were written as entries in a zip file. URIs are now used as the zip entry name.

2.6.0

This release addresses the following items:

• Can now extract text from binary documents via Apache Tika .
• Can now classify text via Progress Semaphore.
• Can now specify document properties and metadata values when writing documents to MarkLogic.

2.5.1

This patch release addresses the following items:

1. Depends on the MarkLogic Java Client 7.1.0 release, which includes an important bug fix that affects how the connector reads data via custom code.
2. Added debug-level logging for reading and writing data via custom code.
3. Fixed an issue with logging progress when reading rows via an Optic query.

2.5.0

This release addresses the following items:

1. Can now split text in documents when writing them to MarkLogic. Chunks of text can be added to the source document itself or written to separate sidecar documents.
2. Can now add embeddings to chunks in documents before writing them to MarkLogic. You can reuse the Flux embedding model integrations available from the Flux releases site by adding one or more of these JAR files to your Spark classpath.
3. When reading rows via an Optic query, the Optic query no longer requires the use of op.fromView. However, when not using op.fromView, the Optic query will be executed in a single call to MarkLogic.
4. When writing files to a directory, the given path will be created automatically if it does not exist, matching the behavior of Spark file-based data sources.

Please see the writing guide for more information on the splitter and embedder features.

2.4.2

This patch release addresses the following two issues:

1. spark.marklogic.read.snapshot was added to allow a user to configure a non-consistent snapshot when reading documents by setting the option to false. This avoids bugs where a consistent snapshot is not feasible and the downsides of reading at multiple times are not a concern.
2. Issues with importing JSON Lines files via Flux - such as keys being reordered and added - can be avoided by setting the existing
   spark.marklogic.read.files.type option to a value of json_lines. The connector will read each line as a separate JSON document and will not perform any modifications on any line, thereby avoiding the issue in Flux of JSON documents being unexpectedly altered.

2.4.1

This patch release addresses a single issue:

• The org.slf4j:slf4j-api transitive dependency is forced to be version 2.0.13, ensuring that no occurrences of the 1.x version of that dependency are included in the connector jar. This resolves a logging issue in the Flux application.

2.4.0

This minor release addresses the following items:

1. Can now stream regular files, ZIP files, gzip files, and archive files by setting the new spark.marklogic.streamFiles option to a value of true. Using this option in the reader phase results in the reading of files being deferred until the writer phase. Using this option in the writer phase results in each file being streamed to MarkLogic in a separate request to MarkLogic, thus avoiding ever reading the contents of the file or zip entry into memory.
2. Can now stream documents from MarkLogic to regular files, ZIP files, gzip files, and archive files by setting the same option above - spark.marklogic.streamFiles - to a value of `true. Using this option in the reader phase results in the reading of documents being deferred until the writer phase. Using this option in the writer phase results in each document being streamed from MarkLogic to a file or zip entry, thus avoiding ever reading the contents of the document into memory.
3. Files with spaces in the path are now handled correctly when reading files into MarkLogic. However, when streaming files into MarkLogic, the spaces in the path will be encoded due to a pending server fix.
4. Archive files - zip files containing content and metadata - now contain the metadata entry followed by the content entry for each document. This supports streaming archive files. Archive files generated by version 2.3.x of the connector - with the content entry followed by the metadata entry - can still be read, though they cannot be streamed.
5. Now compiled and tested against Spark 3.5.3.

2.3.1

This patch release addresses the following issues:

1. Can now read document URIs that include non-US-ASCII characters. This was fixed via an upgrade of the Java Client to its 7.0.0 release, whose breaking changes do not have impact on this connector release.
2. Registered collatedString as a known TDE type, thereby avoiding warnings when reading rows from a TDE that uses that type.
3. Significantly improved performance when reading aggregate XML files and extracting a URI value from an element.
4. Fixed bug where a message of "Wrote failed documents to archive file at" was logged when no documents failed.

2.3.0

This minor release provides significant new functionality in support of the 1.0.0 release of the new MarkLogic Flux data movement tool. Much of this functionality is documented in the Flux documentation. We will soon have complete documentation of all the new options in this repository's documentation as well.

In the meantime, the new options in this release are listed below.

Read Options

1. spark.marklogic.read.javascriptFile and spark.marklogic.read.xqueryFile allow for custom code to be read from a file path.
2. spark.marklogic.read.partitions.javascriptFile and spark.marklogic.read.partitions.xqueryFile allow for custom code to be read from a file path.
3. Can now read document rows by specifying a list of newline-delimited URIs via the spark.marklogic.read.documents.uris option.
4. Can now read rows containing semantic triples in MarkLogic via spark.marklogic.read.triples.graphs, spark.marklogic.read.triples.collections, spark.marklogic.read.triples.query, spark.marklogic.read.triples.stringQuery, spark.marklogic.read.triples.uris, spark.marklogic.read.triples.directory, spark.marklogic.read.triples.options, spark.marklogic.read.triples.filtered, and spark.marklogic.read.triples.baseIri.
5. Can now read Flux and MLCP archives by setting spark.marklogic.read.files.type to archive or mlcp_archive.
6. Can control which categories of metadata are read from Flux archives via spark.marklogic.read.archives.categories.
7. Can now specify the encoding of a file to read via spark.marklogic.read.files.encoding.
8. Can now see progress logged of reading data from MarkLogic via spark.marklogic.read.logProgress.
9. Can specify whether to fail on a file read error via spark.marklogic.read.files.abortOnFailure.

Write Options

1. spark.marklogic.write.threadCount has been altered to reflect the common user understanding of "number of threads used to connect to MarkLogic". If you need to specify a thread count per partition, use spark.marklogic.write.threadCountPerPartition.
2. Can now see progress logged of data written to MarkLogic via spark.marklogic.write.logProgress.
3. spark.marklogic.write.javascriptFile and spark.marklogic.write.xqueryFile allow for custom code to be read from a file path.
4. Settingspark.marklogic.write.archivePathForFailedDocuments to a file path will result in any failed documents being added to an archive zip file at that file path.
5. spark.marklogic.write.jsonRootName allows for a root field to be added to a JSON document constructed from an arbitrary row.
6. spark.marklogic.write.xmlRootName and spark.marklogic.write.xmlNamespace allow for an XML document to be constructed from an arbitrary row.
7. Options starting with spark.marklogic.write.json. will be used to configure how the connector serializes a Spark row into a JSON object.
8. Can use spark.marklogic.write.graph and spark.marklogic.write.graphOverride to specify the graph when writing RDF triples to MarkLogic.
9. Deprecated spark.marklogic.write.fileRows.documentType in favor of using spark.marklogic.write.documentType to force a document type on documents written to MarkLogic with an extension unrecognized by MarkLogic.
10. Can use spark.marklogic.write.files.prettyPrint to pretty-print JSON and XML files written by the connector.
11. Can use spark.marklogic.write.files.encoding to write files in a different encoding.
12. Can use spark.marklogic.write.files.rdf.format to specify an RDF type when writing triples to RDF files.
13. Can use spark.marklogic.write.files.rdf.graph to specify a graph when writing RDF files.