6.3.3

📢 Spark NLP 6.3.3: ModernBERT Embeddings, Vector DB Integration, and Layout-Aware Document Processing

Spark NLP 6.3.3 is a feature-packed release aimed at practitioners building modern NLP and multimodal document pipelines. This release introduces ModernBertEmbeddings for dramatically faster and more memory-efficient text embeddings, VectorDBConnector to close the gap between embedding pipelines and vector search infrastructure.

Moreover, we introduce a new suite of document-understanding annotators: LayoutAlignerForVision and LayoutAlignerForText which together enable coherent end-to-end multimodal pipelines over complex documents like PDFs and PowerPoint files. To gain an in-depth walkthrough on how to build use your own pipelines and documents, please see our Medium blog post at Efficient Document Ingestion with Layout Aware Annotators: A Case Study on Mixed-Type Documents.

In addition, a new MultiColumnAssembler can merge multiple annotation column into one and LightPipeline also gains metadata column support for more powerful batch inference workflows.

🔥 Highlights

• ModernBertEmbeddings: A state-of-the-art encoder that is 8x faster and uses 5x less memory than traditional BERT, with native support for sequences up to 8192 tokens which is ideal for long-document NLP tasks.
• VectorDBConnector: Bridges Spark NLP embedding pipelines with external vector databases (initially Pinecone), so teams building semantic search and RAG systems no longer need custom glue code to store embeddings.
• LayoutAlignerForVision and LayoutAlignerForText: New annotators for multimodal document pipelines that spatially align text and images from complex documents, giving downstream Vision-Language Models (VLMs) the coherent context they need to produce better results.
• MultiColumnAssembler: Closes a common pipeline gap when ReaderAssembler splits document content across multiple columns (text, table, image captions). This annotator merges them back into a single column that downstream annotators like AutoGGUFVisionModel expect.
• Enhanced LightPipeline with metadata column support for richer, context-aware inference workflows.

🚀 New Features & Enhancements

ModernBertEmbeddings

Teams working with long documents, code, or large-scale embedding workloads will benefit from ModernBertEmbeddings, which brings the latest generation of bidirectional encoder models to Spark NLP. Based on the paper Smarter, Better, Faster, Longer, ModernBERT was trained on 2 trillion tokens with a native sequence length of up to 8,192 tokens (eight times the limit of classic BERT) enabling faster, cheaper embeddings for longer sequences without truncation.

• Default pretrained model: "modernbert-base" (English)
• 768-dimensional token-level WORD_EMBEDDINGS output

embeddings = ModernBertEmbeddings.pretrained() \
    .setInputCols(["document", "token"]) \
    .setOutputCol("modernbert_embeddings")

See the ModernBertEmbeddings notebook for extended examples, including how to import custom HuggingFace ModernBERT models via ONNX.

VectorDBConnector

For teams building semantic search, retrieval-augmented generation (RAG), or similarity-based recommendation systems, manually bridging Spark NLP with a vector database has historically required custom integration code. VectorDBConnector eliminates this gap by letting you store embeddings from any Spark NLP embedding annotator directly into a vector database as part of the pipeline. It initially supports Pinecone, with more providers planned.

vectorDB = VectorDBConnector() \
    .setInputCols(["document", "sentence_embeddings"]) \
    .setOutputCol("vectordb_result") \
    .setProvider("pinecone") \
    .setIndexName("my-semantic-index") \
    .setNamespace("production") \
    .setIdColumn("doc_id") \
    .setMetadataColumns(["text", "category"]) \
    .setBatchSize(100)

The Pinecone API key is configured via spark.jsl.settings.vectordb.api_key. See the VectorDBConnector Pinecone Demo notebook for a full walkthrough.

LayoutAlignerForVision and LayoutAlignerForText

When processing rich documents like PDFs or PowerPoint presentations, text and images are spatially interleaved. For example, A chart sits next to the paragraph it illustrates, a diagram is surrounded by its explanation. Without layout awareness, VLMs operating on extracted content lose this spatial context entirely. LayoutAlignerForVision and LayoutAlignerForText solve this problem for teams building multimodal document intelligence pipelines.

LayoutAlignerForVision takes document chunks and images extracted by ReaderAssembler and aligns each image with its spatially nearby text paragraphs based on actual page coordinates. It produces three output columns <outputCol>_doc, <outputCol>_image, and <outputCol>_prompt, ready to be fed directly into a VLM (e.g. AutoGGUFVisionModel) for captioning or question answering.

Key parameters:

• setMaxDistance(int): Maximum vertical distance (px) for image-paragraph alignment
• setIncludeContextWindow(bool): Include neighboring paragraphs as context for floating images
• setAddNeighborText(bool): Include aligned text in the prompt output
• setImageCaptionBasePrompt(str): Customize the captioning prompt sent to downstream VLMs
• setNeighborTextCharsWindow(int): Include surrounding text characters as prompt context
• setExplodeDocs(bool): Emit one output row per aligned doc/image pair

LayoutAlignerForText takes the VLM-generated image captions produced after LayoutAlignerForVision and weaves them back into the document's text flow, replacing raw image placeholders with meaningful captions and re-computing begin/end offsets so the resulting document is coherent for downstream NLP tasks.

Key parameters:

• setJoinDelimiter(str) – Delimiter used to join rebuilt text segments
• setExplodeElements(bool) – Emit one output row per aligned text element

For more extended examples and walkthroughs, see refer to the notebook Spark NLP LayoutAligners for Document Understanding and our Medium blog post Efficient Document Ingestion with Layout Aware Annotators: A Case Study on Mixed-Type Documents.

MultiColumnAssembler

When using ReaderAssembler to process documents such as PDFs or PPTX files, content is extracted into separate typed columns: document_text, document_table, and image-related outputs. However, many downstream annotators expect a single input column. Previously, bridging this split required custom Spark transformations. MultiColumnAssembler fills this gap directly within Spark NLP pipelines.

It merges any number of DOCUMENT-type annotation columns into a single output column, preserving all annotation metadata and adding a source_column key to track provenance. Annotations can optionally be sorted by their begin offset using setSortByBegin(True).

multiColumnAssembler = MultiColumnAssembler() \
    .setInputCols(["document_text", "document_table"]) \
    .setOutputCol("merged_document")

Key parameters:

• setInputCols([...]) – List of DOCUMENT-type annotation columns to merge
• setOutputAsAnnotatorType(str) – Override the output annotator type (default: "document")
• setSortByBegin(bool) – Sort merged annotations by begin position (default: False)

Note: Columns using the AnnotationImage schema (i.e., IMAGE-typed columns from ReaderAssembler) are not supported.

See the Merging Annotation Columns notebook for a full walkthrough.

LightPipeline Metadata Support

Users running inference with LightPipeline on data that carries additional context — such as document source, language, or category — previously had no way to pass that context through alongside the text. LightPipeline now supports passing metadata columns alongside text inputs in both annotate() and fullAnnotate(), enabling richer, context-aware inference for applications like routing, filtering, and conditional processing.

New supported call signatures:

• fullAnnotate(text: str, metadata: dict[str, list[str]])
• fullAnnotate(texts: list[str], metadata: list[dict])
• fullAnnotate(texts: list[str], metadata: dict[str, list[str]]) (columnar format)
• Same patterns apply to annotate()

Metadata can be passed as a keyword argument or as a positional trailing argument:

result = light_pipeline.fullAnnotate(
    "U.N. official Ekeus heads for Baghdad.",
    metadata={"source": ["news_article"]}
)

This feature is also surfaced through PretrainedPipeline.annotate() and PretrainedPipeline.fullAnnotate().

🐛 Bug Fixes

• Apache POI upgraded to 5.4.1: The Apache POI dependency used by document readers has been upgraded from 4.1.2 to 5.4.1 (poi-ooxml-full) to avoid deprecated dependencies.

...

6.3.2

📢 Spark NLP 6.3.2: Scala 2.13 Support, Layout-Aware Images, and Enhanced LightPipeline Tracking

Spark NLP 6.3.2 is a foundational release that introduces official support for Scala 2.13, alongside important improvements in document layout understanding and lightweight inference workflows.
This release improves long-term model portability through JSON-based serialization, enriches document image extraction with spatial metadata, and enhances LightPipeline with document ID tracking and output filtering.

🔥 Highlights

• Official Scala 2.13 support
• Layout-aware image extraction with spatial coordinates added to Reader2Image for HTML, DOCX, and PPTX documents.
• Enhanced LightPipeline with document ID propagation and output column filtering for better batch inference workflows.

🚀 New Features & Enhancements

Scala 2.13 Support

Spark NLP now supports Scala 2.13 with this release! This will enable you to run your Spark NLP pipelines on Spark versions that run on Scala 2.13, such as used by Databricks and Dataproc. See our Installation Instructions for Scala 2.13 on how to use it with our project.

There are some things you have to consider when using the Scala 2.13 version

1. You need to adjust your dependency from spark-nlp_2.12 to spark-nlp_2.13.
2. If you install PySpark from PyPi, then the session will be Scala 2.12 by default. If you need to start a Scala 2.13 instance, you can set the SPARK_HOME environment variable to a Spark Scala 2.13 installation, or install PySpark from the official Spark archives.
3. If you want to load DependencyParserModel or TextMatcherModel from Scala 2.12 into Scala 2.13, you will need to manually export them again with the latest version. See the notebook

Layout-Aware Image Metadata in Reader2Image

The Reader2Image annotator now extracts spatial image coordinates from rich document formats, adding layout awareness to image annotations.

• Supported formats:
  • HTML
  • Word (DOCX)
  • PowerPoint (PPTX)
• New metadata fields:
  • x, y, width, height
• Coordinates are included alongside existing metadata such as image format, type, and DOM position

This enables:

• Layout-aware document and multimodal pipelines
• Visual reconstruction of documents
• More accurate association of images with surrounding text content

Document ID Support in LightPipeline

LightPipeline now supports passing document IDs together with text inputs, improving traceability in batch and production inference scenarios.

Key capabilities:

• New overloads:
  • fullAnnotate(ids, texts)
  • annotate(ids, texts)
• Document IDs are propagated as annotation metadata (doc_id)
• New output_cols parameter to restrict returned annotation types

Benefits:

• Reliable document-to-result mapping
• Easier debugging and downstream integration
• Reduced memory usage through selective outputs

Existing LightPipeline usage remains unchanged and backward compatible.

🐛 Bug Fixes

• Fix out of memory error when copying big models to a cloud storage

❤️ Community Support

• Slack – real-time discussion with the Spark NLP community and team
• GitHub – issue tracking, feature requests, and contributions
• Discussions – community ideas and showcases
• Medium – latest Spark NLP articles and tutorials
• YouTube – educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.3.2

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.2

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.2

Maven

Supported on on Apache Spark 3.x.

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.3.2</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.3.2.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.3.2.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.3.2.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.3.2.jar

What's Changed

• [SPARKNLP-1136] JSON Serialization for Features #14722 by @DevinTDHa
• [SPARKNLP-1329] Adding image coordinates to metadata for Reader2Image #14725 by @danilojsl
• [SPARKNLP-1333] Adding ids input for LightPipeline #14726 by @danilojsl

Full Changelog: 6.3.1...6.3.2

6.3.1

📢 Spark NLP 6.3.1: LLM Backend Upgrade and Document Processing Improvements

Spark NLP 6.3.1 focuses on strengthening distributed local LLM inference by upgrading the jsl-llamacpp backend to a newer llama.cpp release, while also delivering important improvements in document structure handling and metadata consistency.

This enables you to use the latest LLMs and embeddings compatible with llama.cpp and perform advanced ingestion of tables and images.

🔥 Highlights

• Upgraded jsl-llamacpp backend to llama.cpp tag b7247, bringing upstream performance improvements, stability fixes, and expanded model compatibility for local LLM inference.
• Improved Reader2X annotator capabilities with structural position metadata for tables and images and integration with AutoGGUFVisionModel
  • For an exhaustive overview on how to use Spark NLP for unstructured document ingestion, see our blog post Evaluating Document AI Frameworks: Spark NLP vs Unstructured for Large-Scale Text Processing

🚀 New Features & Enhancements

LLM Backend Upgrade (llama.cpp)

The jsl-llamacpp backend has been upgraded to llama.cpp tag b7247, applying upstream fixes and enabling the use of the latest LLMs. These benefit distributed LLM workloads in Spark NLP and affects the annotators AutoGGUFModel, AutoGGUFEmbeddings, AutoGGUFVisionModel, AutoGGUFReranker:

• Performance and memory improvements, bug fixes from upstream llama.cpp for offline LLM inference within Spark NLP pipelines
• Better support for newer GGUF/GGML model variants. This means you can now load models such as gpt-oss, Qwen3 and embeddinggemma.

Structural Metadata for Document Readers

Previously, our document parsers (HTMLReader, XMLReader, WordReader, PowerPointReader, ExcelReader) relied heavily on positional or page-based coordinates for layout metadata.
However, non-PDF formats such as HTML, XML, DOC(X), PPT(X), and XLS(X) do not have fixed pages
To ensure deterministic element referencing and structural traceability across all document types, we needed to adopt a unified DOM-like metadata model.

This change standardizes metadata extraction so every element can be uniquely identified and re-located within its source document, independent of visual layout.

These additions enable layout-aware downstream processing and more precise filtering especially for HTML and rich document formats.

Reader2Image Integration with AutoGGUFVisionModel

Previously, you could use Reader2Image to ingest images from various file formats into Spark NLP. However, processing was limited to Spark NLP native VLM implementations (such as Qwen2VLTransformer).

Reader2Image now supports interoperability with our llama.cpp backend with AutoGGUFVisionModel by introducing flexible handling of encoded vs. decoded image bytes and optional prompt output.

• Added a new boolean parameter useEncodedImageBytes to control whether the image result stores:
  • true: Encoded (compressed) file bytes for models like AutoGGUFVisionModel
  • false: Decoded pixel matrix for models such as Qwen2VLTransformer
• outputPromptColumn parameter to optionally output a separate prompt column containing text prompts as Spark NLP Annotations. This is the required format for AutoGGUFVisionModel.

Platform Setup Documentation

Added official documentation and instructions for setting up and running Spark NLP on Microsoft Fabric, simplifying configuration and improving developer onboarding on the platform. You can see them at Spark NLP - Installation

🐛 Bug Fixes

• Sentence metadata is now consistently included in DocumentAssembler outputs when using LightPipeline.
• Fixed an issue where resetting the cache in ResourceDownloader could fail under certain conditions.
• Fixed a document parsing bug where some HTML elements (such as section titles or diagnosis entries) could appear multiple times in the parsed output.
• Improved robustness when loading ONNX BertEmbeddings models with non-standard output tensor names.

❤️ Community Support

• Slack – real-time discussion with the Spark NLP community and team
• GitHub – issue tracking, feature requests, and contributions
• Discussions – community ideas and showcases
• Medium – latest Spark NLP articles and tutorials
• YouTube – educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.3.1

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.3.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.3.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.3.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.3.1

Maven

Supported on on Apache Spark 3.x.

spark-nlp

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

spark-nlp-gpu

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

spark-nlp-silicon

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

spark-nlp-aarch64

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.3.1</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.3.1.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.3.1.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.3.1.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.3.1.jar

What's Changed

• #14705 by @danilojsl
• #14710 by @danilojsl
• #14711 by @AbdullahMubeenAnwar
• #14712 by @mehmetbutgul
• #14713 by @danilojsl
• #14714 by @mehmetbutgul
• #14719 by @DevinTDHa

Full Changelog: 6.3.0...6.3.1

6.2.3

📢 Spark NLP 6.2.3: Further Improvements for NerDL

Spark NLP 6.2.3 introduces targeted improvements to training performance and stability of NerDLApproach and bug fixes for CamemBertForTokenClassification.

NerDLApproach now uses new internal data-loading behavior, and improving training speed and preventing out-of-memory errors.

🔥 Highlights

Enhanced NerDLApproach training performance through threaded data loading and optimized partitioning.

🚀 New Features & Enhancements

NerDLApproach Training Optimizations

Significant performance improvements for training of NerDLApproach:

Threaded Data Loading: When enabling the memory optimizer (setEnableMemoryOptimizer(true)), data can now be pre-fetched through a threaded data loader. By default, it is disabled but can be tuned by using:

.setPrefetchBatches(int)

By tuning this parameter (for example 20 batches), you can get training time reductions of about 10%.

Optimized Partitioning Strategy: NerDLApproach now applies optimized dataframe partitioning when using the memory optimizer (setEnableMemoryOptimizer(true)) by default, improving parallelization efficiency during training and preventing out-of-memory errors.

For manual tuning of the input data frames, this behavior can be disabled with:

.setOptimizePartitioning(false)

🐛 Bug Fixes

• CamemBertForTokenClassification: Fixed an issue with expected input types during inference.

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.3

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.3

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.3

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.3</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.3.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.3.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.3.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.3.jar

What's Changed

• #14701 by @ahmedlone127
• #14699 by @DevinTDHa

Full Changelog: 6.2.2...6.2.3

6.2.2

📢 Spark NLP 6.2.2: Bugfix Release

Spark NLP 6.2.2 brings bug fixes to WordEmbeddings and NerDLApproach logging.

🐛 Bug Fixes

• WordEmbeddings: Fixed a bug where WordEmbeddings would duplicate input tokens in the output
• NerDLApproach: Fixed a bug during logging, that would show inaccurate training/validation counts.

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.2

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.2

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.2

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.2</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.2.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.2.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.2.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.2.jar

What's Changed

• #14693 by @ahmedlone127

Full Changelog: 6.2.0...6.2.2

6.2.1

📢 Spark NLP 6.2.1: Enhanced hierarchical document processing and training optimizations

Spark NLP 6.2.1 brings significant improvements to document ingestion with expanded hierarchical support, XML processing enhancements, and optimizations for NerDL training. This release builds on the foundation of 6.2.0, continuing to focus on structure-awareness, flexibility, and performance for production NLP pipelines.

🔥 Highlights

• Hierarchical Document Processing: Extended support for PDF, Word, and Markdown with parent-child element relationships
• NerDLApproach Training Optimizations: Reduced memory footprint and improved training performance with BERT based embeddings
• Improved Document Output Format: Single document annotations by default for more intuitive behavior with large documents
• Enhanced XML Reading: Attribute extraction and improved tag handling in Reader2Doc

🚀 New Features & Enhancements

Hierarchical Support for Multiple Document Formats

Building on the HTMLReader hierarchical features introduced in 6.2.0, this release extends structured element tracking to additional document formats:

• Reader2Doc now supports hierarchical processing for PDF, Microsoft Word, and Markdown files

• Each extracted element includes:

  • element_id: Unique UUID identifier per element
  • parent_id: References the parent element's ID for logical document structure

• Enables tree-like navigation and contextual understanding of document hierarchy:

  Chapter 1
   ├── Narrative Text A
   ├── Narrative Text B
  Chapter 2
   ├── Paragraph C
  

• Supports advanced use cases including hierarchical retrieval, graph-based indexing, and multi-level document analysis

• Metadata propagation ensures downstream annotators maintain structural relationships

NerDLApproach Training Optimizations

Significant performance improvements for training of NerDLApproach:

• Reduced Memory Usage with BERT based embeddings: Optimized output embeddings allocations, lowering peak memory footprint during training
• Automatic Dataset Caching: When using setEnableMemoryOptimizer(true) with maxEpoch > 1, input datasets are automatically cached to improve training speed
• Graph Metadata Reuse: NerDLGraphChecker now populates TensorFlow graph metadata that NerDLApproach can reuse, reducing redundant computations during training initialization

With all these improvements you can expect up half the memory consumption and training time on RAM constrained environments (when using setEnableMemoryOptimizer(true)). For larger distributed datasets, the effect will be more pronounced.

XML Reader and Reader2Doc Enhancements

• Single Document Output by Default: Reader2Doc now creates single document annotations per file by default, providing more expected behavior when processing large documents

  • Lines are joined by newline character \n by default, configurable via new setJoinString(string) parameter for custom separators
  • Automatically includes specified attribute values in document output

• Improved Tag Handling: XML reader now ignores empty tags without text content, reducing noise in parsed output

• Enhanced content type handling for application/xml documents

• XML Tag Attribute Extraction: New setExtractTagAttributes(attributes: list[str]) parameter enables extraction of XML attribute values. Example:

  <bookstore>
      <book category="children">
          <title lang="en">Harry Potter</title>
          <author>J K. Rowling</author>
          <year>2005</year>
          <price>29.99</price>
      </book>
      <book category="web">
          <title lang="en">Learning XML</title>
          <author>Erik T. Ray</author>
          <year>2003</year>
          <price>39.95</price>
      </book>
  </bookstore>

  We can extract category and lang values with the Reader2Doc Config

  reader2doc = Reader2Doc() \
      .setContentType("application/xml") \
      .setContentPath("../src/test/resources/reader/xml/test.xml") \
      .setOutputCol("document") \
      .setExtractTagAttributes(["category", "lang"])

  Resulting in

  children
  en
  Harry Potter
  J K. Rowling
  2005
  29.99
  web
  en
  Learning XML
  Erik T. Ray
  2003
  39.95
  

🐛 Bug Fixes

• Colab Environment Setup: Added Java installation to Colab setup script for improved out-of-the-box compatibility

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.1

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.1

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.1
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.1

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.1</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.1.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.1.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.1.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.1.jar

What's Changed

• #14681 by @danilojsl
• #14682 by @AbdullahMubeenAnwar
• #14685 by @DevinTDHa
• #14691 by @DevinTDHa

Full Changelog: 6.2.0...6.2.1

6.2.0

📢 Spark NLP 6.2.0: A new stage for unstructured document ingestion and processing at scale

Spark NLP 6.2.0 introduces key upgrades across entity extraction, document normalization, HTML reading, and GGUF-based models. To recap, since the releases of Spark NLP 6.1 you can:

• Infer quantized cutting-edge LLMs and VLMs such as Gemma 3, Phi-4, Llama 3.1, Qwen 2.5
• Rerank documents using llama.cpp with AutoGGUFReranker
• Ingest unstructured documents of diverse formats
  • Reader2Doc: streamlines the process of loading and integrating diverse file formats (PDFs, Word, Excel, PowerPoint, HTML, Text, Email, Markdown) directly into Spark NLP pipelines with a unified and flexible interface.
  • Reader2Table: streamlines tabular data extraction from multiple document formats with seamless pipeline integration.
  • Reader2Image: extract structured image content from various document types

Spark NLP release 6.2.0 further focuses on automation, structure-awareness, and resource efficiency, making pipelines easier to configure, manage, and extend.

🔥 Highlights

• Auto Modes for EntityRuler and DocumentNormalizer: automatic regex and text-cleaning presets for faster setup.
• Hierarchical Element Tracking in HTMLReader: adds element and parent identifiers for structure-aware document processing.
• Resource Management for AutoGGUF Annotators: improved control and cleanup of llama.cpp-based models.

🚀 New Features & Enhancements

EntityRulerModel and DocumentNormalizer Auto Modes

EntityRulerModel

• Added autoMode parameter to enable predefined regex entity groups ("network_entities", "communication_entities", "media_entities", "email_entities", "all_entities").
• Added extractEntities parameter to filter entities within auto modes.
• Automatically applies case-insensitive regex presets and falls back to manual mode if not specified.
• Retains full backward compatibility with JSON or RocksDB-based rules.

DocumentNormalizer

• Added presetPattern and autoMode parameters to apply built-in text cleaning patterns.
• New modes include "light_clean", "document_clean", "social_clean", "html_clean", and "full_auto".
• Enables quick application of multiple cleaning operations without manual configuration.

Together, these additions significantly reduce boilerplate setup for common text extraction and normalization workflows.

Hierarchical Element Identification in HTMLReader

• Introduced element_id and parent_id metadata fields for each parsed HTML element.
• Enables explicit structural relationships (e.g., title → paragraph → link) for hierarchical retrieval and contextual reasoning.
• Supports graph-based indexing, hybrid search, and multi-level document analysis.
• Metadata propagation improvements ensure Sentence Detector outputs also retain upstream hierarchy information.

AutoGGUF Annotator Enhancements

For AutoGGUFModel, AutoGGUFVision, AutoGGUFEmbeddings, AutoGGUFReranker

• Added close() method to explicitly release llama.cpp model resources, preventing memory retention in long-running sessions.
• Introduced setRemoveThinkingTag(tag: String) parameter to remove internal <think>...</think> sections from model outputs.
  • Regex pattern: (?s)<$tag>.+?</$tag>
  • Simplifies downstream processing for chat and reasoning models.

🐛 Bug Fixes

• RobertaEmbeddings Warmup Test - fixed token sequence bug where unknown tokens caused initialization errors.

❤️ Community Support

• Slack - real-time discussion with the Spark NLP community and team
• GitHub - issue tracking, feature requests, and contributions
• Discussions - community ideas and showcases
• Medium - latest Spark NLP articles and tutorials
• YouTube - educational videos and demos

💻 Installation

Python

pip install spark-nlp==6.2.0

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.2.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.2.0

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.2.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.0
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.2.0

Maven

<dependency>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.12</artifactId>
  <version>6.2.0</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.2.0.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.2.0.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.2.0.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.2.0.jar

What's Changed

• #14671 by @DevinTDHa
• #14672 by @DevinTDHa
• #14674 by @danilojsl
• #14675 by @danilojsl
• #14677 by @ahmedlone127
• #14673 by @AbdullahMubeenAnwar

Full Changelog: 6.1.5...6.2.0

6.1.5

📢 Spark NLP 6.1.5: Smarter Readers and More Resilient Pipelines

Spark NLP 6.1.5 focuses on improving data ingestion reliability and pipeline flexibility. This release enhances reader components with better fault tolerance, broader input support, and introduces a new ReaderAssembler annotator for streamlined integration. Several key fixes also improve model loading and stability in distributed environments.

🔥 Highlights

• New ReaderAssembler Annotator: Unify multiple reader annotators into one configurable component for simpler and cleaner ingestion pipelines.

🚀 New Features & Enhancements

Reader Pipeline Enhancements

• ReaderAssembler Annotator
  A new meta-annotator that unifies Reader2X components (e.g., Reader2Doc, Reader2Image, Reader2Table) under a single interface.

  • Automatically selects the right reader(s) based on configuration.
  • Supports declarative assembly of reading stages.
  • Provides parameters for reader selection, fallback rules, and error handling.
    This simplifies pipeline construction and improves maintainability for multi-format ingestion workflows. (Link to notebook)

• Support for String Input Columns in Readers (SPARKNLP-1291)
  Spark NLP readers only supported inputs via file paths. That means if you already had a DataFrame with text content (say from another pipeline or a preliminary load), you had to write it to disk just to let the reader ingest it. This adds friction and overhead, especially in streaming or in-memory pipelines.

  With this change, you can:

  • Feed raw text stored in a DataFrame column directly into Spark NLP readers — zero I/O overhead when not needed.
  • Simplify workflows and pipelines (no need for temporary file staging just to “read” back data).
  • Improve performance and resource usage in scenarios where input is already available as strings (e.g. generated, preprocessed, or coming from another system).
  • Make the reader APIs more flexible and general-purpose.

• Fault-Tolerant XML Reader
  The XML reader now skips malformed XML fragments (e.g., mismatched tags, missing closures, invalid characters) instead of failing the job.
  Enhanced error handling ensures more resilient ingestion of imperfect real-world data.

🐛 Bug Fixes

• GGUF Model Loading Duplication
  Fixed an issue in FeaturesFallbackReader that caused duplicate loading or missing model files when calling .pretrained() on GGUF-based annotators such as AutoGGUFModel and rerankers, especially in Databricks environments.

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

💻 Installation

Python

pip install spark-nlp==6.1.5

Spark Packages

CPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.5

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.5

Apple Silicon

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.5

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.5
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.5

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.5</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.5.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.5.jar
• Apple Silicon: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.5.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.5.jar

What's Changed

• #14665 by @danilojsl
• #14666 by @danilojsl
• #14668 by @danilojsl
• #14667 by @C-K-Loan, @DevinTDHa

Full Changelog: 6.1.4...6.1.5

6.1.4

📢 Spark NLP 6.1.4: Advancing Multimodal Workflows with Reader2Image

We are excited to announce the release of Spark NLP 6.1.4!
This version introduces a powerful new annotator, Reader2Image, which extends Spark NLP’s universal ingestion capabilities to embedded images across a wide range of document formats. With this release, Spark NLP users can now seamlessly integrate text and image processing in the same pipeline, unlocking new opportunities for vision-language modeling (VLM), multimodal search, and document understanding.

---

🔥 Highlights

• New Reader2Image Annotator: Extract and structure image content directly from documents like PDFs, Word, PowerPoint, Excel, HTML, Markdown, and Email files.
• Multimodal Pipeline Expansion: Build workflows that combine text, tables, and now images for comprehensive document AI applications.
• Consistent Structured Output: Access image metadata (filename, dimensions, channels, mode) alongside binary image data in Spark DataFrames, fully compatible with other visual annotators.

---

🚀 New Features & Enhancements

Document Ingestion

• Reader2Image Annotator
  A new multimodal annotator designed to parse image content embedded in structured documents. Supported formats include:

  • PDFs
  • Word (.doc/.docx)
  • Excel (.xls/.xlsx)
  • PowerPoint (.ppt/.pptx)
  • HTML & Markdown (.md)
  • Email files (.eml, .msg)

  Output Fields:

  • File name
  • Image dimensions (height, width)
  • Number of channels
  • Mode
  • Binary image data
  • Metadata

  This enables seamless integration with vision-language models (VLMs), multimodal embeddings, and downstream Spark NLP annotators, all within the same distributed pipeline.

---

🐛 Bug Fixes

• None

---

❤️ Community Support

• Slack For live discussion with the Spark NLP community and the team
• GitHub Bug reports, feature requests, and contributions
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
• Medium Spark NLP articles
• JohnSnowLabs official Medium
• YouTube Spark NLP video tutorials

---

⚙️ Installation

Python

pip install spark-nlp==6.1.4

Spark Packages

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.4

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.4

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.4

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.4
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.4

Maven

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.4</version>
</dependency>

• GPU: spark-nlp-gpu_2.12:6.1.4
• Apple Silicon: spark-nlp-silicon_2.12:6.1.4
• AArch64: spark-nlp-aarch64_2.12:6.1.4

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.4.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.4.jar
• M1/M2: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.4.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.4.jar

---

📊 What’s Changed

• [SPARKNLP-1261] Introducing Reader2Image Annotator (#14658) by @danilojsl

Full Changelog: 6.1.3...6.1.4

What's Changed

• SPARKNLP-1261 Introducing Reader2Image Annotator by @danilojsl in #14658

Full Changelog: 6.1.3...6.1.4

6.1.3

📢 Spark NLP 6.1.3: NerDL Graph Checker, Reader2Doc Enhancements, Ranking Finisher

We are pleased to announce Spark NLP 6.1.3, introducing a new graph validation annotator for NER training, enhancements to Reader2Doc for flexible document handling, and a new ranking finisher for AutoGGUFReranker outputs. This release focuses on improving training robustness, document processing flexibility, and retrieval ranking capabilities.

🔥 Highlights

• New NerDLGraphChecker annotator to validate NER training graphs before training starts.
• Reader2Doc enhancements with options for consolidated output and filtering.
• New AutoGGUFRerankerFinisher for ranking, filtering, and normalizing reranker outputs.

🚀 New Features & Enhancements

Named Entity Recognition (NER)

NerDLGraphChecker:
A new annotator that validates whether a suitable NerDL graph is available for a given training dataset before embeddings or training start. This helps avoid wasted computation in custom training scenarios. (Link to notebook)

• Must be placed before embedding or NerDLApproach annotators.
• Requires token and label columns in the dataset.
• Automatically extracts embedding dimensions from the pipeline to validate graph compatibility.

Document Processing

Reader2Doc Enhancements:
New configuration options provide more control over output formatting:

• outputAsDocument: Concatenates all sentences into a single document.
• excludeNonText: Filters out non-textual elements (e.g., tables, images) from the document.

Ranking & Retrieval

AutoGGUFRerankerFinisher:
A finisher for processing AutoGGUFReranker outputs, adding advanced ranking and filtering capabilities (Link to notebook):

• Top-k document selection.
• Score threshold filtering.
• Min-max score normalization (0–1 range).
• Sorting by relevance score.
• Rank assignment in metadata while preserving document structure.

🐛 Bug Fixes

None.

❤️ Community Support

• Slack Live discussion with the Spark NLP community and team
• GitHub Bug reports, feature requests, and contributions
• Discussions Share ideas and engage with other community members
• Medium Spark NLP technical articles
• JohnSnowLabs Medium Official blog
• YouTube Spark NLP tutorials and demos

Installation

Python

pip install spark-nlp==6.1.3

Spark Packages

spark-nlp on Apache Spark 3.0.x–3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:6.1.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:6.1.3

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:6.1.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.3
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:6.1.3

Maven

spark-nlp:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>6.1.3</version>
</dependency>

FAT JARs

• CPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-6.1.3.jar
• GPU: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-6.1.3.jar
• M1: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-silicon-assembly-6.1.3.jar
• AArch64: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-aarch64-assembly-6.1.3.jar

What's Changed

• #14657
• #14656
• #14653

Full Changelog: 6.1.2...6.1.3