Add default value for properties

Author: pprados

docs/docs/integrations/document_loaders/pymupdf.ipynb

@@ -42,8 +42,8 @@
   {
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:01.204685Z",
-     "start_time": "2025-01-16T07:13:01.202180Z"
+     "end_time": "2025-01-16T09:48:32.087401Z",
+     "start_time": "2025-01-16T09:48:32.084843Z"
     }
    },
    "cell_type": "code",
@@ -67,8 +67,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:04.314631Z",
-     "start_time": "2025-01-16T07:13:02.540177Z"
+     "end_time": "2025-01-16T09:48:34.720803Z",
+     "start_time": "2025-01-16T09:48:33.057015Z"
     }
    },
    "source": "%pip install -qU langchain_community pymupdf",
@@ -96,8 +96,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:06.238762Z",
-     "start_time": "2025-01-16T07:13:05.543744Z"
+     "end_time": "2025-01-16T09:48:36.618850Z",
+     "start_time": "2025-01-16T09:48:35.787958Z"
     }
    },
    "source": [
@@ -120,8 +120,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:07.522535Z",
-     "start_time": "2025-01-16T07:13:07.378088Z"
+     "end_time": "2025-01-16T09:48:37.650774Z",
+     "start_time": "2025-01-16T09:48:37.492137Z"
     }
    },
    "source": [
@@ -132,7 +132,7 @@
     {
      "data": {
       "text/plain": [
-       "Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'file_path': './example_data/layout-parser-paper.pdf', 'total_pages': 16, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'creator': 'LaTeX with hyperref', 'producer': 'pdfTeX-1.40.21', 'creationdate': '2021-06-22T01:27:10+00:00', 'moddate': '2021-06-22T01:27:10+00:00', 'trapped': '', 'page': 0}, page_content='LayoutParser: A Unified Toolkit for Deep\\nLearning Based Document Image Analysis\\nZejiang Shen1 (\\x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\\nLee4, Jacob Carlson3, and Weining Li5\\n1 Allen Institute for AI\\[email protected]\\n2 Brown University\\nruochen [email protected]\\n3 Harvard University\\n{melissadell,jacob carlson}@fas.harvard.edu\\n4 University of Washington\\[email protected]\\n5 University of Waterloo\\[email protected]\\nAbstract. Recent advances in document image analysis (DIA) have been\\nprimarily driven by the application of neural networks. Ideally, research\\noutcomes could be easily deployed in production and extended for further\\ninvestigation. However, various factors like loosely organized codebases\\nand sophisticated model configurations complicate the easy reuse of im-\\nportant innovations by a wide audience. Though there have been on-going\\nefforts to improve reusability and simplify deep learning (DL) model\\ndevelopment in disciplines like natural language processing and computer\\nvision, none of them are optimized for challenges in the domain of DIA.\\nThis represents a major gap in the existing toolkit, as DIA is central to\\nacademic research across a wide range of disciplines in the social sciences\\nand humanities. This paper introduces LayoutParser, an open-source\\nlibrary for streamlining the usage of DL in DIA research and applica-\\ntions. The core LayoutParser library comes with a set of simple and\\nintuitive interfaces for applying and customizing DL models for layout de-\\ntection, character recognition, and many other document processing tasks.\\nTo promote extensibility, LayoutParser also incorporates a community\\nplatform for sharing both pre-trained models and full document digiti-\\nzation pipelines. We demonstrate that LayoutParser is helpful for both\\nlightweight and large-scale digitization pipelines in real-word use cases.\\nThe library is publicly available at https://layout-parser.github.io.\\nKeywords: Document Image Analysis · Deep Learning · Layout Analysis\\n· Character Recognition · Open Source library · Toolkit.\\n1\\nIntroduction\\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\\ndocument image analysis (DIA) tasks including document image classification [11,\\narXiv:2103.15348v2  [cs.CV]  21 Jun 2021')"
+       "Document(metadata={'producer': 'pdfTeX-1.40.21', 'creator': 'LaTeX with hyperref', 'creationdate': '2021-06-22T01:27:10+00:00', 'source': './example_data/layout-parser-paper.pdf', 'file_path': './example_data/layout-parser-paper.pdf', 'total_pages': 16, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'moddate': '2021-06-22T01:27:10+00:00', 'trapped': '', 'page': 0}, page_content='LayoutParser: A Unified Toolkit for Deep\\nLearning Based Document Image Analysis\\nZejiang Shen1 (\\x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\\nLee4, Jacob Carlson3, and Weining Li5\\n1 Allen Institute for AI\\[email protected]\\n2 Brown University\\nruochen [email protected]\\n3 Harvard University\\n{melissadell,jacob carlson}@fas.harvard.edu\\n4 University of Washington\\[email protected]\\n5 University of Waterloo\\[email protected]\\nAbstract. Recent advances in document image analysis (DIA) have been\\nprimarily driven by the application of neural networks. Ideally, research\\noutcomes could be easily deployed in production and extended for further\\ninvestigation. However, various factors like loosely organized codebases\\nand sophisticated model configurations complicate the easy reuse of im-\\nportant innovations by a wide audience. Though there have been on-going\\nefforts to improve reusability and simplify deep learning (DL) model\\ndevelopment in disciplines like natural language processing and computer\\nvision, none of them are optimized for challenges in the domain of DIA.\\nThis represents a major gap in the existing toolkit, as DIA is central to\\nacademic research across a wide range of disciplines in the social sciences\\nand humanities. This paper introduces LayoutParser, an open-source\\nlibrary for streamlining the usage of DL in DIA research and applica-\\ntions. The core LayoutParser library comes with a set of simple and\\nintuitive interfaces for applying and customizing DL models for layout de-\\ntection, character recognition, and many other document processing tasks.\\nTo promote extensibility, LayoutParser also incorporates a community\\nplatform for sharing both pre-trained models and full document digiti-\\nzation pipelines. We demonstrate that LayoutParser is helpful for both\\nlightweight and large-scale digitization pipelines in real-word use cases.\\nThe library is publicly available at https://layout-parser.github.io.\\nKeywords: Document Image Analysis · Deep Learning · Layout Analysis\\n· Character Recognition · Open Source library · Toolkit.\\n1\\nIntroduction\\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\\ndocument image analysis (DIA) tasks including document image classification [11,\\narXiv:2103.15348v2  [cs.CV]  21 Jun 2021')"
       ]
      },
      "execution_count": 4,
@@ -146,8 +146,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:08.814226Z",
-     "start_time": "2025-01-16T07:13:08.810836Z"
+     "end_time": "2025-01-16T09:48:38.072178Z",
+     "start_time": "2025-01-16T09:48:38.069508Z"
     }
    },
    "source": [
@@ -160,17 +160,17 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "{'source': './example_data/layout-parser-paper.pdf',\n",
+      "{'producer': 'pdfTeX-1.40.21',\n",
+      " 'creator': 'LaTeX with hyperref',\n",
+      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
+      " 'source': './example_data/layout-parser-paper.pdf',\n",
       " 'file_path': './example_data/layout-parser-paper.pdf',\n",
       " 'total_pages': 16,\n",
       " 'format': 'PDF 1.5',\n",
       " 'title': '',\n",
       " 'author': '',\n",
       " 'subject': '',\n",
       " 'keywords': '',\n",
-      " 'creator': 'LaTeX with hyperref',\n",
-      " 'producer': 'pdfTeX-1.40.21',\n",
-      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
       " 'moddate': '2021-06-22T01:27:10+00:00',\n",
       " 'trapped': '',\n",
       " 'page': 0}\n"
@@ -190,8 +190,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:09.863650Z",
-     "start_time": "2025-01-16T07:13:09.813260Z"
+     "end_time": "2025-01-16T09:48:39.349546Z",
+     "start_time": "2025-01-16T09:48:39.295384Z"
     }
    },
    "source": [
@@ -223,8 +223,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:10.355177Z",
-     "start_time": "2025-01-16T07:13:10.351604Z"
+     "end_time": "2025-01-16T09:48:39.991257Z",
+     "start_time": "2025-01-16T09:48:39.987732Z"
     }
    },
    "source": [
@@ -239,17 +239,17 @@
       "LayoutParser: A Unified Toolkit for DL-Based DIA\n",
       "11\n",
       "focuses on precision, efficiency, and robustness. T\n",
-      "{'source': './example_data/layout-parser-paper.pdf',\n",
+      "{'producer': 'pdfTeX-1.40.21',\n",
+      " 'creator': 'LaTeX with hyperref',\n",
+      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
+      " 'source': './example_data/layout-parser-paper.pdf',\n",
       " 'file_path': './example_data/layout-parser-paper.pdf',\n",
       " 'total_pages': 16,\n",
       " 'format': 'PDF 1.5',\n",
       " 'title': '',\n",
       " 'author': '',\n",
       " 'subject': '',\n",
       " 'keywords': '',\n",
-      " 'creator': 'LaTeX with hyperref',\n",
-      " 'producer': 'pdfTeX-1.40.21',\n",
-      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
       " 'moddate': '2021-06-22T01:27:10+00:00',\n",
       " 'trapped': '',\n",
       " 'page': 10}\n"
@@ -303,8 +303,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:13.015263Z",
-     "start_time": "2025-01-16T07:13:12.962686Z"
+     "end_time": "2025-01-16T09:48:43.180738Z",
+     "start_time": "2025-01-16T09:48:43.132909Z"
     }
    },
    "source": [
@@ -322,17 +322,17 @@
      "output_type": "stream",
      "text": [
       "16\n",
-      "{'source': './example_data/layout-parser-paper.pdf',\n",
+      "{'producer': 'pdfTeX-1.40.21',\n",
+      " 'creator': 'LaTeX with hyperref',\n",
+      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
+      " 'source': './example_data/layout-parser-paper.pdf',\n",
       " 'file_path': './example_data/layout-parser-paper.pdf',\n",
       " 'total_pages': 16,\n",
       " 'format': 'PDF 1.5',\n",
       " 'title': '',\n",
       " 'author': '',\n",
       " 'subject': '',\n",
       " 'keywords': '',\n",
-      " 'creator': 'LaTeX with hyperref',\n",
-      " 'producer': 'pdfTeX-1.40.21',\n",
-      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
       " 'moddate': '2021-06-22T01:27:10+00:00',\n",
       " 'trapped': '',\n",
       " 'page': 0}\n"
@@ -359,8 +359,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:15.907276Z",
-     "start_time": "2025-01-16T07:13:15.852748Z"
+     "end_time": "2025-01-16T09:48:45.358999Z",
+     "start_time": "2025-01-16T09:48:45.305168Z"
     }
    },
    "source": [
@@ -378,17 +378,17 @@
      "output_type": "stream",
      "text": [
       "1\n",
-      "{'source': './example_data/layout-parser-paper.pdf',\n",
+      "{'producer': 'pdfTeX-1.40.21',\n",
+      " 'creator': 'LaTeX with hyperref',\n",
+      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
+      " 'source': './example_data/layout-parser-paper.pdf',\n",
       " 'file_path': './example_data/layout-parser-paper.pdf',\n",
       " 'total_pages': 16,\n",
       " 'format': 'PDF 1.5',\n",
       " 'title': '',\n",
       " 'author': '',\n",
       " 'subject': '',\n",
       " 'keywords': '',\n",
-      " 'creator': 'LaTeX with hyperref',\n",
-      " 'producer': 'pdfTeX-1.40.21',\n",
-      " 'creationdate': '2021-06-22T01:27:10+00:00',\n",
       " 'moddate': '2021-06-22T01:27:10+00:00',\n",
       " 'trapped': ''}\n"
      ]
@@ -414,8 +414,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:18.026765Z",
-     "start_time": "2025-01-16T07:13:17.976302Z"
+     "end_time": "2025-01-16T09:48:47.435722Z",
+     "start_time": "2025-01-16T09:48:47.389Z"
     }
    },
    "source": [
@@ -571,8 +571,8 @@
    "cell_type": "code",
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:22.172739Z",
-     "start_time": "2025-01-16T07:13:20.671450Z"
+     "end_time": "2025-01-16T09:48:52.381641Z",
+     "start_time": "2025-01-16T09:48:50.979344Z"
     }
    },
    "source": [
@@ -592,8 +592,8 @@
   {
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-16T07:13:46.802768Z",
-     "start_time": "2025-01-16T07:13:46.743696Z"
+     "end_time": "2025-01-16T09:49:19.155886Z",
+     "start_time": "2025-01-16T09:48:56.454277Z"
     }
    },
    "cell_type": "code",
@@ -607,45 +607,65 @@
     ")\n",
     "docs = loader.load()\n",
     "\n",
-    "print(docs[5].page_content)"
+    "print(docs[6].page_content)"
    ],
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "6\n",
-      "Z. Shen et al.\n",
-      "Fig. 2: The relationship between the three types of layout data structures.\n",
-      "Coordinate supports three kinds of variation; TextBlock consists of the co-\n",
-      "ordinate information and extra features like block text, types, and reading orders;\n",
-      "a Layout object is a list of all possible layout elements, including other Layout\n",
-      "objects. They all support the same set of transformation and operation APIs for\n",
-      "maximum flexibility.\n",
-      "Shown in Table 1, LayoutParser currently hosts 9 pre-trained models trained\n",
-      "on 5 different datasets. Description of the training dataset is provided alongside\n",
-      "with the trained models such that users can quickly identify the most suitable\n",
-      "models for their tasks. Additionally, when such a model is not readily available,\n",
-      "LayoutParser also supports training customized layout models and community\n",
-      "sharing of the models (detailed in Section 3.5).\n",
-      "3.2\n",
-      "Layout Data Structures\n",
-      "A critical feature of LayoutParser is the implementation of a series of data\n",
-      "structures and operations that can be used to efficiently process and manipulate\n",
-      "the layout elements. In document image analysis pipelines, various post-processing\n",
-      "on the layout analysis model outputs is usually required to obtain the final\n",
-      "outputs. Traditionally, this requires exporting DL model outputs and then loading\n",
-      "the results into other pipelines. All model outputs from LayoutParser will be\n",
-      "stored in carefully engineered data types optimized for further processing, which\n",
-      "makes it possible to build an end-to-end document digitization pipeline within\n",
-      "LayoutParser. There are three key components in the data structure, namely\n",
-      "the Coordinate system, the TextBlock, and the Layout. They provide different\n",
-      "levels of abstraction for the layout data, and a set of APIs are supported for\n",
-      "transformations or operations on these classes.\n"
+      "LayoutParser: A Unified Toolkit for DL-Based DIA\n",
+      "7\n",
+      "Coordinates are the cornerstones for storing layout information. Currently,\n",
+      "three types of Coordinate data structures are provided in LayoutParser, shown\n",
+      "in Figure 2. Interval and Rectangle are the most common data types and\n",
+      "support specifying 1D or 2D regions within a document. They are parameterized\n",
+      "with 2 and 4 parameters. A Quadrilateral class is also implemented to support\n",
+      "a more generalized representation of rectangular regions when the document\n",
+      "is skewed or distorted, where the 4 corner points can be specified and a total\n",
+      "of 8 degrees of freedom are supported. A wide collection of transformations\n",
+      "like shift, pad, and scale, and operations like intersect, union, and is_in,\n",
+      "are supported for these classes. Notably, it is common to separate a segment\n",
+      "of the image and analyze it individually. LayoutParser provides full support\n",
+      "for this scenario via image cropping operations crop_image and coordinate\n",
+      "transformations like relative_to and condition_on that transform coordinates\n",
+      "to and from their relative representations. We refer readers to Table 2 for a more\n",
+      "detailed description of these operations13.\n",
+      "Based on Coordinates, we implement the TextBlock class that stores both\n",
+      "the positional and extra features of individual layout elements. It also supports\n",
+      "specifying the reading orders via setting the parent field to the index of the parent\n",
+      "object. A Layout class is built that takes in a list of TextBlocks and supports\n",
+      "processing the elements in batch. Layout can also be nested to support hierarchical\n",
+      "layout structures. They support the same operations and transformations as the\n",
+      "Coordinate classes, minimizing both learning and deployment effort.\n",
+      "3.3\n",
+      "OCR\n",
+      "LayoutParser provides a unified interface for existing OCR tools. Though there\n",
+      "are many OCR tools available, they are usually configured differently with distinct\n",
+      "APIs or protocols for using them. It can be inefficient to add new OCR tools into\n",
+      "an existing pipeline, and difficult to make direct comparisons among the available\n",
+      "tools to find the best option for a particular project. To this end, LayoutParser\n",
+      "builds a series of wrappers among existing OCR engines, and provides nearly\n",
+      "the same syntax for using them. It supports a plug-and-play style of using OCR\n",
+      "engines, making it effortless to switch, evaluate, and compare different OCR\n",
+      "modules:\n",
+      "1 ocr_agent = lp.TesseractAgent ()\n",
+      "2 # Can be easily\n",
+      "switched to other OCR\n",
+      "software\n",
+      "3 tokens = ocr_agent.detect(image)\n",
+      "The OCR outputs will also be stored in the aforementioned layout data\n",
+      "structures and can be seamlessly incorporated into the digitization pipeline.\n",
+      "Currently LayoutParser supports the Tesseract and Google Cloud Vision OCR\n",
+      "engines.\n",
+      "LayoutParser also comes with a DL-based CNN-RNN OCR model [6] trained\n",
+      "with the Connectionist Temporal Classification (CTC) loss [10]. It can be used\n",
+      "like the other OCR modules, and can be easily trained on customized datasets.\n",
+      "13 This is also available in the LayoutParser documentation pages.\n"
      ]
     }
    ],
-   "execution_count": 13
+   "execution_count": 12
   },
   {
    "cell_type": "markdown",

libs/community/langchain_community/document_loaders/parsers/pdf.py

@@ -655,6 +655,9 @@ def _extract_metadata(self, doc: pymupdf.Document, blob: Blob) -> dict:
         return _purge_metadata(
             dict(
                 {
+                    "producer": "PyMuPDF",
+                    "creator": "PyMuPDF",
+                    "creationdate": "",
                     "source": blob.source,  # type: ignore[attr-defined]
                     "file_path": blob.source,  # type: ignore[attr-defined]
                     "total_pages": len(doc),
@@ -679,7 +682,7 @@ def _extract_images_from_page(
         Returns:
             str: The extracted text from the images on the page.
         """
-        if not self.extract_images:
+        if not self.images_parser:
             return ""
         import pymupdf