Update text splitter chunking logic and add full test coverage

Author: pamelafox

CONTRIBUTING.md

@@ -73,7 +73,7 @@ Once tests are passing, generate a coverage report to make sure your changes are
 
 ```shell
 pytest --cov --cov-report=xml && \
-diff-cover coverage.xml --format html:coverage_report.html && \
+diff-cover coverage.xml --html-report coverage_report.html && \
 open coverage_report.html
 ```
 

app/backend/prepdocslib/textsplitter.py

@@ -0,0 +1,70 @@
+[
+  {
+    "text": "# Financial Market Analysis Report 2023\n\nAn In-Depth Exploration of Stocks, Cryptocurrencies, and Commodities\nPrepared by: Contoso Financial Analytics",
+    "page_num": 0
+  },
+  {
+    "text": "## Executive Summary\n\n. In this comprehensive report, Contoso Financial Analytics provides a\ndeep dive into the financial markets of 2023, focusing on the trends\nand fluctuations within stocks, cryptocurrencies, and commodities.\nOur analysis covers historical patterns, current market conditions, and\nfuture predictions, offering valuable insights for investors, analysts,\nand financial enthusiasts. This report leverages advanced data\nanalytics to present a clear picture of the complex interplay between\ndifferent financial markets and their potential trajectories",
+    "page_num": 1
+  },
+  {
+    "text": "## Introduction to Financial Markets\n\n\n<figure><figcaption>Global Financial Market Distribution (2023)<br>This pie chart represents the distribution of investments across four categories: Stocks, Bonds, Cryptocurrencies, and Commodities. The chart is divided into four colored sections, each representing a different category. Stocks are shown in blue, Bonds in orange, Cryptocurrencies in gray, and Commodities in yellow. The chart visually indicates the proportion of each investment type within a portfolio.<br><br><table><tr><th>Category</th><th>Color</th></tr><tr><td>Stocks</td><td>Blue</td></tr><tr><td>Bonds</td><td>Orange</td></tr><tr><td>Cryptocurrencies</td><td>Gray</td></tr><tr><td>Commodities</td><td>Yellow</td></tr></table></figcaption></figure>",
+    "page_num": 2
+  },
+  {
+    "text": "\n\n\nThe global financial market is a vast and intricate network of\nexchanges, instruments, and assets, ranging from traditional stocks\nand bonds to modern cryptocurrencies and commodities. Each\nsegment plays a crucial role in the overall economy, and their\ninteractions can have profound effects on global financial stability.\nThis section provides an overview of these segments and sets the\nstage for a detailed analysis",
+    "page_num": 2
+  },
+  {
+    "text": "## Stock Market Overview\n\n\n<figure><figcaption><br><p><strong>5-Year Trend of the S&amp;P 500 Index</strong></p> <p>This line chart shows the trend of the S&amp;P 500 Index over a five-year period from 2018 to 2022. The index starts at around 2500 in 2018, rises steadily to a peak of about 4500 in 2021, and then declines slightly to approximately 4000 in 2022.</p> <table><tr><th>Year</th><th>S&amp;P 500 Index</th></tr><tr><td>2018</td><td>2500</td></tr><tr><td>2019</td><td>3000</td></tr><tr><td>2020</td><td>3500</td></tr><tr><td>2021</td><td>4500</td></tr><tr><td>2022</td><td>4000</td></tr></table></figcaption></figure>",
+    "page_num": 3
+  },
+  {
+    "text": "\n\n\nThe stock market is often considered the economy's\nheartbeat, reflecting corporate health and investor\nsentiment. Over the past five years, the S&P 500 index has\nexperienced significant volatility, with notable peaks and\ntroughs corresponding to various economic events. This\noverview examines the key factors that have influenced\nthe stock market's performance and what they indicate\nabout the economy's state",
+    "page_num": 3
+  },
+  {
+    "text": "## Cryptocurrency Market Dynamics\n\n\n<figure><figcaption>Price Fluctuations of Bitcoin and Ethereum (Last 12 Months)<br><p>This line graph shows two data series over the months from January to December. The blue line represents a data series that starts at around 32,500 in January, peaks in May at about 42,500, dips in July, and then rises steadily to approximately 47,500 in December. The orange line represents a much lower data series, remaining relatively flat throughout the year, starting at around 2,500 in January and ending slightly above 2,500 in December.</p></figcaption></figure>",
+    "page_num": 4
+  },
+  {
+    "text": "\n\n\nCryptocurrencies have emerged as a new asset\nclass, captivating investors with their potential for\nhigh returns and their role in the future of finance.\nThis section explores the price dynamics of major\ncryptocurrencies like Bitcoin and Ethereum,\nanalyzing the factors driving their volatility and the\nimplications for the broader financial market.\n\n\n<figure><figcaption><br>The image shows a legend with two colored lines and labels. A blue line is labeled \"Bitconin\" and an orange line is labeled \"Ethereum.\" This legend is likely used to differentiate between two data sets or categories in a chart or graph, with \"Bitconin\" and \"Ethereum\" representing different entities or variables.</figcaption></figure>",
+    "page_num": 4
+  },
+  {
+    "text": "### Commodity Market Fluctuations\n\n\n<figure><figcaption>Price Changes of Oil, Gold, and Wheat<br>This is a horizontal bar chart showing the annual percentage change in prices for Wheat, Gold, and Oil from 2014 to 2022. The chart uses different colors to represent each commodity: gray for Wheat, orange for Gold, and blue for Oil. The x-axis represents the percentage change, ranging from -25% to 35%, while the y-axis lists the years from 2014 to 2022.\n\n<table>\n<tr><th>Year</th><th>Wheat</th><th>Gold</th><th>Oil</th></tr>\n<tr><td>2022</td><td>5%</td><td>2%</td><td>0%</td></tr>\n<tr><td>2021</td><td>3%</td><td>4%</td><td>30%</td></tr>\n<tr><td>2020</td><td>1%</td><td>5%</td><td>-20%</td></tr>\n<tr><td>2019</td><td>2%</td><td>3%</td><td>10%</td></tr>\n<tr><td>2018</td><td>0%</td><td>1%</td><td>15%</td></tr>\n<tr><td>2017</td><td>4%</td><td>2%</td><td>5%</td></tr>\n<tr><td>2016</td><td>3%</td><td>6%</td><td>-5%</td></tr>\n<tr><td>2015</td><td>1%</td><td>0%</td><td>10%</td></tr>\n<tr><td>2014</td><td>2%</td><td>5%</td><td>-10%</td></tr>\n</table></figcaption></figure>",
+    "page_num": 5
+  },
+  {
+    "text": "\n\n\nCommodities such as oil, gold, and\nwheat are fundamental to the global\neconomy, influencing everything from\nenergy costs to food prices. This section\ndelves into the trends and factors\naffecting commodity prices, including\ngeopolitical events, supply-chain\ndisruptions, and environmental factors,\nproviding a comprehensive view of this\ncrucial market segment.",
+    "page_num": 5
+  },
+  {
+    "text": "### Interplay Between Different Market Segments\n\n\n<figure><table><tr><th></th><th>S&amp;P 500</th><th>NASDAQ</th><th>Bitcoin</th><th>Ethereum</th><th>Oil</th><th>Gold</th></tr><tr><td>S&amp;P 500</td><td>1</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>NASDAQ</td><td>0.95</td><td>1</td><td></td><td></td><td></td><td></td></tr><tr><td>Bitcoin</td><td>0.3</td><td>0.4</td><td>1</td><td></td><td></td><td></td></tr><tr><td>Ethereum</td><td>0.35</td><td>0.45</td><td>0.9</td><td>1</td><td></td><td></td></tr><tr><td>Oil</td><td>0.6</td><td>0.65</td><td>0.2</td><td>0.25</td><td>1</td><td></td></tr><tr><td>Gold</td><td>-0.2</td><td>-0.15</td><td>-0.1</td><td>-0.05</td><td>-0.3</td><td>1</td></tr></table></figure>",
+    "page_num": 6
+  },
+  {
+    "text": "\n\n\nFinancial markets are interconnected, with movements in one segment often influencing others. This\nsection examines the correlations between stock indices, cryptocurrency prices, and commodity prices,\nrevealing how changes in one market can have ripple effects across the financial ecosystem.",
+    "page_num": 6
+  },
+  {
+    "text": "### Impact of Macroeconomic Factors\n\n\n<figure><figcaption>Impact of Interest Rates, Inflation, and GDP Growth on Financial Markets<br><p>The image is a line graph titled \"On Financial Markets\" showing the trends of Interest Rates %, Inflation Data %, and GDP Growth % from 2018 to 2023. The graph has three lines representing each of these metrics over the years.</p>\n\n<table>\n<thead>\n<tr><th>Year</th><th>Interest Rates %</th><th>Inflation Data %</th><th>GDP Growth %</th></tr>\n</thead>\n<tbody>\n<tr><td>2018</td><td>2</td><td>2</td><td>3</td></tr>\n<tr><td>2019</td><td>2</td><td>2.5</td><td>2</td></tr>\n<tr><td>2020</td><td>1</td><td>1.5</td><td>-4</td></tr>\n<tr><td>2021</td><td>1.5</td><td>3</td><td>3</td></tr>\n<tr><td>2022</td><td>2</td><td>3.5</td><td>2</td></tr>\n<tr><td>2023</td><td>2.5</td><td>3</td><td>2.5</td></tr>\n</tbody>\n</table>\n\n<p>The graph shows that GDP Growth % experienced a significant drop in 2020, while Inflation Data % and Interest Rates % remained relatively stable with slight fluctuations over the years.</p></figcaption></figure>",
+    "page_num": 7
+  },
+  {
+    "text": "\n\n\nMacroeconomic factors such as interest\nrates, inflation, and GDP growth play a\npivotal role in shaping financial markets.\nThis section analyzes how these factors\nhave influenced stock, cryptocurrency,\nand commodity markets over recent\nyears, providing insights into the\ncomplex relationship between the\neconomy and financial market\nperformance.",
+    "page_num": 7
+  },
+  {
+    "text": "## Future Predictions and Trends\n\n\n<figure><figcaption>Relative Growth Trends for S&P 500, Bitcoin, and Oil Prices (2024 Indexed to 100)<br><p><strong>Prices (2024 Indexed to 100)</strong></p>\n<p>This bar chart compares the indexed prices of Oil, Bitcoin, and the S&amp;P 500 from 2024 to 2028, with 2024 set as the base year (indexed to 100). The chart shows the relative price changes over the years for each asset.</p>\n<table>\n<thead>\n<tr><th>Year</th><th>Oil</th><th>Bitcoin</th><th>S&amp;P 500</th></tr>\n</thead>\n<tbody>\n<tr><td>2024</td><td>100</td><td>100</td><td>100</td></tr>\n<tr><td>2025</td><td>105</td><td>110</td><td>108</td></tr>\n<tr><td>2026</td><td>110</td><td>115</td><td>112</td></tr>\n<tr><td>2027</td><td>115</td><td>120</td><td>116</td></tr>\n<tr><td>2028</td><td>120</td><td>125</td><td>120</td></tr>\n</tbody>\n</table></figcaption></figure>",
+    "page_num": 8
+  },
+  {
+    "text": "\n\n\nBased on historical data, current trends,\nand economic indicators, this section\npresents predictions for the future of\nfinancial markets. We explore potential\ntrajectories for stock indices,\ncryptocurrency values, and commodity\nprices, offering investors and analysts\nforesight into what the coming years\nmight hold.",
+    "page_num": 8
+  },
+  {
+    "text": "## Conclusions\n\n. In conclusion, this report has traversed the multifaceted landscape of\nfinancial markets, shedding light on the intricate patterns and\ninterdependencies that define their behavior. From the volatility of\ncryptocurrencies to the steadiness of commodities, each segment\ntells a part of the story of our global economy. As Contoso Financial\nAnalytics, we are committed to providing our clients with the most\ncomprehensive and nuanced analysis, empowering them to make\ninformed financial decisions in an ever-evolving market.",
+    "page_num": 9
+  }
+]