44 < meta charset ="utf-8 "/>
55 < title > MultiMarkdown v6 Development Notes</ title >
66 < meta name ="author " content ="Fletcher T. Penney "/>
7- < meta name ="date " content ="2018-02-05 "/>
7+ < meta name ="date " content ="2018-09-01 "/>
88 < meta name ="uuid " content ="dd2d8e76-dc2d-416d-9acd-5395d20871c2 "/>
99</ head >
1010< body >
1111
12- < h3 id ="introduction "> Introduction </ h3 >
12+ < h3 id ="introduction "> Introduction</ h3 >
1313
1414< p > This document includes some notes on the development of MultiMarkdown (< abbr title ="MultiMarkdown "> MMD</ abbr > ) v6. Most of it
1515will be interesting only to other developers or those needing to choose the
1616absolute “best” Markdown (< abbr title ="Markdown "> MD</ abbr > ) implementation for their needs – it is not required
1717reading to understand how the software works.</ p >
1818
19- < h4 id ="whyanewversion "> Why a New Version? </ h4 >
19+ < h4 id ="whyanewversion "> Why a New Version?</ h4 >
2020
2121< p > MultiMarkdown version 5 was released in November of 2015, but the codebase was
2222essentially the same as that of v4 – and that was released in beta in April
@@ -43,7 +43,7 @@ <h4 id="whyanewversion">Why a New Version? </h4>
4343into parser code. It worked well overall, but lacked some features I needed,
4444requiring a lot of workarounds.)</ p >
4545
46- < h3 id ="firstattempt "> First Attempt </ h3 >
46+ < h3 id ="firstattempt "> First Attempt</ h3 >
4747
4848< p > My first attempt started by hand-crafting a parser that scanned through the
4949document a line at a time, deciding what to do with each line as it found
@@ -137,7 +137,7 @@ <h3 id="firstattempt">First Attempt </h3>
137137< p > In the end, I scrapped this effort, but kept the lessons learned in the token
138138pairing algorithm.</ p >
139139
140- < h3 id ="secondattempt "> Second Attempt </ h3 >
140+ < h3 id ="secondattempt "> Second Attempt</ h3 >
141141
142142< p > I tried again this past Fall. This time, I approached the problem with lots
143143of reading. < em > Lots and lots</ em > of reading – tons of websites, computer science
@@ -195,13 +195,13 @@ <h3 id="secondattempt">Second Attempt </h3>
195195about how to create the grammar used. But so far, it has been able to handle
196196everything I have thrown at it.</ p >
197197
198- < h3 id ="optimization "> Optimization </ h3 >
198+ < h3 id ="optimization "> Optimization</ h3 >
199199
200200< p > One of my goals for < abbr title ="MultiMarkdown "> MMD</ abbr > 6 was performance. So I’ve paid attention to speed
201201along the way, and have tried to use a few tricks to keep things fast. Here
202202are some things I’ve learned along the way. In no particular order:</ p >
203203
204- < h4 id ="memoryallocation "> Memory Allocation </ h4 >
204+ < h4 id ="memoryallocation "> Memory Allocation</ h4 >
205205
206206< p > When parsing a long document, a < em > lot</ em > of token structures are created. Each
207207one requires a small bit of memory to be allocated. In aggregate, that time
@@ -230,7 +230,7 @@ <h4 id="memoryallocation">Memory Allocation </h4>
230230downside is remember to check for a single space character in a few instances
231231where it matters.</ p >
232232
233- < h4 id ="properinputbuffering "> Proper input buffering </ h4 >
233+ < h4 id ="properinputbuffering "> Proper input buffering</ h4 >
234234
235235< p > When I first began last spring, I was amazed to see how much time was being
236236spent by MultiMarkdown simply reading the input file. Then I discovered it
@@ -239,14 +239,14 @@ <h4 id="properinputbuffering">Proper input buffering </h4>
239239experimented with different buffer sizes, but they did not seem to make a
240240measurable difference.</ p >
241241
242- < h4 id ="outputbuffering "> Output Buffering </ h4 >
242+ < h4 id ="outputbuffering "> Output Buffering</ h4 >
243243
244244< p > I experimented with different approaches to creating the output after parsing.
245245I tried printing directly to < code > stdout</ code > , and even played with different
246246buffering settings. None of those seemed to work well, and all were slower
247247than using the < code > d_string</ code > approach (formerly called < code > GString</ code > in MMD 5).</ p >
248248
249- < h4 id ="fastsearches "> Fast Searches </ h4 >
249+ < h4 id ="fastsearches "> Fast Searches</ h4 >
250250
251251< p > After getting basic Markdown functionality complete, I discovered during
252252testing that the time required to parse a document grew exponentially as the
@@ -261,7 +261,7 @@ <h4 id="fastsearches">Fast Searches </h4>
261261This allowed me to get < abbr title ="MultiMarkdown "> MMD</ abbr > ’s performance back to O(n), taking roughly twice as
262262much time to process a document that is twice as long.</ p >
263263
264- < h4 id ="efficientutilityfunctions "> Efficient Utility Functions </ h4 >
264+ < h4 id ="efficientutilityfunctions "> Efficient Utility Functions</ h4 >
265265
266266< p > It is frequently necessary when parsing Markdown to check what sort of
267267character we are dealing with at a certain position – a letter, whitespace,
@@ -272,7 +272,7 @@ <h4 id="efficientutilityfunctions">Efficient Utility Functions </h4>
272272slightly differently under different circumstances. I also suspect it
273273improved performance, but don’t have the data to back it up.</ p >
274274
275- < h4 id ="testingwhilewriting "> Testing While Writing </ h4 >
275+ < h4 id ="testingwhilewriting "> Testing While Writing</ h4 >
276276
277277< p > I developed several chunks of code in parallel while creating < abbr title ="MultiMarkdown "> MMD</ abbr > 6. The vast
278278majority of it was developed largely in a < a href ="https://en.wikipedia.org/wiki/Test-driven_development "> test-driven development</ a > approach.
@@ -286,7 +286,7 @@ <h4 id="testingwhilewriting">Testing While Writing </h4>
286286broken. At this time, there are 29 text files in the test suite, and many
287287more to come.</ p >
288288
289- < h4 id ="otherlessons "> Other Lessons </ h4 >
289+ < h4 id ="otherlessons "> Other Lessons</ h4 >
290290
291291< p > Some things that didn’t do me any good….</ p >
292292
@@ -304,7 +304,7 @@ <h4 id="otherlessons">Other Lessons </h4>
304304the tools to measure whether I could have improved memory usage at all. Not
305305sure this would be worth the effort – much lower hanging fruit available.</ p >
306306
307- < h3 id ="performance "> Performance </ h3 >
307+ < h3 id ="performance "> Performance</ h3 >
308308
309309< p > Basic tests show that currently < abbr title ="MultiMarkdown "> MMD</ abbr > 6 takes about 20–25% longer the CommonMark
3103100.27.0 to process long files (e.g. 0.2 MB). However, it is around 5% < em > faster</ em >
@@ -345,9 +345,9 @@ <h3 id="performance">Performance </h3>
345345I’m sure there’s still a lot of room for further improvement to be made.
346346Suggestions welcome!</ p >
347347
348- < h3 id ="testing "> Testing </ h3 >
348+ < h3 id ="testing "> Testing</ h3 >
349349
350- < h4 id ="testsuite "> Test Suite </ h4 >
350+ < h4 id ="testsuite "> Test Suite</ h4 >
351351
352352< p > The development of < abbr title ="MultiMarkdown "> MMD</ abbr > v6 was heavily, but not absolutely, influenced by the
353353philosophy of test-driven development. While coding, I made use of test
@@ -360,7 +360,7 @@ <h4 id="testsuite">Test Suite </h4>
360360identified. This helps make proper integration testing of the entire
361361application with every release.</ p >
362362
363- < h4 id ="fuzztesting "> Fuzz Testing </ h4 >
363+ < h4 id ="fuzztesting "> Fuzz Testing</ h4 >
364364
365365< p > I was not familiar with the concept of
366366< a href ="https://en.wikipedia.org/wiki/Fuzzing "> Fuzz Testing</ a > until a user mentioned
@@ -388,18 +388,19 @@ <h4 id="fuzztesting">Fuzz Testing </h4>
388388sometimes identified. I have found some interesting edge cases this way.
389389Definitely a useful tool!</ p >
390390
391- < h4 id ="unittesting "> Unit Testing </ h4 >
391+ < h4 id ="unittesting "> Unit Testing</ h4 >
392392
393393< p > Some of the original development was done with unit testing in some other
394394tools I developed. This code formed the basis of a few parts of < abbr title ="MultiMarkdown "> MMD</ abbr > .
395395Otherwise, it was hard to see how to really create very good unit tests for
396396the development of < abbr title ="MultiMarkdown "> MMD</ abbr > . So there is really not much unit testing built into
397397the code or used during the development.</ p >
398398
399- < h3 id ="dependencieslibraries "> Dependencies/Libraries </ h3 >
399+ < h3 id ="dependencieslibraries "> Dependencies/Libraries</ h3 >
400400
401401< p > < abbr title ="MultiMarkdown "> MMD</ abbr > v6 has no external dependencies when compiling, aside from the standard
402- libraries for C development.</ p >
402+ libraries for C development (Except that it will use < code > libcurl</ code > if available in
403+ order to support downloading remote images/files for EPUB/FODT exporting.</ p >
403404
404405< p > < abbr title ="MultiMarkdown "> MMD</ abbr > can be compiled without any other tools beside the build system (cmake).</ p >
405406
@@ -429,9 +430,77 @@ <h3 id="dependencieslibraries">Dependencies/Libraries </h3>
429430TextBundle/TextPack, OpenDocument, etc.</ p > </ li >
430431</ ul >
431432
432- < h3 id ="changelog "> Changelog </ h3 >
433+ < h3 id ="changelog "> Changelog</ h3 >
433434
434435< ul >
436+ < li > < p > 2018–09–01 - v 6.4.0:</ p >
437+
438+ < ul >
439+ < li > ADDED: Add ODF Header metadata and fix issue with LaTeX Header metadata</ li >
440+ < li > ADDED: Add additional tests for special characters</ li >
441+ < li > ADDED: Add initial OPML export support (address #9)</ li >
442+ < li > ADDED: Add opml option to read for MultiMarkdown OPML files</ li >
443+ < li > Add missing Latex support files</ li >
444+ < li > Avoid potential error with stack_free</ li >
445+ < li > CHANGED: Remove unnecessary code</ li >
446+ < li > FIXED: Allow caption without trailing newline at end of document</ li >
447+ < li > FIXED: Escape square brackets, e.g. ‘{[foo]}’ (addresses #128 and #129)</ li >
448+ < li > FIXED: Fix escpaing of % character in LaTeX code spans and blocks</ li >
449+ < li > FIXED: Fix html comments inside code blocks (fixes #118)</ li >
450+ < li > FIXED: Fix issue where < code > ~</ code > is mistakenly interpreted as fence delimiter</ li >
451+ < li > FIXED: Fix issue with BOM and files > 4k</ li >
452+ < li > FIXED: Fix issue with dollar math delimiters inside code (fixes #134)</ li >
453+ < li > FIXED: Fix token length in OPML</ li >
454+ < li > FIXED: Improve OPML export; add OPML tests</ li >
455+ < li > FIXED: Normalize line spacing in CriticMarkup notes in LaTeX (fixes #120)</ li >
456+ < li > FIXED: Preserve tabs following leading hashes in code blocks</ li >
457+ < li > FIXED: Prevent potential null dereference</ li >
458+ < li > FIXED: Remove lock file</ li >
459+ < li > FIXED: Trim single remaining whitespace when exporting headers</ li >
460+ < li > FIXED: Trim trailing newlines in definition blocks</ li >
461+ < li > FIXED: Use Setext headers when necessary to convert from OPML to text (fixes #138)</ li >
462+ < li > FIXED: Use \ul instead of \underline when soul package is in use (fixes #121)</ li >
463+ < li > Merge pull request #132 from jlargentaye/develop</ li >
464+ < li > UPDATE: Clarify DevelopmentNotes re: libcurl</ li >
465+ < li > UPDATE: Clarify README re: libcurl</ li >
466+ < li > UPDATED: ‘\item{}’ no longer needed since square brackets escaped on their own</ li >
467+ < li > UPDATED: Add 6.3.1 release notes</ li >
468+ < li > UPDATED: Add allowfullscreen to list of boolean HTML attributes</ li >
469+ < li > UPDATED: Add more BibTeX test cases</ li >
470+ < li > UPDATED: Adjust metadata for test files</ li >
471+ < li > UPDATED: Allow '' to preserve line break in metadata. (Addresses #86)</ li >
472+ < li > UPDATED: Apply astyle</ li >
473+ < li > UPDATED: Fix whitespace with boolean HTML attributes</ li >
474+ < li > UPDATED: Ignore escaped space at end of code span</ li >
475+ < li > UPDATED: Test % escaping in URLs</ li >
476+ </ ul > </ li >
477+ < li > < p > 2018–03–28 - v 6.3.2:</ p >
478+
479+ < ul >
480+ < li > UPDATED: Update documentation</ li >
481+ </ ul > </ li >
482+ < li > < p > 2018–03–27 - v 6.3.1:</ p >
483+
484+ < ul >
485+ < li > ADDED: Add beamer support for code highlighting</ li >
486+ < li > ADDED: Add extra tests for variations on link types</ li >
487+ < li > ADDED: Add integrated memoir test</ li >
488+ < li > ADDED: Migrate manuscript and memoir(book) LaTeX formats to < abbr title ="MultiMarkdown "> MMD</ abbr > -6</ li >
489+ < li > FIXED: Checking for metadata was partially erasing token tree and leaving engine in invalid state</ li >
490+ < li > FIXED: Explicitly disable metadata in compatibility mode</ li >
491+ < li > FIXED: Fix Makefile for Windows 32-bit zip</ li >
492+ < li > FIXED: Fix detection of invalid request to copy substring of DString</ li >
493+ < li > FIXED: Fix issue with empty metadata lines</ li >
494+ < li > FIXED: Fix issue with raw source markers(fixes #115)</ li >
495+ < li > FIXED: Fix issue with trailing ATX hash without final newline</ li >
496+ < li > FIXED: Fix potential pointer overflow if line starts with ‘:::’</ li >
497+ < li > FIXED: Improve internal link creation in LaTeX</ li >
498+ < li > FIXED: Improve logic for disabling metadata</ li >
499+ < li > FIXED: Improve parsing of HTML 5 < video > tag</ li >
500+ < li > UPDATED: Add beginnings of Beamer test suite; add support for beamer/memoir and raw text blocks</ li >
501+ < li > UPDATED: Update quick start guide</ li >
502+ < li > UPDATED: Update remaining tests for LaTeX change</ li >
503+ </ ul > </ li >
435504< li > < p > 2018–02–05 - v 6.3.0:</ p >
436505
437506< ul >
0 commit comments