HTML ('headings' as classes) to MD?

[dwids]

There's a great old music info website. Each band/artist has a page and I'd like to grab some as Markdown...and later merge into an ePub. But it uses 'non-standard' HTML for the "Headings". So, what would be the H1 (the band name) is:

<p class="head">THE ANGELS<br>

And the History etc subheadings are:

<p class="sub">History </p>
...
<p class="sub">Discography </p>

...which would be H2s.

Can i use Pandoc, with some coding, to convert these to Markdown # THE ANGELS and ## History ?

If it helps, I know enough about Regex to find/replace them:

f: <p class="head">(.*)<br>
r: # $1

f: <p class="sub">(.*)\n?</p>
r: ## $1

(Some of the 'sub' items have a LR inside the paragraph.)

I'm on Windows and am using a PowerShell terminal.

Thanks

[dude-at-RA]

This would be pretty easy to do with a pandoc lua filter:

Disregard the lua below - it doesn't work because Para doesn't have classes.

non working filter

function Para (elem)
  for i, c in pairs(elem.classes) do    
    if c == "head" then
      return pandoc.Header(1, elem.content) 
    if c == "sub" then
      return pandoc.Header(2, elem.content)
  end
  return elem
end

[dwids]

Thanks for both replies, much appreciated. I had gone to one of the AI chatbots and had a back and forth on LUA for Pandoc generation - from simple code to more complex - but nothing worked. Given the same system got something binary wrong with a simple PowerShell cmd, I'm not surprised. At the end, FYI, is the final version of its script.

I actually do know BeautifulSoup, so that code makes sense to me. I too had thought of the chain approach (preprocessing) and was going to look for a Browser Plugin to do the class-to-heading swap. I'll see how both goes.

io.stderr:write("headings.lua filter loaded\n")

-- Function to process each paragraph element
function Para (para)
    -- Check if the paragraph has the class "head"
    if para.class and para.class:includes("head") then
        -- Initialize an empty list to store the modified blocks
        local blocks = {}
        
        -- Iterate through each element in the paragraph
        for _, element in ipairs(para.content) do
            -- Check if the element is a Str (text element)
            if element.t == pandoc.Str then
                -- Check if the text contains the band name (assuming the band name is the first text element)
                local band_name = element.text
                -- Create a new Header element with the band name
                local header = pandoc.Header(1, band_name)
                -- Add the header to the blocks list
                table.insert(blocks, header)
                -- Remove the original text element from the paragraph
                element = nil
            else
                -- Add other elements (like <br>, <span>) to the blocks list
                table.insert(blocks, element)
            end
        end
        
        -- If we found a header, return the modified blocks
        if #blocks > 0 then
            return blocks
        else
            -- Otherwise, return the original paragraph
            return para
        end
    else
        -- Return the original paragraph if it doesn't match the criteria
        return para
    end
end

[fiapps]

No Lua filter can work because the class is already gone by the time it runs. The pandoc AST does not allow Para to have classes. Look at what you get if you convert to native format. That's the AST the filter sees.

% echo '<p class="head">A heading</p>' | pandoc -f html -t native
[ Para [ Str "A" , Space , Str "heading" ] ]

[dwids]

I can see that when I run it here on Windows PowerShell too. Thanks.

'<p class="head">A heading</p>' | pandoc -f html -t native
# [ Para [ Str "A" , Space , Str "heading" ] ]

[dude-at-RA]

It would be that easy if Para had classes,

Ah, dang, I totally forgot about that! And sadly, I've been stung by that before!

Is there a reason why para tags don't bring their attributes with them?

[fiapps]

I imagine the AST was designed that way because Markdown (at least originally) didn't have a way of representing attributes on a paragraph. I thought there was an issue that suggested allowing all AST elements to have attributes (id, classes, and attribute list), a suggestion that I believe was resisted because it would be too much of a change to the AST. For now, the only way to apply classes or attributes (like custom-style) to a paragraph is to wrap it in a div that has the class. The HTML and ePub reader could do that, but don't, unless there's some extension I'm unaware of. I find this particularly problematic for ePub files, where the formatting is often done with classes on paragraphs. This means I can't use pandoc's ePub reader: instead, I read a Python script that opens the ePub as a Zip file, parses and transforms the HTML, and then outputs it in a form that pandoc can import without information loss (e.g., transform <p class="para1">…</p> to <div class="para1"><p>…</p></div>).