Trouble converting docx to md

[eiger3970]

Hello, I'm having trouble removing TOC (table of contents) from docx files, when converting to md.
I have some nasty AI code, but I think manually solving this is the way.

#!/bin/bash

# This script converts a single Google Docs (.gdoc) or Microsoft Word (.docx) file
# from a Google Drive backup into a Markdown file suitable for Logseq.
# It handles directory structures, renames files for Logseq compatibility,
# extracts text, converts tables, and manages image attachments.
#
# This version includes:
# 1. Robust removal of Pandoc's non-standard '{.underline}' and similar formatting.
# 2. Conversion of Pandoc's non-standard '[[external_url]](external_url)' links to standard Markdown
#    using SED for improved compatibility.
# 3. Simplified output directory structure within Logseq.
# 4. Addition of 'title::' property for human-readable page names in Logseq UI.
#
# IMPORTANT NOTE REGARDING TABLE OF CONTENTS (TOC):
# This script is designed to programmatically remove any Pandoc-generated Table of Contents
# from the converted Markdown file. This ensures a cleaner page within Logseq.
# To get a functional, navigable TOC in Logseq's left sidebar or within the page,
# please use Logseq's native '/Table of Contents' block manually within the page,
# or rely on a Logseq plugin that dynamically generates a TOC from the page's headings.

# --- Configuration ---

# Your Logseq graph vault directory.
# This is where your Logseq 'journals', 'logseq', and 'pages' folders are.
USER_LOGSEQ_VAULT="/home/rapi5/Documents/Logseq Vault"

# Full path to the single Google Drive document file you want to convert.
# !!! IMPORTANT: Update this variable with the exact path to your file before running !!!
SOURCE_FILE="/home/rapi5/Downloads/google_drive_backup/Ken's folder 😊/House 🏠/Office/Network/Device 13 PC Raspberry Pi/Knowledge base Raspberry Pi.docx"

# Base directory for all converted Google Drive pages within your Logseq vault.
# This will be created inside USER_LOGSEQ_VAULT/pages/.
LOGSEQ_IMPORT_BASE_DIR="Google_Drive_Imports"

# Name of the directory where non-converted files will be moved within the Logseq vault.
# This directory will be created inside your Logseq vault.
NON_CONVERTED_DIR_NAME="Non_Converted_Google_Docs"

# Name of the directory for Logseq attachments within your Logseq vault.
# This directory will be created inside your Logseq vault.
LOGSEQ_ATTACHMENTS_DIR_NAME="assets" # Logseq often uses 'assets' by default for attachments

# Name of the directory for Logseq pages within your Logseq vault.
# This directory will be created inside your Logseq vault.
LOGSEQ_PAGES_DIR_NAME="pages"

# Temporary directory for processing files.
TEMP_DIR="/tmp/gdrive_conversion_single_$$"

# --- End Configuration ---

# Ensure required commands are available
command -v pandoc >/dev/null 2>&1 || { echo >&2 "Error: pandoc is not installed. Please install it (e.g., sudo apt install pandoc)."; exit 1; }
command -v unzip >/dev/null 2>&1 || { echo >&2 "Error: unzip is not installed. Please install it (e.g., sudo apt install unzip)."; exit 1; }
command -v jq >/dev/null 2>&1 || { echo >&2 "Error: jq is not installed. Please install it (e.g., sudo apt install jq)."; exit 1; }


echo "Starting Single Google Drive to Logseq Conversion..."
echo "Source File: $SOURCE_FILE"
echo "Logseq Vault: $USER_LOGSEQ_VAULT"
echo "Logseq Import Base Directory: $USER_LOGSEQ_VAULT/$LOGSEQ_PAGES_DIR_NAME/$LOGSEQ_IMPORT_BASE_DIR"
echo "Non-Converted Files Destination: $USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME"
echo "Temporary Directory: $TEMP_DIR"

# Create necessary directories
mkdir -p "$TEMP_DIR"
mkdir -p "$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME"
mkdir -p "$USER_LOGSEQ_VAULT/$LOGSEQ_PAGES_DIR_NAME/$LOGSEQ_IMPORT_BASE_DIR"
mkdir -p "$USER_LOGSEQ_VAULT/$LOGSEQ_ATTACHMENTS_DIR_NAME" # This is the root assets folder


# Function to sanitize filenames for Logseq compatibility (for the .md file name itself)
# Converts spaces to hyphens, removes problematic characters, ensures lowercase.
sanitize_filename() {
    local filename="$1"
    # Convert spaces to hyphens, remove non-alphanumeric (except hyphens and periods), convert to lowercase
    # Keep directory separators as they are
    echo "$filename" | sed \
        -e 's/[[:space:]]\+/\-/g' \
        -e 's/[^a-zA-Z0-9_\-\.\/]//g' \
        -e 's/\.\.\//\//g' \
        -e 's/-\+/-/g' \
        -e 's/^-//' \
        -e 's/-$//' \
        -e 's/\/\//\//g' \
        | tr '[:upper:]' '[:lower:]'
}

# Function to escape a string for use in sed replacement patterns
# This correctly escapes all characters that might be interpreted by sed
escape_sed_replacement() {
    printf %s "$1" | sed -e 's/[\/&]/\\&/g'
}


# Function to convert DOCX to Markdown for Logseq
convert_docx_to_logseq_md() {
    local docx_file="$1"
    local original_base_name=$(basename "$docx_file" .docx)
    local sanitized_md_filename=$(sanitize_filename "$original_base_name") # This is for the .md filename on disk

    echo "Processing DOCX: $docx_file"

    # Define the output Markdown file path in the simplified structure
    local output_md_file="$USER_LOGSEQ_VAULT/$LOGSEQ_PAGES_DIR_NAME/$LOGSEQ_IMPORT_BASE_DIR/${sanitized_md_filename}.md"

    # Define the assets directory for this specific document's images
    # This will be a subdirectory within the main Logseq assets folder
    local doc_assets_dir="$USER_LOGSEQ_VAULT/$LOGSEQ_ATTACHMENTS_DIR_NAME/${sanitized_md_filename}_assets"
    mkdir -p "$doc_assets_dir"

    # Pandoc command: No --toc or --toc-depth.
    pandoc "$docx_file" -f docx -t markdown+hard_line_breaks --extract-media="$doc_assets_dir" --wrap=none --markdown-headings=atx -o "$output_md_file"

    if [ $? -eq 0 ]; then
        echo "Successfully converted '$docx_file' to '$output_md_file'"

        # --- Post-processing for Logseq compatibility ---

        # 1. Insert title property at the top for Logseq's display name
        # This will make Logseq show "Knowledge base Raspberry Pi" instead of the full path.
        echo "title:: $original_base_name" | cat - "$output_md_file" > "${output_md_file}.tmp" && mv "${output_md_file}.tmp" "$output_md_file"

        # 2. Correct image paths for Logseq's /assets/ structure
        # Logseq expects image links in the format: ![](/assets/image.png) or ![](/assets/subfolder/image.png)
        # Pandoc extracts media to a folder named after the document (e.g., knowledgebaseraspberrypi_assets).
        # So the path from Logseq vault root should be /assets/knowledgebaseraspberrypi_assets/image.png
        local relative_assets_path_from_vault_root="${LOGSEQ_ATTACHMENTS_DIR_NAME}/${sanitized_md_filename}_assets"
        local escaped_relative_assets_path_for_sed=$(escape_sed_replacement "$relative_assets_path_from_vault_root")

        # Replace pandoc's default 'media/' prefix with the correct Logseq assets path
        sed -i -E "s#!\\[(.*?)\\]\\(media/(.*?)\\)#![\\1](/${escaped_relative_assets_path_for_sed}/\\2)#g" "$output_md_file"
        # Catch-all for other relative image paths (e.g., if pandoc doesn't use 'media/')
        sed -i -E "s#!\\[(.*?)\\]\\(([^/)]+?\\.(png|jpg|jpeg|gif|svg|webp))\\)#![\\1](/${escaped_relative_assets_path_for_sed}/\\2)#g" "$output_md_file"

        # 3. Remove Pandoc's non-standard formatting spans like '{.underline}' or '{.something}'
        sed -i -E 's/\{\.[^}]+\}//g' "$output_md_file"

        # 4. Convert Pandoc's non-standard [[URL]](URL) or [[Link Text]]{.attr}(URL) to standard [Link Text](URL) using SED.
        # This regex captures the link text (group 1) and the URL (group 2).
        # It handles `[[Link Text]](url)` cases.
        sed -i -E 's/\[\[([^\]]+)\]\]\((https?:\/\/[^)]+)\)/[\1](\2)/g' "$output_md_file"
        # This regex handles `[[URL]](URL)` where link text and URL are the same.
        sed -i -E 's/\[\[(https?:\/\/[^\]]+)\]\]\((https?:\/\/[^)]+)\)/[\1](\2)/g' "$output_md_file"

        # 5. Robustly remove the Pandoc-generated Table of Contents from the main page using SED.
        #    This now uses a range delete based on the specific TOC pattern identified.
        #    START Pattern: Looks for a line starting with '##' followed by non-breaking spaces
        #    END Pattern: Looks for any line starting with a single hash '#' (meaning a new main heading)
        #    If no single hash heading, it will delete until a blank line or end of file.
        sed -i -E '/^##\xC2\xA0*$/,/^#/!b; /^[[:space:]]*$/{d; :a; n; /^[[:space:]]*$/!ba; d}' "$output_md_file"
        # The above sed command attempts to remove from the TOC heading until the next actual heading.
        # If the TOC is immediately followed by content without a heading, this might need refinement.
        # Let's try a simpler one based on the sample: deleting from "##  " until the first non-blockquote, non-link line.
        # Simpler approach: delete from the "##  " line, and any subsequent line starting with '>' that contains an internal link.
        sed -i -E '/^##\xC2\xA0*$/d' "$output_md_file" # Delete the TOC title line
        sed -i -E '/^>[\s\xC2\xA0]*\[[^\]]+\]\(#.*\)/d' "$output_md_file" # Delete blockquote TOC entries

        echo "Post-processing complete for '$output_md_file'."
    else
        echo "Error converting '$docx_file'. Pandoc exited with status $?."
        mv "$docx_file" "$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME/"
        echo "Moved '$docx_file' to '$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME/'."
    fi
}

# Function to convert GDOC to Markdown for Logseq
convert_gdoc_to_logseq_md() {
    local gdoc_file="$1"
    local original_base_name=$(basename "$gdoc_file" .gdoc)
    local sanitized_md_filename=$(sanitize_filename "$original_base_name")

    echo "Processing GDOC: $gdoc_file"

    local html_file="${gdoc_file%.gdoc}.html"

    if [ ! -f "$html_file" ]; then
        echo "Warning: Corresponding HTML file not found for GDOC: $gdoc_file. Skipping."
        mv "$gdoc_file" "$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME/"
        echo "Moved '$gdoc_file' and '$html_file' to '$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME/'."
        return
    fi

    local output_md_file="$USER_LOGSEQ_VAULT/$LOGSEQ_PAGES_DIR_NAME/$LOGSEQ_IMPORT_BASE_DIR/${sanitized_md_filename}.md"
    local doc_assets_dir="$USER_LOGSEQ_VAULT/$LOGSEQ_ATTACHMENTS_DIR_NAME/${sanitized_md_filename}_assets"
    mkdir -p "$doc_assets_dir"

    # Pandoc command: No --toc or --toc-depth.
    pandoc "$html_file" -f html -t markdown+hard_line_breaks --extract-media="$doc_assets_dir" --wrap=none --markdown-headings=atx -o "$output_md_file"

    if [ $? -eq 0 ]; then
        echo "Successfully converted '$html_file' (from GDOC) to '$output_md_file'"

        # 1. Insert title property at the top for Logseq's display name
        echo "title:: $original_base_name" | cat - "$output_md_file" > "${output_md_file}.tmp" && mv "${output_md_file}.tmp" "$output_md_file"

        # 2. Correct image paths for Logseq's /assets/ structure
        local relative_assets_path_from_vault_root="${LOGSEQ_ATTACHMENTS_DIR_NAME}/${sanitized_md_filename}_assets"
        local escaped_relative_assets_path_for_sed=$(escape_sed_replacement "$relative_assets_path_from_vault_root")

        sed -i -E "s#!\\[(.*?)\\]\\(media/(.*?)\\)#![\\1](/${escaped_relative_assets_path_for_sed}/\\2)#g" "$output_md_file"
        sed -i -E "s#!\\[(.*?)\\]\\(([^/)]+?\\.(png|jpg|jpeg|gif|svg|webp))\\)#![\\1](/${escaped_relative_assets_path_for_sed}/\\2)#g" "$output_md_file"

        # 3. Remove Pandoc's non-standard formatting spans like '{.underline}' or '{.something}'
        sed -i -E 's/\{\.[^}]+\}//g' "$output_md_file"

        # 4. Convert Pandoc's non-standard [[URL]](URL) or [[Link Text]]{.attr}(URL) to standard [Link Text](URL) using SED.
        sed -i -E 's/\[\[([^\]]+)\]\]\((https?:\/\/[^)]+)\)/[\1](\2)/g' "$output_md_file"
        sed -i -E 's/\[\[(https?:\/\/[^\]]+)\]\]\((https?:\/\/[^)]+)\)/[\1](\2)/g' "$output_md_file"

        # 5. Robustly remove the Pandoc-generated Table of Contents from the main page using SED.
        sed -i -E '/^##\xC2\xA0*$/d' "$output_md_file" # Delete the TOC title line
        sed -i -E '/^>[\s\xC2\xA0]*\[[^\]]+\]\(#.*\)/d' "$output_md_file" # Delete blockquote TOC entries

        echo "Post-processing complete for '$output_md_file'."
    else
        echo "Error converting '$html_file' (from GDOC). Pandoc exited with status $?."
        mv "$gdoc_file" "$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME/"
        mv "$html_file" "$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME/" # Move the HTML too
        echo "Moved '$gdoc_file' and '$html_file' to '$USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME/'."
    fi
}

# --- Main execution for a single file ---
file_path="$SOURCE_FILE"

if [[ "$file_path" == *.docx ]]; then
    convert_docx_to_logseq_md "$file_path"
elif [[ "$file_path" == *.gdoc ]]; then
    convert_gdoc_to_logseq_md "$file_path"
else
    echo "Error: The specified SOURCE_FILE is not a .docx or .gdoc file: $SOURCE_FILE"
    exit 1
fi

echo "Single file conversion process complete."
echo "Non-converted files (if any) are in: $USER_LOGSEQ_VAULT/$NON_CONVERTED_DIR_NAME"
echo "Converted Markdown file and assets are in: $USER_LOGSEQ_VAULT/$LOGSEQ_PAGES_DIR_NAME/$LOGSEQ_IMPORT_BASE_DIR/ and $USER_LOGSEQ_VAULT/$LOGSEQ_ATTACHMENTS_DIR_NAME/"

# Clean up temporary directory
rm -rf "$TEMP_DIR"

[denismaier]

Maybe it's only me, but I suppose it will be much easier to get some help with this if you can break this down into smaller and more specific questions. That script is already rather long. (Also, the title of the discussion is not exactly helpful.)