MDX conversion pipeline and Astro sync workflow#62
Merged
Conversation
- Upgrade `markdown-link-check` to version `^3.13.6`. - Add new devDependencies for Prettier and its plugins to improve code formatting: - `prettier`: "^3.5.1". - `prettier-plugin-astro`: "^0.14.1". - `prettier-plugin-astro-organize-imports`: "^0.4.11". - `prettier-plugin-css-order`: "^2.1.2". - `prettier-plugin-jsdoc`: "^1.3.2". - `prettier-plugin-organize-attributes`: "^1.0.0". - `prettier-plugin-organize-imports`: "^4.1.0". - `prettier-plugin-tailwindcss`: "^0.6.11". - Add a new script, `"format-sync"`, to format MDX files using Prettier with the specified plugins. Enhance the MDX transformation process in `md2mdx.py` - After transforming Markdown to MDX, automatically format the resulting file by running the newly added `"format-sync"` script. - Add console output messages to indicate the start and completion of the formatting process for better user feedback during execution.
- Modify the way `PROJECT_ROOT` is determined in `md2mdx.py` to handle both direct execution and execution from the test-sync environment. - Add debug print statements to log the script location, project root, source file, and target file paths for easier troubleshooting. Update script path in `test-sync.sh` - Change directory context to ensure `md2mdx.py` is executed from the correct path within the test-sync environment. - Adjust commands to navigate directories before and after running the script.
- Eliminate the `transform_code_blocks()` function from `md2mdx.py` as it is no longer needed. - Update the `transform_markdown_to_mdx()` function to remove its invocation of `transform_code_blocks()`. - This change simplifies the code by removing unnecessary transformations of code blocks, which are now handled elsewhere. Enhance ASTRO block transformation - Rename `process_astro_blocks()` to `transform_astro_blocks()` for clarity and consistency. - Improve the handling of ASTRO comments by converting them into component tags with detailed processing: - Transform `CodeSwitcher` and `CodeExample` components, ensuring redundant titles are removed. - Handle components like `<Illustration>`, `<Diagram>`, and `<Picture>` effectively. - Add detailed logging within the transformation process for better traceability.
- Import the `glob` module to facilitate file searching. - Introduce `SOURCE_ROOT` and `TARGET_ROOT` to dynamically set source and target directories based on execution context. - Add a new function, `get_target_path()`, to transform source paths into target paths with required filename transformations. - Update the `process_files()` function to handle multiple markdown files by: - Searching for all `.md` files in specified directories. - Iterating over found files, transforming, and writing them to corresponding `.mdx` files in the target directory. - Maintaining existing functionality for formatting MDX files post-processing. Update `test-sync.sh` to include pages directory - Modify the copy command within the script to include the 'pages' directory when copying source files. This ensures that all relevant markdown files are available for processing during synchronization.
Introduce a new function `should_process_file()` to determine if a file should be processed based on an ignore list. This includes a case-insensitive check against `IGNORED_FILES`, which currently contains 'license.md'. The function prints a message when skipping an ignored file and returns `False`. Update the `process_files()` method to utilize this function, filtering out ignored files before processing.
Collaborator
Author
|
Not yet, please. I had to change this from the draft so that the action would kick in. There are still some merging bugs that I need to smash, but now I am in the diagram world. Soon! |
Refactoring the project by renaming all instances of `test-sync` to `sync`. This includes updates to the `.gitignore`, `package.json`, and various script files. - Updated `.gitignore` to ignore the new `sync` directory. - Modified scripts in `package.json` to reflect the new directory name. - Adjusted paths in the Python script `md2mdx.py`. - Refactored the shell script `test-sync.sh` to use the new directory name and improved its output messages for clarity.
Collaborator
Author
Member
|
Great stuff. I'd merge it asap to avoid conflicts, it shouldn't break anything :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR introduces an MDX conversion pipeline and automated sync workflow to transform Markdown documentation into MDX format for the Astro-based documentation site at https://github.com/apify/actor-whitepaper-web. The system handles complex transformations while preserving the original content structure and adding Astro-specific components.
Key changes
MDX Conversion Pipeline (
scripts/md2mdx.py)README.mdand other documentation filesGitHub Actions Workflow (
.github/workflows/sync-to-astro.yml)Development Scripts
test-sync.sh: Local testing environment for the sync processsetup.sh: Development environment setup with Python venvDocumentation Updates
README.mdand schema filesImplementation details
MDX conversion features
PicturecomponentCodeSwitcher,CodeExample, etc.)Workflow automation
Development tools
Testing
test-sync.shDocumentation
README.mdwith Astro component markersDependencies
Added to package.json:
markdown-toc:^1.2.0markdown-link-check:^3.13.6prettier:^3.5.1prettier-plugin-astro:^0.14.1prettier-plugin-astro-organize-imports:^0.4.11prettier-plugin-css-order:^2.1.2prettier-plugin-jsdoc:^1.3.2prettier-plugin-organize-attributes:^1.0.0prettier-plugin-organize-imports:^4.1.0prettier-plugin-tailwindcss:^0.6.11Next steps
Notes