Support for converting HTML results into text [GH-1]

[kenorb]

Fixes GH-1.

Summary by Sourcery

Adds the ability to convert MetaTrader HTML reports into CSV format for orders, deals, and header information, and also supports converting optimization reports.

New Features:

• Adds support for converting MetaTrader HTML reports into CSV format.
• Adds support for converting optimization reports.
• Introduces a new lookup plugin to parse MetaTrader reports and extract data.

Tests:

• Adds testing for the new lookup plugin.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request introduces a new lookup plugin, parse_mt_report.py, which parses MetaTrader 5 strategy tester reports (in HTML format) and converts specific sections ('orders', 'deals', 'header', and 'optimization') into CSV format. The changes also include new tasks in tasks/main.yml that utilize this plugin to generate CSV files from the MT5 reports.

Sequence diagram for parsing MT5 report and converting to CSV

sequenceDiagram
    participant Ansible
    participant parse_mt_report.py
    participant MT5 Report (HTML)
    participant CSV File

    Ansible->>parse_mt_report.py: lookup('parse_mt_report', report_path, report_type, include_titles)
    activate parse_mt_report.py
    parse_mt_report.py->>MT5 Report (HTML): Read HTML content
    MT5 Report (HTML)-->>parse_mt_report.py: HTML content
    alt report_type is 'header'
        parse_mt_report.py->>parse_mt_report.py: Extract header table
        parse_mt_report.py->>parse_mt_report.py: Convert header table to JSON
        parse_mt_report.py-->>Ansible: JSON string
    else report_type is 'orders' or 'deals'
        parse_mt_report.py->>parse_mt_report.py: Extract orders/deals table
        parse_mt_report.py->>parse_mt_report.py: Convert table to CSV
        parse_mt_report.py-->>Ansible: CSV string
    else report_type is 'opt'
        parse_mt_report.py->>parse_mt_report.py: Parse XML to extract optimization data
        parse_mt_report.py->>parse_mt_report.py: Convert optimization data to CSV
        parse_mt_report.py-->>Ansible: CSV string
    end
    deactivate parse_mt_report.py
    Ansible->>CSV File: Write CSV string to file

Loading

Class diagram for parse_mt_report.py

classDiagram
    class LookupModule {
        +run(terms, variables, **kwargs)
    }
    class main {
        +main(input_file_path, output_file_path, include_titles, type, return_string)
    }
    class write_to_csv {
        +write_to_csv(data, output_file, include_titles, type, return_string)
    }
    class write_to_json {
        +write_to_json(html_content, output_file, type, return_string)
    }
    class write_opt {
        +write_opt(input_file, output_file, include_titles, return_string)
    }
    class extract_header_table {
        +extract_header_table(html_content)
    }
    class extract_orders_table {
        +extract_orders_table(html_content)
    }
    class extract_deals_table {
        +extract_deals_table(html_content)
    }
    class parse_period {
        +parse_period(value)
    }
    class parse_val_prc {
        +parse_val_prc(value)
    }
    class parse_prc_val {
        +parse_prc_val(value)
    }
    class parse_val_of {
        +parse_val_of(value)
    }
    class parse_val_diff {
        +parse_val_diff(value)
    }
    class parse_time {
        +parse_time(value)
    }
    class convert_value {
        +convert_value(value)
    }

    LookupModule -- main : calls
    main -- write_to_csv : calls
    main -- write_to_json : calls
    main -- write_opt : calls
    write_to_csv -- extract_orders_table : calls
    write_to_csv -- extract_deals_table : calls
    write_to_json -- extract_header_table : calls
    extract_header_table -- parse_period : calls
    extract_header_table -- parse_val_prc : calls
    extract_header_table -- parse_prc_val : calls
    extract_header_table -- parse_val_of : calls
    extract_header_table -- parse_val_diff : calls
    extract_header_table -- parse_time : calls
    extract_header_table -- convert_value : calls

Loading

File-Level Changes

Change	Details	Files
Implements a new lookup plugin to parse MetaTrader 5 strategy tester reports.	Adds a new lookup plugin named parse_mt_report.py. The plugin uses BeautifulSoup4 to parse HTML reports. The plugin extracts data from the HTML report and converts it into CSV format. The plugin supports extracting 'orders', 'deals', and 'header' data. The plugin includes helper functions to parse specific data formats within the HTML report. The plugin can extract data from optimization reports in XML format. The plugin returns a CSV formatted string.	lookup_plugins/parse_mt_report.py
Adds tasks to convert MetaTrader 5 report data into CSV format using the new lookup plugin.	Adds tasks to convert 'orders', 'deals', 'header', and 'optimization' sections of the MT5 report to CSV. Each task uses the copy module to write the CSV data to a file. The lookup function calls the parse_mt_report plugin to generate the CSV content. The destination file names are based on the original report name with suffixes indicating the data type (e.g., -Orders.csv).	tasks/main.yml

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!
• Generate a plan of action for an issue: Comment @sourcery-ai plan on
  an issue to generate a plan of action for it.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[gemini-code-assist]

Hello @kenorb, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

This pull request introduces a new lookup plugin, parse_mt_report.py, to convert HTML reports generated by MetaTrader (MT4/MT5) into structured data formats like CSV and JSON. It also adds tasks to the main.yml file to utilize this plugin and generate CSV files from the MT5 reports for Orders, Deals, Header, and Optimization data. The primary goal is to automate the extraction of data from these reports for further analysis or integration.

Highlights

• New Lookup Plugin: The parse_mt_report.py plugin is introduced to parse MetaTrader HTML reports. It uses BeautifulSoup for HTML parsing and regular expressions to extract specific data points.
• Data Extraction: The plugin extracts data from different sections of the HTML report, including header information, order details, deal records, and optimization results.
• Output Formatting: The extracted data can be formatted into CSV or JSON. The main.yml file uses the plugin to generate CSV files for different report sections.
• Ansible Integration: The plugin is designed as an Ansible lookup module, allowing it to be used directly within Ansible playbooks to process report files.

Changelog

Click here to see the changelog

• lookup_plugins/parse_mt_report.py
  • Initial commit of the parse_mt_report.py lookup plugin.
  • Implements functions to parse different sections of the HTML report: extract_header_table, extract_orders_table, extract_deals_table.
  • Includes functions to write data to CSV (write_to_csv) and JSON (write_to_json).
  • Adds helper functions for data conversion and parsing specific value formats (e.g., parse_period, parse_val_prc).
  • Defines the LookupModule class to integrate the plugin with Ansible.
  • Adds main function to orchestrate the parsing and writing processes based on the specified type (orders, deals, header, opt).
• tasks/main.yml
  • Adds new tasks to convert MT5 reports to CSV format using the parse_mt_report lookup plugin.
  • Creates separate tasks for converting Orders, Deals, Header, and Optimization data to CSV files.
  • Uses the copy module to write the CSV content to files with specific names based on the report type.
  • Specifies become: true and become_user: root for these tasks.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Did you know?

The term 'plugin' comes from the world of hardware, where physical components could be 'plugged in' to extend a system's capabilities.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request introduces a new lookup plugin to parse MetaTrader reports and convert them into CSV format. It also includes changes to the default timeout value and adds tasks to generate CSV reports for orders, deals, headers, and optimization results. Overall, the changes seem well-structured and address the intended functionality.

Merge Readiness

The changes introduced by this pull request seem well-structured and address the intended functionality. I am unable to directly approve the pull request, and recommend that others review and approve this code before merging. Given the absence of CRITICAL or HIGH severity issues, the pull request appears to be in good shape for merging.

[sourcery-ai]

Hey @kenorb - I've reviewed your changes - here's some feedback:

Overall Comments:

• Consider using a loop in your tasks file to avoid repeating the same task multiple times.
• The lookup plugin has a lot of code - consider breaking it up into smaller functions or classes.

Here's what I looked at during the review

• 🟡 General issues: 3 issues found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}