Conversation
…ation in Rust tests
- Introduced a new REPL feature with a corresponding test suite to validate its output. - Added `expectrl` as a dependency for handling REPL interactions. - Created a new `repl.feature` file to define the REPL behavior and expectations. - Implemented `repl.rs` to manage REPL sessions and validate output against expected results. - Updated `Cargo.toml` to include the new `expectrl` dependency and added a test configuration for the REPL.
- Changed expected output for the version command from "datu 0.2.4" to "datu 0.3.0-alpha". - Updated the first line validation in the head feature tests to specify "the first line of the output should be" for clarity. - Adjusted regex patterns in Rust tests to match the new output expectations for first line assertions.
- Introduced a new feature for converting data from Parquet to CSV format, with a scenario to validate the output file's existence and content. - Added a `Count` variant to the `PipelineStage` enum, enabling row counting across batches in the REPL. - Implemented the `exec_count` method to calculate and print the total number of rows. - Enhanced REPL tests to include scenarios for the new count functionality and conversion feature, ensuring robust validation of expected behaviors.
- Expanded the conversion feature in the REPL by adding multiple scenarios for converting data between various formats including Parquet, Avro, and ORC to CSV, JSON, YAML, and XLSX. - Implemented new validation steps in the tests to ensure output files exist, are valid in their respective formats, and contain expected content. - Enhanced the REPL test suite with additional assertions for line counts and content validation, improving overall test coverage and reliability.
- Enhanced the conversion feature tests by adding validation steps to ensure that output files generated from various conversions (Avro, Parquet, ORC, XLSX) are not only created but also valid according to their respective formats. - Implemented new test functions to check the validity of Avro, Parquet, ORC, and XLSX files, improving the robustness of the REPL test suite and ensuring accurate data handling across formats.
- Introduced new feature files for head, select, and tail operations in the REPL, covering various scenarios for reading data from Parquet, Avro, and ORC formats. - Each feature includes validation steps to ensure output files are created, exist in the correct format (CSV, JSON, YAML, Parquet, Avro), and contain expected content. - Enhanced test coverage for data manipulation operations, improving the robustness of the REPL test suite.
- Introduced a new section in the README detailing the interactive REPL (Read-Eval-Print Loop) feature of `datu`. - Documented the usage of key functions such as `read`, `write`, `select`, `head`, and `tail`, along with examples for composing data pipelines. - Enhanced user guidance on chaining functions to build complex data processing workflows.
- Changed code block formatting in the README from plain to text for better readability. - Enhanced feature tests for the `head` command in CLI and REPL by specifying that output files should have a defined number of records. - Added a new step in the REPL tests to validate the number of records in output files, improving test coverage and ensuring accurate data handling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
features/cli/andfeatures/repl/directories, with separate test harnesses (tests/features.rsfor CLI,tests/repl.rsfor REPL)head,tail, andselectoperations with pipeline composition (|>)count()command in the REPL pipeline, including unit tests and pipeline integration tests0.3.0-alphaexpectrldev dependency for spawning and interacting with the REPL process in teststhe first line should be→the first line of the output should be) for consistency across CLI and REPL testsKey changes
features/repl/{repl,conversion,head,select,tail}.featuretests/repl.rs(new, 340 lines)count()src/cli/repl.rsfeatures/cli/{cli,convert,count,head,schema,tail}.featuretests/features.rs(updated path + new step defs)Cargo.toml(expectrl),Cargo.lockCommits (11)
features/cli