diff --git a/data-platform/autonomous-database/autonomous-json/MongoDBReplay/README.md b/data-platform/autonomous-database/autonomous-json/MongoDBReplay/README.md index 0c600ce1c..31cfc00d4 100644 --- a/data-platform/autonomous-database/autonomous-json/MongoDBReplay/README.md +++ b/data-platform/autonomous-database/autonomous-json/MongoDBReplay/README.md @@ -1,150 +1,172 @@ -## MongoDB Replay +**MongoDB Replay** -Purpose of this tool is to provide support with assessment of a MongoDB application with Oracle MongoDB API. +The purpose of MongoDB Replay is to provide support with assessment of a +MongoDB application with Oracle MongoDB API. You can extract statements +from the MongoDB log file and replay it in a test environment with +MongoDB Replay. -The repository contains source code of two artifacts -1. MDBExtractor.jar - - this is the tool, which extracts stream of commands from a log file generated by a MongoDB instance. +**Disclaimer** -2. MDBApplier.jar - - this is the tool, which executes commands extracted during the extraction process against an Oracle MongoDB API instance +The use of the scripts is at your sole risk. All information is provided +\"as -is\", without any warranty of its accuracy, completeness, fitness +for a particular purpose. -## License +MongoDB Replay has been developed to replay the MongoDB statements in +test environments only! Do not use it against production databases. +Oracle does not take any responsibility of possible +problems/issues/damages/data corruptions caused by using MongoDB Replay. -Copyright (c) 2025 Oracle and/or its affiliates. +**Useful Links** -Licensed under the Universal Permissive License (UPL), Version 1.0. +- [Tools and Drivers for Oracle Database API for + MongoDB](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-9B704340-5E1D-4CA9-8944-6296A69221B0) -See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details. +- [Oracle Database Requirements and + Configuration](https://docs.oracle.com/en/database/oracle/oracle-rest-data-services/24.4/ordig/enabling-and-configuring-oracle-database-api-mongodb.html#GUID-F79F2BB4-BB6E-4890-AFD7-E21DC4462835) -## WARNING -#### This tool has been developed for test purposes only! -#### Do not use this tool against production databases! It may drive to serious damages of a database or complete data loss! Oracle does not guarantee and does not take any responsibility of possible problems/issues/damages/data corruptions caused by using this tool. +**Artifacts** +The repository contains source code of two artifacts: -## Usage -### Generic requirements - - MR_CONFIG_DIR variable has to be set to a directory, where configuration and log files are/will be stored - - JDK 21 (it can be Oracle or OpenJDK) - - Oracle JDK can be downloaded from the following link: https://www.oracle.com/java/technologies/downloads/#java21 - - OpenJDK is usually available in a Linux packates repository, but also can be downloaded and installed independly from the following link: https://openjdk.org/ + ----------------------------------------------------------------------- + Artifact Description + ----------------------- ----------------------------------------------- + MDBExtractor.jar extracts a stream of commands from a log file + generated by a MongoDB instance. + + MDBApplier.jar executes commands extracted during the + extraction process against an Oracle MongoDB + API instance + ----------------------------------------------------------------------- + +**Requirements** + +- A variable named MR_CONFIG_DIR has to be set to a directory, where + configuration and log files are stored. + +- JDK version 21 either from Oracle be downloaded from + or\ + OpenJDK available in a Linux packages repository or downloaded and + installed + from [[https://openjdk.org/]{.underline}](https://openjdk.org/) ### MDBExtractor - requirements: - - Source MongoDB instance up and running - - Profiling at the MongoDB instance - - to extract data from traditional Mongod log file, there is need to enable logging so called slow operations at the Mongod instance level. It can be done by executing the following command: - - db.setProfilingLevel(0, -1) - - to extract data from db.system.profile collection or its dump there is need to enable traditional profiling, by executing the following command: - - db.setProfilingLevel(2) - - - The following configuration file $MR_CONFIG_DIR/MDBExtractConfig.json needs to be existing with the following parameters - - - OUTPUT_DIR : optional; points to directory where output files will be created; - if this parameter is not set, then MDBExtractor will print out results into standard output - - - INPUT_FILE : optional; points to an input log file generated by a MongoDB Instance with profiling enabled - if this parameter is not set, then MDBExtractor will read data from standard input - - - INPUT_FILE_FORMAT : optional; provides information about from which file the data will be read. It accepts two values - MONGO_LOG : in that case Extractor will be reading data from traditional Mongod log file. Profiling needs to be set to (0,-1) - MONGO_PROFILE : in that case Extractor will be reading data from dump of db.system.profile collection. Profiling needs to be set to (2) - - - INPUT_CONNECT_STRING : optional; if set then it points to a MongoDB instance, from which input data will be read directly. Profiling needs to be set to (2). When this parameter is set, then INPUT_FILE and INPUT_FILE_FORMAT cannot be set. - - - COMMANDS_LOGGING : optional; default value "true", if set to "true" then every command found by the extractor will be - additional logged with console.log(...). Useful, when output will be processed by mongosh tool, which - has limited capabilities related to error reporting. When set to "false" only commands will be reported. - - - INCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be extracted; - all commands not listed here will not be printed out into output; this parameter CANNOT be used - simultanously with EXCLUDE_COMMANDS - - - EXCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be ignored during extraction process; - all commands listed here will not be printed out into output; this parameter CANNOT be used - simultanously with INCLUDE_COMMANDS - if none of above parameters is set, then all commands will be traced - - - INCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will be traced. - all databases not listed here will not be traced. - This parameter cannot be used simultaneously with EXCLUDE_DATABASES - - - EXCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will not be traced. - all databases not listed here will be traced. - This parameter cannot be used simultaneously with INCLUDE_DATABASES - if none of above parameters is set, then all databases will be traced - - - EXECUTION_PLAN_TRACING : if set to a non-zero value, then output will contain explain() commands instead of runCommand; - useful, when there is need to check/compare execution plans between a source and a target system - - - OUTPUT_MODE : can be set to JSON, which result in that the output data will be formated into sequence of JSON documents - or to SCRIPT, wich will result in that the output data will be formated into NodeJS script, which can be - consumed by mongosh - - LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to - standard diagnostic output - - - LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0 - 0 means that only summary of initialization and processing will be logged - 1 means that additionally to summaries also errors will be logged - 2 means that additionally to summaries and errors all commands will be logged - - Configuration File example: - ``` - { - "OUTPUT_DIR" : "/opt/mongo_tests/output", - "INPUT_FILE" : "/opt/mongo_tests/mongod01.log", - "COMMANDS_LOGGING" : "false", - "INCLUDE_DATABASES" : ["oradev","test"], - "EXCLUDE_COMMANDS" : ["insert"], - "EXECUTION_PLAN_TRACING" : 0, - "OUTPUT_MODE" : "JSON" - } - examples of using the tool - 1. ``` - java -jar ./MDBExtract.jar - ``` - this example reads data from a file pointed by INPUT_FILE parameter and generates output into set of files created in - OUTPUT_DIR (every traced database will use a separate output file) - - 2. ``` - tail -n 10000 -f ./mongod.log|java -jar ./MDBExtract.jar - ``` - this example assumes, that INPUT_FILE parameter IS NOT SET. Instead of the tool reads data from its standard input - -### MDBApplier - requirements - - target Oracle Database (19c or later) with [Oracle API for MongoDB](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-1CF44843-6294-45F0-8065-B9E8034D6CB1) enabled - - subset of the data representing all collections which will be traced - - the following configuration file $MR_CONFIG_DIR/MDBApplierConfig.json needs to be existing with the following parameters - - INPUT_FILE : optional; points to an input log file generated by MDBExtractor - if this parameter is not set, then MDBApplier will read data from standard input - - DB_NAME : optional; allows for providing target database name - - CONNECT_STRING : mandatoryl; connect string to a target MongoDB or Oracle API for MongoDB instance - - LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to - standard diagnostic output - - - LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0 - 0 means that only summary of initialization and processing will be logged - 1 means that additionally to summaries also errors will be logged - 2 means that additionally to summaries and errors all commands will be logged - - Configuration file example: - ``` - { - "INPUT_FILE" : "/opt/mongo_tests/output/oradev.json", - "DB_NAME" : "test", - "LOG_FILE" : "/opt/mongo_tests/MDBApply.log", - "CONNECT_STRING" : "mongodb://mongo-oci.acompany.com/test" - } - ``` -examples of using applier: -1. ``` - java -jar ./MDBApplier.jar - ``` - this example reads data from a file pointed by INPUT_FILE parameter and executes found commands against the database pointed by - CONNECT_STRING and DB_NAME - - - + +**Prerequisites** + +- The source MongoDB instance needs to be up and running. + +- To extract data from traditional MongoDB log file, enable logging -- + the so-called slow operations at the MongoDB instance level - by + executing the following command *db.setProfilingLevel(0, -1)* + +- To extract data from db.system.profile collection or its dump file, + enable traditional profiling, by executing the following command + *db.setProfilingLevel(2*) + +- The configuration file \$MR_CONFIG_DIR/MDBExtractConfig.json with the + required parameter settings is created. + +**Quick Start** + +Configuration File example: + +{ + +\"OUTPUT_DIR\" : \"/opt/mongo_tests/output\", + +\"INPUT_FILE\" : \"/opt/mongo_tests/mongod01.log\", + +\"EXCLUDE_COMMANDS\" : \[\"insert\"\], + +\"EXECUTION_PLAN_TRACING\" : 0, + +\"OUTPUT_MODE\" : \"JSON\" + +} + +- **OUTPUT_DIR** (default: standard output) defines the directory of the + output files. + +- **INPUT_FILE** (default: standard input) defines the input log file + generated by a MongoDB Instance with profiling enabled. + +- **EXCLUDE_COMMANDS** provides a JSON array with the list of commands, + which will be ignored during extraction process. + +- **EXECUTION_PLAN_TRACING** provides explain() commands, if set to a + non-zero value. Otherwise only runCommand will be provided in the + output. + +- **OUTPUT_MODE** defines the output format. If set to JSON, the output + will be formatted in sequence of JSON documents. If set to SCRIPT, the + output is formatted in NodeJS script, which can be consumed by + mongosh. + +All parameters and the complete description can be found here. (!!!!!!!! +add link) + +**Execution** + +java -jar ./MDBExtract.jar + +This example reads data from a file pointed by INPUT_FILE parameter and +executes found commands against the database pointed by CONNECT_STRING +and DB_NAME. + +tail -n 10000 -f ./mongod.log\|java -jar ./MDBExtract.jar + +If the INPUT_FILE parameter is not set, MDBExtractor reads the data from +its standard input. + +**MDBApplier** + +**Prerequisites** + +- The target Oracle Database with [[Oracle API for + MongoDB]{.underline}](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-1CF44843-6294-45F0-8065-B9E8034D6CB1) is + enabled. + +- A subset of the data is defined representing all collections which + will be traced. + +- The configuration file \$MR_CONFIG_DIR/MDBApplierConfig.json with the + required parameter settings is created. + +**Quick Start** + +Configuration File example: + +{ + +\"INPUT_FILE\" : \"/opt/mongo_tests/output/oradev.json\", + +\"DB_NAME\" : \"test\", + +\"LOG_FILE\" : \"/opt/mongo_tests/MDBApply.log\", + +\"CONNECT_STRING\" : \"mongodb://mongo-oci.acompany.com/test\" + +} + +- **INPUT_FILE** (default: standard input) defines the input log file + generated by a MDBExtractor. + +- **DB_NAME** (optional) defines the target database name. + +- **LOG_FILE** (default: standard diagnostic output) is the name of log + file used. + +- **CONNECT_STRING** (mandatory) defines the connect string for a target + MongoDB or Oracle API for MongoDB Instance. + +All parameters and the complete description can be found here. (!!! Add +link) + +**Execution** + +java -jar ./MDBApplier.jar + +The example reads data from a file defined by the INPUT_FILE parameter +and executes the commands using the parameters CONNECT_STRING and +DB_NAME.