Skip to content

Update README.md #1927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 1, 2025
Merged

Update README.md #1927

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
298 changes: 160 additions & 138 deletions data-platform/autonomous-database/autonomous-json/MongoDBReplay/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,150 +1,172 @@
## MongoDB Replay
**MongoDB Replay**

Purpose of this tool is to provide support with assessment of a MongoDB application with Oracle MongoDB API.
The purpose of MongoDB Replay is to provide support with assessment of a
MongoDB application with Oracle MongoDB API. You can extract statements
from the MongoDB log file and replay it in a test environment with
MongoDB Replay.

The repository contains source code of two artifacts
1. MDBExtractor.jar
- this is the tool, which extracts stream of commands from a log file generated by a MongoDB instance.
**Disclaimer**

2. MDBApplier.jar
- this is the tool, which executes commands extracted during the extraction process against an Oracle MongoDB API instance
The use of the scripts is at your sole risk. All information is provided
\"as -is\", without any warranty of its accuracy, completeness, fitness
for a particular purpose.

## License
MongoDB Replay has been developed to replay the MongoDB statements in
test environments only! Do not use it against production databases.
Oracle does not take any responsibility of possible
problems/issues/damages/data corruptions caused by using MongoDB Replay.

Copyright (c) 2025 Oracle and/or its affiliates.
**Useful Links**

Licensed under the Universal Permissive License (UPL), Version 1.0.
- [Tools and Drivers for Oracle Database API for
MongoDB](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-9B704340-5E1D-4CA9-8944-6296A69221B0)

See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.
- [Oracle Database Requirements and
Configuration](https://docs.oracle.com/en/database/oracle/oracle-rest-data-services/24.4/ordig/enabling-and-configuring-oracle-database-api-mongodb.html#GUID-F79F2BB4-BB6E-4890-AFD7-E21DC4462835)

## WARNING
#### This tool has been developed for test purposes only!
#### Do not use this tool against production databases! It may drive to serious damages of a database or complete data loss! Oracle does not guarantee and does not take any responsibility of possible problems/issues/damages/data corruptions caused by using this tool.
**Artifacts**

The repository contains source code of two artifacts:

## Usage
### Generic requirements
- MR_CONFIG_DIR variable has to be set to a directory, where configuration and log files are/will be stored
- JDK 21 (it can be Oracle or OpenJDK)
- Oracle JDK can be downloaded from the following link: https://www.oracle.com/java/technologies/downloads/#java21
- OpenJDK is usually available in a Linux packates repository, but also can be downloaded and installed independly from the following link: https://openjdk.org/
-----------------------------------------------------------------------
Artifact Description
----------------------- -----------------------------------------------
MDBExtractor.jar extracts a stream of commands from a log file
generated by a MongoDB instance.

MDBApplier.jar executes commands extracted during the
extraction process against an Oracle MongoDB
API instance
-----------------------------------------------------------------------

**Requirements**

- A variable named MR_CONFIG_DIR has to be set to a directory, where
configuration and log files are stored.

- JDK version 21 either from Oracle be downloaded from
<https://www.oracle.com/java/technologies/downloads/#java21> or\
OpenJDK available in a Linux packages repository or downloaded and
installed
from [[https://openjdk.org/]{.underline}](https://openjdk.org/)

### MDBExtractor
requirements:
- Source MongoDB instance up and running
- Profiling at the MongoDB instance
- to extract data from traditional Mongod log file, there is need to enable logging so called slow operations at the Mongod instance level. It can be done by executing the following command:

db.setProfilingLevel(0, -1)
- to extract data from db.system.profile collection or its dump there is need to enable traditional profiling, by executing the following command:

db.setProfilingLevel(2)

- The following configuration file $MR_CONFIG_DIR/MDBExtractConfig.json needs to be existing with the following parameters

- OUTPUT_DIR : optional; points to directory where output files will be created;
if this parameter is not set, then MDBExtractor will print out results into standard output

- INPUT_FILE : optional; points to an input log file generated by a MongoDB Instance with profiling enabled
if this parameter is not set, then MDBExtractor will read data from standard input

- INPUT_FILE_FORMAT : optional; provides information about from which file the data will be read. It accepts two values
MONGO_LOG : in that case Extractor will be reading data from traditional Mongod log file. Profiling needs to be set to (0,-1)
MONGO_PROFILE : in that case Extractor will be reading data from dump of db.system.profile collection. Profiling needs to be set to (2)

- INPUT_CONNECT_STRING : optional; if set then it points to a MongoDB instance, from which input data will be read directly. Profiling needs to be set to (2). When this parameter is set, then INPUT_FILE and INPUT_FILE_FORMAT cannot be set.

- COMMANDS_LOGGING : optional; default value "true", if set to "true" then every command found by the extractor will be
additional logged with console.log(...). Useful, when output will be processed by mongosh tool, which
has limited capabilities related to error reporting. When set to "false" only commands will be reported.

- INCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be extracted;
all commands not listed here will not be printed out into output; this parameter CANNOT be used
simultanously with EXCLUDE_COMMANDS

- EXCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be ignored during extraction process;
all commands listed here will not be printed out into output; this parameter CANNOT be used
simultanously with INCLUDE_COMMANDS
if none of above parameters is set, then all commands will be traced

- INCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will be traced.
all databases not listed here will not be traced.
This parameter cannot be used simultaneously with EXCLUDE_DATABASES

- EXCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will not be traced.
all databases not listed here will be traced.
This parameter cannot be used simultaneously with INCLUDE_DATABASES
if none of above parameters is set, then all databases will be traced

- EXECUTION_PLAN_TRACING : if set to a non-zero value, then output will contain explain() commands instead of runCommand;
useful, when there is need to check/compare execution plans between a source and a target system

- OUTPUT_MODE : can be set to JSON, which result in that the output data will be formated into sequence of JSON documents
or to SCRIPT, wich will result in that the output data will be formated into NodeJS script, which can be
consumed by mongosh
- LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to
standard diagnostic output

- LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0
0 means that only summary of initialization and processing will be logged
1 means that additionally to summaries also errors will be logged
2 means that additionally to summaries and errors all commands will be logged

Configuration File example:
```
{
"OUTPUT_DIR" : "/opt/mongo_tests/output",
"INPUT_FILE" : "/opt/mongo_tests/mongod01.log",
"COMMANDS_LOGGING" : "false",
"INCLUDE_DATABASES" : ["oradev","test"],
"EXCLUDE_COMMANDS" : ["insert"],
"EXECUTION_PLAN_TRACING" : 0,
"OUTPUT_MODE" : "JSON"
}
examples of using the tool
1. ```
java -jar ./MDBExtract.jar
```
this example reads data from a file pointed by INPUT_FILE parameter and generates output into set of files created in
OUTPUT_DIR (every traced database will use a separate output file)

2. ```
tail -n 10000 -f ./mongod.log|java -jar ./MDBExtract.jar
```
this example assumes, that INPUT_FILE parameter IS NOT SET. Instead of the tool reads data from its standard input

### MDBApplier
requirements
- target Oracle Database (19c or later) with [Oracle API for MongoDB](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-1CF44843-6294-45F0-8065-B9E8034D6CB1) enabled
- subset of the data representing all collections which will be traced
- the following configuration file $MR_CONFIG_DIR/MDBApplierConfig.json needs to be existing with the following parameters
- INPUT_FILE : optional; points to an input log file generated by MDBExtractor
if this parameter is not set, then MDBApplier will read data from standard input
- DB_NAME : optional; allows for providing target database name
- CONNECT_STRING : mandatoryl; connect string to a target MongoDB or Oracle API for MongoDB instance
- LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to
standard diagnostic output

- LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0
0 means that only summary of initialization and processing will be logged
1 means that additionally to summaries also errors will be logged
2 means that additionally to summaries and errors all commands will be logged

Configuration file example:
```
{
"INPUT_FILE" : "/opt/mongo_tests/output/oradev.json",
"DB_NAME" : "test",
"LOG_FILE" : "/opt/mongo_tests/MDBApply.log",
"CONNECT_STRING" : "mongodb://mongo-oci.acompany.com/test"
}
```
examples of using applier:
1. ```
java -jar ./MDBApplier.jar
```
this example reads data from a file pointed by INPUT_FILE parameter and executes found commands against the database pointed by
CONNECT_STRING and DB_NAME




**Prerequisites**

- The source MongoDB instance needs to be up and running.

- To extract data from traditional MongoDB log file, enable logging --
the so-called slow operations at the MongoDB instance level - by
executing the following command *db.setProfilingLevel(0, -1)*

- To extract data from db.system.profile collection or its dump file,
enable traditional profiling, by executing the following command
*db.setProfilingLevel(2*)

- The configuration file \$MR_CONFIG_DIR/MDBExtractConfig.json with the
required parameter settings is created.

**Quick Start**

Configuration File example:

{

\"OUTPUT_DIR\" : \"/opt/mongo_tests/output\",

\"INPUT_FILE\" : \"/opt/mongo_tests/mongod01.log\",

\"EXCLUDE_COMMANDS\" : \[\"insert\"\],

\"EXECUTION_PLAN_TRACING\" : 0,

\"OUTPUT_MODE\" : \"JSON\"

}

- **OUTPUT_DIR** (default: standard output) defines the directory of the
output files.

- **INPUT_FILE** (default: standard input) defines the input log file
generated by a MongoDB Instance with profiling enabled.

- **EXCLUDE_COMMANDS** provides a JSON array with the list of commands,
which will be ignored during extraction process.

- **EXECUTION_PLAN_TRACING** provides explain() commands, if set to a
non-zero value. Otherwise only runCommand will be provided in the
output.

- **OUTPUT_MODE** defines the output format. If set to JSON, the output
will be formatted in sequence of JSON documents. If set to SCRIPT, the
output is formatted in NodeJS script, which can be consumed by
mongosh.

All parameters and the complete description can be found here. (!!!!!!!!
add link)

**Execution**

java -jar ./MDBExtract.jar

This example reads data from a file pointed by INPUT_FILE parameter and
executes found commands against the database pointed by CONNECT_STRING
and DB_NAME.

tail -n 10000 -f ./mongod.log\|java -jar ./MDBExtract.jar

If the INPUT_FILE parameter is not set, MDBExtractor reads the data from
its standard input.

**MDBApplier**

**Prerequisites**

- The target Oracle Database with [[Oracle API for
MongoDB]{.underline}](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-1CF44843-6294-45F0-8065-B9E8034D6CB1) is
enabled.

- A subset of the data is defined representing all collections which
will be traced.

- The configuration file \$MR_CONFIG_DIR/MDBApplierConfig.json with the
required parameter settings is created.

**Quick Start**

Configuration File example:

{

\"INPUT_FILE\" : \"/opt/mongo_tests/output/oradev.json\",

\"DB_NAME\" : \"test\",

\"LOG_FILE\" : \"/opt/mongo_tests/MDBApply.log\",

\"CONNECT_STRING\" : \"mongodb://mongo-oci.acompany.com/test\"

}

- **INPUT_FILE** (default: standard input) defines the input log file
generated by a MDBExtractor.

- **DB_NAME** (optional) defines the target database name.

- **LOG_FILE** (default: standard diagnostic output) is the name of log
file used.

- **CONNECT_STRING** (mandatory) defines the connect string for a target
MongoDB or Oracle API for MongoDB Instance.

All parameters and the complete description can be found here. (!!! Add
link)

**Execution**

java -jar ./MDBApplier.jar

The example reads data from a file defined by the INPUT_FILE parameter
and executes the commands using the parameters CONNECT_STRING and
DB_NAME.