Skip to content

Commit 30a3770

Browse files
Update README.md
1 parent cafe004 commit 30a3770

File tree

1 file changed

+160
-138
lines changed
  • data-platform/autonomous-database/autonomous-json/MongoDBReplay

1 file changed

+160
-138
lines changed
Lines changed: 160 additions & 138 deletions
Original file line numberDiff line numberDiff line change
@@ -1,150 +1,172 @@
1-
## MongoDB Replay
1+
**MongoDB Replay**
22

3-
Purpose of this tool is to provide support with assessment of a MongoDB application with Oracle MongoDB API.
3+
The purpose of MongoDB Replay is to provide support with assessment of a
4+
MongoDB application with Oracle MongoDB API. You can extract statements
5+
from the MongoDB log file and replay it in a test environment with
6+
MongoDB Replay.
47

5-
The repository contains source code of two artifacts
6-
1. MDBExtractor.jar
7-
- this is the tool, which extracts stream of commands from a log file generated by a MongoDB instance.
8+
**Disclaimer**
89

9-
2. MDBApplier.jar
10-
- this is the tool, which executes commands extracted during the extraction process against an Oracle MongoDB API instance
10+
The use of the scripts is at your sole risk. All information is provided
11+
\"as -is\", without any warranty of its accuracy, completeness, fitness
12+
for a particular purpose.
1113

12-
## License
14+
MongoDB Replay has been developed to replay the MongoDB statements in
15+
test environments only! Do not use it against production databases.
16+
Oracle does not take any responsibility of possible
17+
problems/issues/damages/data corruptions caused by using MongoDB Replay.
1318

14-
Copyright (c) 2025 Oracle and/or its affiliates.
19+
**Useful Links**
1520

16-
Licensed under the Universal Permissive License (UPL), Version 1.0.
21+
- [Tools and Drivers for Oracle Database API for
22+
MongoDB](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-9B704340-5E1D-4CA9-8944-6296A69221B0)
1723

18-
See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.
24+
- [Oracle Database Requirements and
25+
Configuration](https://docs.oracle.com/en/database/oracle/oracle-rest-data-services/24.4/ordig/enabling-and-configuring-oracle-database-api-mongodb.html#GUID-F79F2BB4-BB6E-4890-AFD7-E21DC4462835)
1926

20-
## WARNING
21-
#### This tool has been developed for test purposes only!
22-
#### Do not use this tool against production databases! It may drive to serious damages of a database or complete data loss! Oracle does not guarantee and does not take any responsibility of possible problems/issues/damages/data corruptions caused by using this tool.
27+
**Artifacts**
2328

29+
The repository contains source code of two artifacts:
2430

25-
## Usage
26-
### Generic requirements
27-
- MR_CONFIG_DIR variable has to be set to a directory, where configuration and log files are/will be stored
28-
- JDK 21 (it can be Oracle or OpenJDK)
29-
- Oracle JDK can be downloaded from the following link: https://www.oracle.com/java/technologies/downloads/#java21
30-
- OpenJDK is usually available in a Linux packates repository, but also can be downloaded and installed independly from the following link: https://openjdk.org/
31+
-----------------------------------------------------------------------
32+
Artifact Description
33+
----------------------- -----------------------------------------------
34+
MDBExtractor.jar extracts a stream of commands from a log file
35+
generated by a MongoDB instance.
36+
37+
MDBApplier.jar executes commands extracted during the
38+
extraction process against an Oracle MongoDB
39+
API instance
40+
-----------------------------------------------------------------------
41+
42+
**Requirements**
43+
44+
- A variable named MR_CONFIG_DIR has to be set to a directory, where
45+
configuration and log files are stored.
46+
47+
- JDK version 21 either from Oracle be downloaded from
48+
<https://www.oracle.com/java/technologies/downloads/#java21> or\
49+
OpenJDK available in a Linux packages repository or downloaded and
50+
installed
51+
from [[https://openjdk.org/]{.underline}](https://openjdk.org/)
3152

3253
### MDBExtractor
33-
requirements:
34-
- Source MongoDB instance up and running
35-
- Profiling at the MongoDB instance
36-
- to extract data from traditional Mongod log file, there is need to enable logging so called slow operations at the Mongod instance level. It can be done by executing the following command:
37-
38-
db.setProfilingLevel(0, -1)
39-
- to extract data from db.system.profile collection or its dump there is need to enable traditional profiling, by executing the following command:
40-
41-
db.setProfilingLevel(2)
42-
43-
- The following configuration file $MR_CONFIG_DIR/MDBExtractConfig.json needs to be existing with the following parameters
44-
45-
- OUTPUT_DIR : optional; points to directory where output files will be created;
46-
if this parameter is not set, then MDBExtractor will print out results into standard output
47-
48-
- INPUT_FILE : optional; points to an input log file generated by a MongoDB Instance with profiling enabled
49-
if this parameter is not set, then MDBExtractor will read data from standard input
50-
51-
- INPUT_FILE_FORMAT : optional; provides information about from which file the data will be read. It accepts two values
52-
MONGO_LOG : in that case Extractor will be reading data from traditional Mongod log file. Profiling needs to be set to (0,-1)
53-
MONGO_PROFILE : in that case Extractor will be reading data from dump of db.system.profile collection. Profiling needs to be set to (2)
54-
55-
- INPUT_CONNECT_STRING : optional; if set then it points to a MongoDB instance, from which input data will be read directly. Profiling needs to be set to (2). When this parameter is set, then INPUT_FILE and INPUT_FILE_FORMAT cannot be set.
56-
57-
- COMMANDS_LOGGING : optional; default value "true", if set to "true" then every command found by the extractor will be
58-
additional logged with console.log(...). Useful, when output will be processed by mongosh tool, which
59-
has limited capabilities related to error reporting. When set to "false" only commands will be reported.
60-
61-
- INCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be extracted;
62-
all commands not listed here will not be printed out into output; this parameter CANNOT be used
63-
simultanously with EXCLUDE_COMMANDS
64-
65-
- EXCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be ignored during extraction process;
66-
all commands listed here will not be printed out into output; this parameter CANNOT be used
67-
simultanously with INCLUDE_COMMANDS
68-
if none of above parameters is set, then all commands will be traced
69-
70-
- INCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will be traced.
71-
all databases not listed here will not be traced.
72-
This parameter cannot be used simultaneously with EXCLUDE_DATABASES
73-
74-
- EXCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will not be traced.
75-
all databases not listed here will be traced.
76-
This parameter cannot be used simultaneously with INCLUDE_DATABASES
77-
if none of above parameters is set, then all databases will be traced
78-
79-
- EXECUTION_PLAN_TRACING : if set to a non-zero value, then output will contain explain() commands instead of runCommand;
80-
useful, when there is need to check/compare execution plans between a source and a target system
81-
82-
- OUTPUT_MODE : can be set to JSON, which result in that the output data will be formated into sequence of JSON documents
83-
or to SCRIPT, wich will result in that the output data will be formated into NodeJS script, which can be
84-
consumed by mongosh
85-
- LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to
86-
standard diagnostic output
87-
88-
- LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0
89-
0 means that only summary of initialization and processing will be logged
90-
1 means that additionally to summaries also errors will be logged
91-
2 means that additionally to summaries and errors all commands will be logged
92-
93-
Configuration File example:
94-
```
95-
{
96-
"OUTPUT_DIR" : "/opt/mongo_tests/output",
97-
"INPUT_FILE" : "/opt/mongo_tests/mongod01.log",
98-
"COMMANDS_LOGGING" : "false",
99-
"INCLUDE_DATABASES" : ["oradev","test"],
100-
"EXCLUDE_COMMANDS" : ["insert"],
101-
"EXECUTION_PLAN_TRACING" : 0,
102-
"OUTPUT_MODE" : "JSON"
103-
}
104-
examples of using the tool
105-
1. ```
106-
java -jar ./MDBExtract.jar
107-
```
108-
this example reads data from a file pointed by INPUT_FILE parameter and generates output into set of files created in
109-
OUTPUT_DIR (every traced database will use a separate output file)
110-
111-
2. ```
112-
tail -n 10000 -f ./mongod.log|java -jar ./MDBExtract.jar
113-
```
114-
this example assumes, that INPUT_FILE parameter IS NOT SET. Instead of the tool reads data from its standard input
115-
116-
### MDBApplier
117-
requirements
118-
- target Oracle Database (19c or later) with [Oracle API for MongoDB](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-1CF44843-6294-45F0-8065-B9E8034D6CB1) enabled
119-
- subset of the data representing all collections which will be traced
120-
- the following configuration file $MR_CONFIG_DIR/MDBApplierConfig.json needs to be existing with the following parameters
121-
- INPUT_FILE : optional; points to an input log file generated by MDBExtractor
122-
if this parameter is not set, then MDBApplier will read data from standard input
123-
- DB_NAME : optional; allows for providing target database name
124-
- CONNECT_STRING : mandatoryl; connect string to a target MongoDB or Oracle API for MongoDB instance
125-
- LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to
126-
standard diagnostic output
127-
128-
- LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0
129-
0 means that only summary of initialization and processing will be logged
130-
1 means that additionally to summaries also errors will be logged
131-
2 means that additionally to summaries and errors all commands will be logged
132-
133-
Configuration file example:
134-
```
135-
{
136-
"INPUT_FILE" : "/opt/mongo_tests/output/oradev.json",
137-
"DB_NAME" : "test",
138-
"LOG_FILE" : "/opt/mongo_tests/MDBApply.log",
139-
"CONNECT_STRING" : "mongodb://mongo-oci.acompany.com/test"
140-
}
141-
```
142-
examples of using applier:
143-
1. ```
144-
java -jar ./MDBApplier.jar
145-
```
146-
this example reads data from a file pointed by INPUT_FILE parameter and executes found commands against the database pointed by
147-
CONNECT_STRING and DB_NAME
148-
149-
150-
54+
55+
**Prerequisites**
56+
57+
- The source MongoDB instance needs to be up and running.
58+
59+
- To extract data from traditional MongoDB log file, enable logging --
60+
the so-called slow operations at the MongoDB instance level - by
61+
executing the following command *db.setProfilingLevel(0, -1)*
62+
63+
- To extract data from db.system.profile collection or its dump file,
64+
enable traditional profiling, by executing the following command
65+
*db.setProfilingLevel(2*)
66+
67+
- The configuration file \$MR_CONFIG_DIR/MDBExtractConfig.json with the
68+
required parameter settings is created.
69+
70+
**Quick Start**
71+
72+
Configuration File example:
73+
74+
{
75+
76+
\"OUTPUT_DIR\" : \"/opt/mongo_tests/output\",
77+
78+
\"INPUT_FILE\" : \"/opt/mongo_tests/mongod01.log\",
79+
80+
\"EXCLUDE_COMMANDS\" : \[\"insert\"\],
81+
82+
\"EXECUTION_PLAN_TRACING\" : 0,
83+
84+
\"OUTPUT_MODE\" : \"JSON\"
85+
86+
}
87+
88+
- **OUTPUT_DIR** (default: standard output) defines the directory of the
89+
output files.
90+
91+
- **INPUT_FILE** (default: standard input) defines the input log file
92+
generated by a MongoDB Instance with profiling enabled.
93+
94+
- **EXCLUDE_COMMANDS** provides a JSON array with the list of commands,
95+
which will be ignored during extraction process.
96+
97+
- **EXECUTION_PLAN_TRACING** provides explain() commands, if set to a
98+
non-zero value. Otherwise only runCommand will be provided in the
99+
output.
100+
101+
- **OUTPUT_MODE** defines the output format. If set to JSON, the output
102+
will be formatted in sequence of JSON documents. If set to SCRIPT, the
103+
output is formatted in NodeJS script, which can be consumed by
104+
mongosh.
105+
106+
All parameters and the complete description can be found here. (!!!!!!!!
107+
add link)
108+
109+
**Execution**
110+
111+
java -jar ./MDBExtract.jar
112+
113+
This example reads data from a file pointed by INPUT_FILE parameter and
114+
executes found commands against the database pointed by CONNECT_STRING
115+
and DB_NAME.
116+
117+
tail -n 10000 -f ./mongod.log\|java -jar ./MDBExtract.jar
118+
119+
If the INPUT_FILE parameter is not set, MDBExtractor reads the data from
120+
its standard input.
121+
122+
**MDBApplier**
123+
124+
**Prerequisites**
125+
126+
- The target Oracle Database with [[Oracle API for
127+
MongoDB]{.underline}](https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-1CF44843-6294-45F0-8065-B9E8034D6CB1) is
128+
enabled.
129+
130+
- A subset of the data is defined representing all collections which
131+
will be traced.
132+
133+
- The configuration file \$MR_CONFIG_DIR/MDBApplierConfig.json with the
134+
required parameter settings is created.
135+
136+
**Quick Start**
137+
138+
Configuration File example:
139+
140+
{
141+
142+
\"INPUT_FILE\" : \"/opt/mongo_tests/output/oradev.json\",
143+
144+
\"DB_NAME\" : \"test\",
145+
146+
\"LOG_FILE\" : \"/opt/mongo_tests/MDBApply.log\",
147+
148+
\"CONNECT_STRING\" : \"mongodb://mongo-oci.acompany.com/test\"
149+
150+
}
151+
152+
- **INPUT_FILE** (default: standard input) defines the input log file
153+
generated by a MDBExtractor.
154+
155+
- **DB_NAME** (optional) defines the target database name.
156+
157+
- **LOG_FILE** (default: standard diagnostic output) is the name of log
158+
file used.
159+
160+
- **CONNECT_STRING** (mandatory) defines the connect string for a target
161+
MongoDB or Oracle API for MongoDB Instance.
162+
163+
All parameters and the complete description can be found here. (!!! Add
164+
link)
165+
166+
**Execution**
167+
168+
java -jar ./MDBApplier.jar
169+
170+
The example reads data from a file defined by the INPUT_FILE parameter
171+
and executes the commands using the parameters CONNECT_STRING and
172+
DB_NAME.

0 commit comments

Comments
 (0)