Skip to content

Commit facdc63

Browse files
JesusGitHubOracleJesusGitHubOracle
authored
Merge pull request #1862 from oracle-devrel/witold-swierzy-patch-3
Witold swierzy patch 3
2 parents 33557aa + f3f1b32 commit facdc63

File tree

8 files changed

+833
-0
lines changed

8 files changed

+833
-0
lines changed

‎data-platform/autonomous-database/autonomous-json/MongoDBReplay/README.md

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
## MongoDB Replay
2+
3+
Purpose of this tool is to provide support with assessment of a MongoDB application with Oracle MongoDB API.
4+
5+
The repository contains source code of two artifacts
6+
1. MDBExtractor.jar
7+
- this is the tool, which extracts stream of commands from a log file generated by a MongoDB instance.
8+
9+
2. MDBApplier.jar
10+
- this is the tool, which executes commands extracted during the extraction process against an Oracle MongoDB API instance
11+
12+
## License
13+
14+
Copyright (c) 2025 Oracle and/or its affiliates.
15+
16+
Licensed under the Universal Permissive License (UPL), Version 1.0.
17+
18+
See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.
19+
20+
## WARNING
21+
#### This tool has been developed for test purposes only!
22+
#### Do not use this tool against production databases! It may drive to serious damages of a database or complete data loss! Oracle does not guarantee and does not take any responsibility of possible problems/issues/damages/data corruptions caused by using this tool.
23+
24+
25+
## Usage
26+
1. Generic requirements
27+
- MR_CONFIG_DIR variable has to be set to a directory, where configuration and log files are/will be stored
28+
29+
1.MDBExtractor
30+
requirements:
31+
- Source MongoDB instance up and running
32+
- Profiling at the MongoDB instance
33+
- to extract data from traditional Mongod log file, there is need to enable logging so called slow operations at the Mongod instance level. It can be done by executing the following command:
34+
35+
db.setProfilingLevel(0, -1)
36+
- The following configuration file $MR_CONFIG_DIR/MDBExtractConfig.json needs to be existing with the following parameters
37+
38+
OUTPUT_DIR : optional; points to directory where output files will be created;
39+
if this parameter is not set, then MDBExtractor will print out results into standard output
40+
41+
INPUT_FILE : optional; points to an input log file generated by a MongoDB Instance with profiling enabled
42+
if this parameter is not set, then MDBExtractor will read data from standard input
43+
44+
COMMANDS_LOGGING : optional; default value "true", if set to "true" then every command found by the extractor will be
45+
additional logged with console.log(...). Useful, when output will be processed by mongosh tool, which
46+
has limited capabilities related to error reporting. When set to "false" only commands will be reported.
47+
48+
INCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be extracted;
49+
all commands not listed here will not be printed out into output; this parameter CANNOT be used
50+
simultanously with EXCLUDE_COMMANDS
51+
52+
EXCLUDE_COMMANDS : optional; JSON array allowing for providing list of commands, which will be ignored during extraction process;
53+
all commands listed here will not be printed out into output; this parameter CANNOT be used
54+
simultanously with INCLUDE_COMMANDS
55+
if none of above parameters is set, then all commands will be traced
56+
57+
INCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will be traced.
58+
all databases not listed here will not be traced.
59+
This parameter cannot be used simultaneously with EXCLUDE_DATABASES
60+
61+
EXCLUDE_DATABASES : optional; JSON array allowing for providing list of databases, which will not be traced.
62+
all databases not listed here will be traced.
63+
This parameter cannot be used simultaneously with INCLUDE_DATABASES
64+
if none of above parameters is set, then all databases will be traced
65+
66+
EXECUTION_PLAN_TRACING : if set to a non-zero value, then output will contain explain() commands instead of runCommand;
67+
useful, when there is need to check/compare execution plans between a source and a target system
68+
69+
OUTPUT_MODE : can be set to JSON, which result in that the output data will be formated into sequence of JSON documents
70+
or to SCRIPT, wich will result in that the output data will be formated into NodeJS script, which can be
71+
consumed by mongosh
72+
LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to
73+
standard diagnostic output
74+
75+
LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0
76+
0 means that only summary of initialization and processing will be logged
77+
1 means that additionally to summaries also errors will be logged
78+
2 means that additionally to summaries and errors all commands will be logged
79+
80+
Configuration File example:
81+
```
82+
{
83+
"OUTPUT_DIR" : "/opt/mongo_tests/output",
84+
"INPUT_FILE" : "/opt/mongo_tests/mongod01.log",
85+
"COMMANDS_LOGGING" : "false",
86+
"INCLUDE_DATABASES" : ["oradev","test"],
87+
"EXCLUDE_COMMANDS" : ["insert"],
88+
"EXECUTION_PLAN_TRACING" : 0,
89+
"OUTPUT_MODE" : "JSON"
90+
}
91+
examples of using the tool
92+
1. ```
93+
java -jar ./MDBExtract.jar
94+
```
95+
this example reads data from a file pointed by INPUT_FILE parameter and generates output into set of files created in
96+
OUTPUT_DIR (every traced database will use a separate output file)
97+
98+
2. ```
99+
tail -n 10000 -f ./mongod.log|java -jar ./MDBExtract.jar
100+
```
101+
this example assumes, that INPUT_FILE parameter IS NOT SET. Instead of the tool reads data from its standard input
102+
103+
2.MDBApplier
104+
requirements
105+
- target MongoDB instance up and running
106+
- the following configuration file $MR_CONFIG_DIR/MDBApplierConfig.json needs to be existing with the following parameters
107+
INPUT_FILE : optional; points to an input log file generated by MDBExtractor
108+
if this parameter is not set, then MDBApplier will read data from standard input
109+
DB_NAME : optional; allows for providing target database name
110+
CONNECT_STRING : mandatoryl; connect string to a target MongoDB or Oracle API for MongoDB instance
111+
LOG_FILE : name of log file used by the tool; optional; if not set then all diangostic messages will be redirected to
112+
standard diagnostic output
113+
114+
LOG_LEVEL : optional; can be set to 0,1 or 2. Default value : 0
115+
0 means that only summary of initialization and processing will be logged
116+
1 means that additionally to summaries also errors will be logged
117+
2 means that additionally to summaries and errors all commands will be logged
118+
119+
Configuration file example:
120+
```
121+
{
122+
"INPUT_FILE" : "/opt/mongo_tests/output/oradev.json",
123+
"DB_NAME" : "test",
124+
"LOG_FILE" : "/opt/mongo_tests/MDBApply.log",
125+
"CONNECT_STRING" : "mongodb://mongo-oci.acompany.com/test"
126+
}
127+
```
128+
examples of using applier:
129+
1. ```
130+
java -jar ./MDBApplier.jar
131+
```
132+
this example reads data from a file pointed by INPUT_FILE parameter and executes found commands against the database pointed by
133+
CONNECT_STRING and DB_NAME
134+
135+
136+

‎data-platform/autonomous-database/autonomous-json/MongoDBReplay/bin/MDBApplier.jar

2.76 MB
Binary file not shown.

‎data-platform/autonomous-database/autonomous-json/MongoDBReplay/bin/MDBExtractor.jar

2.76 MB
Binary file not shown.

‎data-platform/autonomous-database/autonomous-json/MongoDBReplay/src/Applier.java

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
package utils.dbutils;
2+
3+
import com.google.gson.Gson;
4+
import com.google.gson.JsonObject;
5+
import com.mongodb.client.MongoClient;
6+
import com.mongodb.client.MongoClients;
7+
import com.mongodb.client.MongoDatabase;
8+
import org.bson.BsonDocument;
9+
import org.bson.Document;
10+
11+
import java.io.*;
12+
import java.time.LocalDateTime;
13+
14+
public class Applier {
15+
private static int numOfSucceses = 0;
16+
private static int numOfErrors = 0;
17+
private static int numOfAllCommands = 0;
18+
private static String dbName, oldDbname;
19+
private static MongoClient client;
20+
private static MongoDatabase db;
21+
private static BufferedReader inputFile;
22+
23+
public static void printSettings(boolean footer) {
24+
if (!footer)
25+
Config.logMessage(LocalDateTime.now() + " : Starting commands application. ", Config.LOG_LEVEL_SUMMARY);
26+
else {
27+
Config.logMessage(LocalDateTime.now() + " : Commands application completed.", Config.LOG_LEVEL_SUMMARY);
28+
Config.logMessage("Summary", Config.LOG_LEVEL_SUMMARY);
29+
}
30+
31+
if (Config.inputFileName != null)
32+
Config.logMessage("Input file : " + Config.inputFileName, Config.LOG_LEVEL_SUMMARY);
33+
else
34+
Config.logMessage("Input redirected to StdIn", Config.LOG_LEVEL_SUMMARY);
35+
36+
if (Config.logFileName != null)
37+
Config.logMessage("Log file : " + Config.logFileName, Config.LOG_LEVEL_SUMMARY);
38+
else
39+
Config.logMessage("Logging set to StdOut.", Config.LOG_LEVEL_SUMMARY);
40+
Config.logMessage("Log Level : "+Config.logLevel, Config.LOG_LEVEL_SUMMARY);
41+
Config.logMessage("Database connect string : "+Config.connectString, Config.LOG_LEVEL_SUMMARY);
42+
if (footer)
43+
{
44+
Config.logMessage("Number of all commands : " + numOfAllCommands, Config.LOG_LEVEL_SUMMARY);
45+
Config.logMessage("Number of successful executions : " + numOfSucceses, Config.LOG_LEVEL_SUMMARY);
46+
Config.logMessage("Number of errors : " + numOfErrors, Config.LOG_LEVEL_SUMMARY);
47+
}
48+
}
49+
50+
public static void initialize() throws Exception {
51+
Config.readConfiguration(Config.APPLY);
52+
printSettings(false);
53+
client = MongoClients.create(Config.connectString);
54+
db = client.getDatabase(Config.dbName);
55+
if (Config.inputFileName != null)
56+
inputFile = new BufferedReader((new FileReader(Config.inputFileName)));
57+
else
58+
inputFile = new BufferedReader(new InputStreamReader(System.in));
59+
dbName = Config.dbName;
60+
oldDbname = Config.dbName;
61+
}
62+
63+
public static synchronized void shutdown() {
64+
try {
65+
printSettings(true);
66+
Config.logFile.close();
67+
} catch (Exception e) {
68+
e.printStackTrace();
69+
System.exit(0);
70+
}
71+
}
72+
73+
public static void main(String[] args) {
74+
String line = "";
75+
JsonObject commandJSON;
76+
BsonDocument commandBSON;
77+
Document commandResult;
78+
79+
try {
80+
initialize();
81+
while ((line = inputFile.readLine()) != null) {
82+
numOfAllCommands++;
83+
try {
84+
Config.logMessage("Command #"+numOfAllCommands+" : "+line, Config.LOG_LEVEL_ALL);
85+
Gson gson = new Gson();
86+
commandJSON = gson.fromJson(line, JsonObject.class);
87+
dbName = commandJSON.getAsJsonPrimitive("$db").getAsString();
88+
if (!dbName.equals(oldDbname)) {
89+
db = client.getDatabase(dbName);
90+
oldDbname = dbName;
91+
}
92+
commandJSON.remove("$db");
93+
commandBSON = BsonDocument.parse(commandJSON.toString());
94+
commandResult = db.runCommand(commandBSON);
95+
Config.logMessage("Result : "+commandResult.toString().substring(8),Config.LOG_LEVEL_ALL);
96+
numOfSucceses++;
97+
} catch(Exception e) {
98+
numOfErrors++;
99+
Config.logMessage("Error #"+numOfErrors+" : "+line, Config.LOG_LEVEL_ERRORS);
100+
Config.logMessage(e.toString(), Config.LOG_LEVEL_ERRORS);
101+
}
102+
}
103+
} catch (Exception e)
104+
{e.printStackTrace();}
105+
shutdown();
106+
}
107+
}

0 commit comments

Comments
 (0)