Skip to content

Commit 444fa19

Browse files
feature(core): add new features and code comments (#11)
* feat(mqtt-provider): ✨ add new MQTT provider to providers collection * perf(general): ⚡ modify some environment variables from some components * perf(general): ⚡ modify some environment variables from some components * docs(general): 📝 documentation process update * docs(general): 📝 add new documentation on some packets * feat(doorkeeper): ✨ added the options to include dynamic defaults * feat(utils): ✨ added a timeout option for each try to retry module * fix(core): 🐛 fix some minor bugs in Ports and Managers * refactor(logger): ♻️ refactor code to use LoggerInstance instead of Logger * refactor(middlewares): ♻️ include a prefix in metrics middleware * fix(logger): 🩹 fix minor bug in logger and dependencies updates * feat(app-wrapper): ✨ add the option to monitor providers from the app-wrapper * chore(limiter): 🚧 working on limiter functionality * feat(limiter): 🚧 work in progress over the limiter/puller functionality * refactor(general): 🚧 working in new release * refactor(general): 🚧 refactor the most of the modules * chore: 🚧 update modules * chore(general): 🚧 update flaky test * chore(general): 🚧 update test * test(general): ✅ correct son flaky test * ci(general): 💚 fix sonarqube project setup * test(general): 🧪 fix some flaky test * fix(general): 🚧 fix flaky tests * ci(ci): 💚 fix sonarqube config file * ci(ci): 💚 fix sonar * ci(ci): 💚 update ci pipeline * ci(ci): 💚 update pipeline * ci(ci): 💚 update sonar config * ci(ci): 💚 fix sonar * ci(ci): 💚 add snyk to test * ci(ci): 💚 update for snyk * ci(ci): 💚 add snyk to pipeline * ci(ci): 💚 update snyk pipeline * ci(ci): 💚 update snyk * ci(ci): 💚 update snyk * ci(ci): 💚 update ci for snyk * ci(ci): 💚 update ci for snyk * ci(ci): 💚 update ci snyk * chore(general): 🚧 work in progress for 1.4 release * chore(ci): ⬆️ update dependencies * fix(app-wrapper): 🐛 update the logger to be public * fix(app-wrapper): 🐛 fix minor bug in default value for bootstrap settings * fix(app-wrapper): 🐛 fix minor bug in version/relase tags * fix(tasks): 🐛 control the the case when start/stop is called several times * feat(tasks): ✨ add new options to add resources to the scheduler * feat(service-config): ✨ add new options to include base and default configs * fix(app-wrapper): 🐛 solve minor bug in port selection in cluster mode * fix(app-wrapper): 🐛 solve a problem with the port in cluster a no cluster mode * fix(service-config): * fix(mqtt-provider): ⬆️ update dependencies * fix(elastic-provider): 🐛 fix health check for new version of elastic * fix(app-wrapper): 🐛 fix proxy on cluster mode * feat(firehose): ✨ add options for metrics in plugs * refactor(firehose): ♻️ refactor how the sinks and source are initialized * fix(app-wrapper): * perf(amqp-provider): ⚡ add own socket management to amqp provider * chore(amqp-provider): * chore(amqp-provider): 🚧 over amqp protocol * chore(amqp-provider): 🚧 amqp protocol * fix(doorkeeper): * fix(mqtt-provider): ♻️ refactor code for new change on mqtt module * fix(tasks): 🐛 fix minor bug related with side effects on tasks * fix(jobs): 🐛 fix a minor bug on delay time in Limiter * fix(jobs): 🐛 minor bug on delay time in Limiter * ci(ci): 💚 update sonarquebe tasks * feat(): add S3 provider (#10) * chore(general): ⬆️ update dependencies for all the packages --------- Co-authored-by: DanielaDurandBartolo <[email protected]>
1 parent 9993fc9 commit 444fa19

File tree

1,493 files changed

+37040
-23100
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,493 files changed

+37040
-23100
lines changed

.config/azure-pipelines.yml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
## Template for a monorepository based on GitFlow, Docker and DockerCompose in Netin Development Ecosystem
1+
## Template for a monorepository based on GitFlow, Docker and DockerCompose in Mytra Development Ecosystem
22
name: mdf-js$(Date:yyyyMMdd)$(Rev:.r)
33
## CI trigger based on GitFlow
44
trigger:
@@ -10,7 +10,7 @@ trigger:
1010
pr:
1111
- master
1212
- develop
13-
## Netin - Variables
13+
## Mytra - Variables
1414
variables:
1515
## Artifact name
1616
- name: artifactName
@@ -28,6 +28,8 @@ stages:
2828
displayName: 'Application Build'
2929
pool:
3030
name: NUCHUB
31+
demands:
32+
- Agent.OS -equals Linux
3133
jobs:
3234
- job: 'CODE_QUALITY'
3335
displayName: 'Code Quality Assurance'

.config/copyMedia.sh

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
3+
DESTINATION_DIR="docs/assets/media" # Ruta de destino
4+
SOURCE_DIR="packages/*/*/media" # Ruta de origen
5+
HTML_DIR="docs" # Ruta de destino de los archivos HTML
6+
7+
# Crear la carpeta de destino si no existe
8+
mkdir -p "$DESTINATION_DIR"
9+
10+
# Copiar todos los archivos de las carpetas media a la carpeta de destino
11+
find $SOURCE_DIR -type f -exec cp {} "$DESTINATION_DIR" \;
12+
13+
# Modificar los enlaces en los archivos HTML
14+
find $HTML_DIR -name "*.html" -type f | while read file; do
15+
if [ -f "$file" ]; then
16+
sed -i'' -e 's/src="media\//src="..\/assets\/media\//g' "$file"
17+
fi
18+
done

.config/envDoc.mjs

Lines changed: 352 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,352 @@
1+
/**
2+
* Copyright 2023 Netin Systems S.L. All rights reserved.
3+
* Note: All information contained herein is, and remains the property of Netin Systems S.L. and its
4+
* suppliers, if any. The intellectual and technical concepts contained herein are property of
5+
* Netin Systems S.L. and its suppliers and may be covered by European and Foreign patents, patents
6+
* in process, and are protected by trade secret or copyright.
7+
*
8+
* Dissemination of this information or the reproduction of this material is strictly forbidden
9+
* unless prior written permission is obtained from Netin Systems S.L.
10+
*/
11+
12+
import fs from 'fs';
13+
import * as glob from 'glob';
14+
import path from 'path';
15+
import { remark } from 'remark';
16+
import remarkGFM from 'remark-gfm';
17+
import remarkLint from 'remark-lint';
18+
import remarkLintUnorderedListMarkerStyle from 'remark-lint-unordered-list-marker-style';
19+
import remarkParse from 'remark-parse';
20+
import remarkStringify from 'remark-stringify';
21+
import remarkToc from 'remark-toc';
22+
23+
/**
24+
* This script is used to prepare the artifacts for the distribution.
25+
* It will generate an artifact for each each application in apps folder and will include all the
26+
* necessary files for official Netin distribution.
27+
* All the specific files for each application is stored in the apps folder, the rest of the files
28+
* are stored in the .distribution folder.
29+
*/
30+
31+
const SOURCE_FILES = './src/**/*.ts';
32+
const README_FILE = 'README.md';
33+
34+
// Expresiones regulares para identificar variables de entorno y comentarios
35+
const envVarRegex = /process\.env\[['"](?<varName>[^'"]+)['"]\]/;
36+
const defaultValueRegex = /(default:|@defaultValue) (?<defaultValue>[^,]+)/;
37+
38+
const srcFiles = glob.sync(SOURCE_FILES, {
39+
nodir: true, ignore: {
40+
ignored: p => /\.test.ts$/.test(p.name),
41+
}
42+
});
43+
const variables = [];
44+
45+
for (const file of srcFiles) {
46+
const fileContent = fs.readFileSync(file, 'utf8');
47+
const matches = fileContent.match(envVarRegex);
48+
if (matches) {
49+
findAllEnvironmentVariablesInFile(fileContent, path.basename(file));
50+
}
51+
}
52+
// Now we have all the environment variables and their information, we need to update the markdown file
53+
// with the information. For this we need to generate a new level 2 title **Environment variables**,
54+
// overwriting the previous one if exists, and then generate a table with the information of each
55+
// environment variable. We must include in the TOC the new title if it does not exist.
56+
const readmeFile = fs.readFileSync(README_FILE, 'utf8');
57+
const readMeMD = remark()
58+
.use(remarkParse)
59+
.use(remarkLint)
60+
.use(remarkLintUnorderedListMarkerStyle, '-')
61+
.use(remarkToc)
62+
.use(remarkGFM)
63+
.parse(readmeFile);
64+
65+
// We need to find all the nodes between the title **Environment variables** and the next title and remove them
66+
// from the tree, including the title itself. We also need to find the node of the license and next nodes and
67+
// save it to add it later.
68+
findAndRemoveEnvironmentVariablesSection(readMeMD);
69+
if (variables.length > 0) {
70+
const licenseNodes = findRemoveAndReturnLinceSection(readMeMD);
71+
const envVariablesNodes = generateListOfEnvironmentVariables(variables);
72+
readMeMD.children.push(...envVariablesNodes);
73+
if (licenseNodes) {
74+
readMeMD.children.push(...licenseNodes);
75+
}
76+
}
77+
const newReadmeFile = remark()
78+
.use(remarkParse)
79+
.use(remarkLint)
80+
.use(remarkLintUnorderedListMarkerStyle, '-')
81+
.use(remarkToc)
82+
.use(remarkGFM)
83+
.use(remarkStringify, { bullet: '-' })
84+
.stringify(readMeMD);
85+
fs.writeFileSync(`${README_FILE}`, newReadmeFile);
86+
87+
/**
88+
* Finds all environment variables in a file and extracts their information.
89+
*
90+
* @param {string} fileContent - The content of the file.
91+
* @param {string} path - The path of the file.
92+
*/
93+
function findAllEnvironmentVariablesInFile(fileContent, path) {
94+
const lines = fileContent.split('\r\n');
95+
for (let index = 0; index < lines.length; index++) {
96+
const matches = lines[index].match(envVarRegex);
97+
if (matches && matches.groups && matches.groups['varName']) {
98+
let comment = findUpperCommentInFile(lines, index);
99+
const defaultValue = getDefaultValueFromComment(comment);
100+
if (defaultValue) {
101+
comment = comment.replace(defaultValueRegex, '').trim();
102+
}
103+
variables.push({ name: matches.groups['varName'], comment, path, defaultValue });
104+
}
105+
}
106+
}
107+
/**
108+
* Finds the upper comment in a file based on a given index.
109+
* The comment is always located above the declaration of a constant.
110+
*
111+
* @param {string[]} lines - The lines of the file.
112+
* @param {number} index - The index to start searching from.
113+
* @returns {string|undefined} - The upper comment or undefined if no comment is found.
114+
*/
115+
function findUpperCommentInFile(lines, index) {
116+
let endOfComment = 0;
117+
let startOfComment = 0;
118+
let comment = undefined;
119+
if (index < 0) {
120+
return;
121+
}
122+
// Need to find the line where the constant is declared, the comment is always above the declaration
123+
// the function isConstantDeclaration is used to find the line where the constant is declared
124+
while (index >= 0) {
125+
if (isConstantDeclaration(lines[index])) {
126+
break;
127+
}
128+
index--;
129+
}
130+
if (index < 0) {
131+
return;
132+
}
133+
// Should check if the upper line has the end of comment, in other case, no comment is found
134+
if (!lines[index - 1].includes('*/')) {
135+
return;
136+
}
137+
endOfComment = index - 1;
138+
// Search for the start of the comment
139+
while (index >= 0) {
140+
if (lines[index].includes('/**')) {
141+
break;
142+
}
143+
index--;
144+
}
145+
if (index < 0) {
146+
return;
147+
}
148+
startOfComment = index;
149+
// Join all the lines of the comment, cleaning:
150+
// - '/**' in the first line
151+
// - '*/' in the last line
152+
// - '*' in the rest of the lines
153+
if (startOfComment === endOfComment) {
154+
comment = lines[startOfComment].replace('/**', '').replace('*/', '');
155+
} else {
156+
const firstLine = lines[startOfComment].replace('/**', '');
157+
const lastLine = lines[endOfComment].replace('*/', '');
158+
const middleLines = lines.slice(startOfComment + 1, endOfComment).map(line => line.replace('*', ''));
159+
comment = [firstLine, ...middleLines, lastLine].join(' ');
160+
}
161+
// Clean spaces if there are more than one space between words or at the beginning or end of the comment
162+
return comment.replace(/\s\s+/g, ' ').trim();
163+
}
164+
/**
165+
* Obtiene el valor predeterminado de un comentario.
166+
* @param {string} comment - El comentario del cual se desea obtener el valor predeterminado.
167+
* @returns {string|undefined} El valor predeterminado del comentario, o undefined si no se encuentra.
168+
*/
169+
function getDefaultValueFromComment(comment) {
170+
if (!comment) {
171+
return;
172+
}
173+
const matches = comment.match(defaultValueRegex);
174+
if (matches && matches.groups && matches.groups['defaultValue']) {
175+
return matches.groups['defaultValue'];
176+
}
177+
return;
178+
}
179+
/**
180+
* Checks if a given line is a constant declaration.
181+
*
182+
* @param {string} line - The line to check.
183+
* @returns {boolean} - True if the line is a constant declaration, false otherwise.
184+
*/
185+
function isConstantDeclaration(line) {
186+
return line.includes('const');
187+
}
188+
/**
189+
* Finds and removes the section containing environment variables from a given Markdown document.
190+
*
191+
* @param {object} readMeMD - The Markdown document object.
192+
*/
193+
function findAndRemoveEnvironmentVariablesSection(readMeMD) {
194+
for (let index = 0; index < readMeMD.children.length; index++) {
195+
const node = readMeMD.children[index];
196+
if (node.type === 'heading') {
197+
let nodeTitle = undefined;
198+
if (node.children[0].type === 'strong') {
199+
nodeTitle = node.children[0].children[0].value;
200+
} else if (node.children[0].type === 'text') {
201+
nodeTitle = node.children[0].value;
202+
}
203+
if (nodeTitle === 'Environment variables') {
204+
// Remove all the nodes between the title and the next title
205+
readMeMD.children.splice(index, 1);
206+
while (index < readMeMD.children.length) {
207+
if (readMeMD.children[index].type === 'heading' && readMeMD.children[index].depth === 2) {
208+
break;
209+
}
210+
readMeMD.children.splice(index, 1);
211+
}
212+
break;
213+
}
214+
}
215+
}
216+
}
217+
/**
218+
* Finds, removes, and returns the license section from the given readMeMD object.
219+
*
220+
* @param {Object} readMeMD - The readMeMD object.
221+
* @returns {Array} - An array of nodes representing the license section.
222+
*/
223+
function findRemoveAndReturnLinceSection(readMeMD) {
224+
let licenseNodes = [];
225+
for (let index = 0; index < readMeMD.children.length; index++) {
226+
const node = readMeMD.children[index];
227+
if (node.type === 'heading') {
228+
let nodeTitle = undefined;
229+
if (node.children[0].type === 'strong') {
230+
nodeTitle = node.children[0].children[0].value;
231+
} else if (node.children[0].type === 'text') {
232+
nodeTitle = node.children[0].value;
233+
}
234+
if (nodeTitle === 'License') {
235+
// Save all the nodes between the title and the next title, including the title
236+
licenseNodes.push(node);
237+
readMeMD.children.splice(index, 1);
238+
while (index < readMeMD.children.length) {
239+
if (readMeMD.children[index].type === 'heading' && readMeMD.children[index].depth === 2) {
240+
break;
241+
}
242+
licenseNodes.push(readMeMD.children[index]);
243+
readMeMD.children.splice(index, 1);
244+
}
245+
}
246+
}
247+
}
248+
return deletePositionPropertyRecursively(licenseNodes);
249+
}
250+
/**
251+
* Recursively deletes the 'position' property from each node in the given array of nodes.
252+
* If a node has children, the function is called recursively on each child node.
253+
* @param {Array} nodes - The array of nodes to process.
254+
* @returns {Array} - The modified array of nodes.
255+
*/
256+
function deletePositionPropertyRecursively(nodes) {
257+
for (const node of nodes) {
258+
delete node.position;
259+
if (node.children) {
260+
deletePositionPropertyRecursively(node.children);
261+
}
262+
}
263+
return nodes;
264+
}
265+
/**
266+
* Generates a table of environment variables.
267+
*
268+
* @param {Array<Object>} envVariables - An array of environment variables.
269+
* @returns {Array<Object>} - An array representing the table of environment variables.
270+
*/
271+
function generateTableOfEnvironmentVariables(envVariables) {
272+
const envVariablesTitle = {
273+
type: 'heading',
274+
depth: 2,
275+
children: [{ type: 'strong', children: [{ type: 'text', value: 'Environment variables' }] }],
276+
};
277+
const envVariablesTable = {
278+
type: 'table',
279+
align: ['left', 'center', 'left'],
280+
children: [
281+
{
282+
type: 'tableRow',
283+
children: [
284+
{ type: 'tableCell', children: [{ type: 'text', value: 'Variable' }] },
285+
{ type: 'tableCell', children: [{ type: 'text', value: 'Default value' }] },
286+
{ type: 'tableCell', children: [{ type: 'text', value: 'Description' }] },
287+
],
288+
},
289+
],
290+
};
291+
for (const envVariable of envVariables) {
292+
envVariablesTable.children.push({
293+
type: 'tableRow',
294+
children: [
295+
{ type: 'tableCell', children: [{ type: 'text', value: envVariable.name }] },
296+
{ type: 'tableCell', children: [{ type: 'text', value: envVariable.defaultValue }] },
297+
{ type: 'tableCell', children: [{ type: 'text', value: envVariable.comment }] },
298+
],
299+
});
300+
}
301+
return [envVariablesTitle, envVariablesTable];
302+
}
303+
/**
304+
* Generates a list of environment variables.
305+
*
306+
* @param {Array<Object>} envVariables - The array of environment variables.
307+
* @returns {Array<Object>} - The list of environment variables in a specific format.
308+
*/
309+
function generateListOfEnvironmentVariables(envVariables) {
310+
const envVariablesTitle = {
311+
type: 'heading',
312+
depth: 2,
313+
children: [{ type: 'strong', children: [{ type: 'text', value: 'Environment variables' }] }],
314+
};
315+
const envVariablesList = {
316+
type: 'list',
317+
ordered: false,
318+
spread: false,
319+
children: [],
320+
};
321+
for (const envVariable of envVariables) {
322+
if (!envVariable.comment) {
323+
console.log(`WARNING: No comment found for environment variable ${envVariable.name} in file ${envVariable.path}`);
324+
};
325+
let defaultValue = [];
326+
if (envVariable.defaultValue) {
327+
defaultValue = [
328+
{ type: 'text', value: ' (default: ', },
329+
{ type: 'inlineCode', value: envVariable.defaultValue },
330+
{ type: 'text', value: `): ${envVariable.comment}` }];
331+
} else {
332+
defaultValue = [{ type: 'text', value: `: ${envVariable.comment}` }];
333+
}
334+
envVariablesList.children.push({
335+
type: 'listItem',
336+
spread: false,
337+
checked: null,
338+
children: [{
339+
type: 'paragraph',
340+
children: [
341+
{
342+
type: 'strong', children: [
343+
{ type: 'text', value: envVariable.name },
344+
]
345+
},
346+
...defaultValue,
347+
],
348+
}],
349+
});
350+
}
351+
return [envVariablesTitle, envVariablesList];
352+
}

0 commit comments

Comments
 (0)