Update readme, allow to configure to log JSON or string format

Author: minhduc140583

README.md

@@ -1,8 +1,117 @@
 # middleware
-middleware log for
-- http
-- echo
-- gin
+Middleware logging is a technique used in software development, particularly in web and microservices applications, to log important information about incoming requests, outgoing responses, and the operations performed by the application.
+- [core-go/middleware](https://github.com/core-go/middleware) is designed to integrate with middleware logging seamlessly for existing Go libraries: [Echo](https://github.com/labstack/echo), [Gin](https://github.com/gin-gonic/gin), or net/http ([Gorilla mux](https://github.com/gorilla/mux), [Go-chi](https://github.com/go-chi/chi)), with any logging libraries ([zap](https://pkg.go.dev/go.uber.org/zap), [logrus](https://github.com/sirupsen/logrus)), to log request headers, request body, response status code, body content, response time, and size
+- Especially, [core-go/middleware](https://github.com/core-go/middleware) supported to encrypt sensitive data, which is useful for Financial Transactions (to comply with <b>PCI-DSS</b> standards) and Healthcare (to comply with <b>HIPAA</b> regulations)
+
+### A typical micro service
+- When you zoom one micro service, the flow is as below, and you can see "middleware" in the full picture:
+  ![A typical micro service](https://cdn-images-1.medium.com/max/800/1*d9kyekAbQYBxH-C6w38XZQ.png)
+
+## Content for logging
+### Request
+#### Features
+- <b>Log Request Method and URL</b>: Log the HTTP method (GET, POST, etc.) and the requested URL.
+- <b>Log Request Headers</b>: Option to log request headers for debugging purposes.
+- <b>Log Request Body</b>: Option to log the request body (with configurable size limits to avoid logging large payloads).
+#### Benefits
+- <b>Debugging</b>: Helps in tracing and debugging issues by providing complete information about incoming requests.
+- <b>Monitoring</b>: Provides visibility into the types of requests being received.
+
+### Response
+#### Features
+- <b>Log Response Status Code</b>: Log the HTTP status code of the response.
+- <b>Log Response Headers</b>: Option to log response headers.
+- <b>Log Response Body</b>: Option to log the response body (with configurable size limits to avoid logging large payloads).
+#### Benefits
+- <b>Debugging</b>: Assists in diagnosing issues by providing complete information about the responses sent by the server.
+- <b>Auditing</b>: Helps in auditing and reviewing server responses for compliance and monitoring purposes.
+
+### Response Time
+#### Features
+- <b>Log Response Time</b>: Calculate and log the time taken to process each request.
+#### Benefits
+- <b>Performance Monitoring</b>: Helps in identifying slow requests and performance bottlenecks.
+- <b>Optimization</b>: Provides data to optimize and improve server response times.
+
+### Response Size
+#### Features
+- <b>Log Response Size</b>: Log the size of the response payload in bytes.
+#### Benefits
+- <b>Bandwidth Monitoring</b>: Helps in monitoring and managing bandwidth usage.
+- <b>Optimization</b>: Provides insights into the response sizes to optimize payloads and improve performance.
+
+## Features
+### Middleware Integration
+#### Features
+- <b>Middleware Function</b>: Designed to integrate seamlessly with existing Go libraries: [Echo](https://github.com/labstack/echo), [Gin](https://github.com/gin-gonic/gin), or net/http ([Gorilla mux](https://github.com/gorilla/mux), [Go-chi](https://github.com/go-chi/chi)).
+  - Sample for [Echo](https://github.com/labstack/echo) is at [go-sql-echo-sample](https://github.com/go-tutorials/go-sql-echo-sample)
+  - Sample for [Gin](https://github.com/gin-gonic/gin) is at [go-sql-gin-sample](https://github.com/go-tutorials/go-sql-gin-sample)
+  - Sample for [Gorilla mux](https://github.com/gorilla/mux) is at [go-sql-sample](https://github.com/go-tutorials/go-sql-sample) 
+- <b>Context Handling</b>: Pass context to handle request-specific data throughout the middleware chain.
+#### Benefits
+- <b>Ease of Use</b>: Simplifies the integration of logging into existing web applications.
+- <b>Consistency</b>: Ensures consistent logging across different parts of the application.
+
+### Logging Libraries Integration
+- Do not depend on any logging libraries.
+- Already supported to integrate with [zap](https://pkg.go.dev/go.uber.org/zap), [logrus](https://github.com/sirupsen/logrus)
+- Can be integrated with any logging library.
+
+### Sensitive Data Encryption
+#### Features
+- Mask/Encrypt sensitive data in the request and response bodies.
+- Sensitive Data Identification: identify and encrypt specific fields in JSON payloads.
+
+#### Benefits:
+- <b>Security</b>: Protects sensitive information from being exposed in logs.
+- <b>Compliance</b>: Helps meet security and compliance requirements by safeguarding sensitive data.
+- <b>Ease of Use</b>: Simplifies the integration of encryption/masking into any existing applications.
+- <b>Consistency</b>: Ensures that sensitive data is consistently encrypted or masked across all logged requests and responses
+
+#### Samples:
+- Sample for [Echo](https://github.com/labstack/echo) is at [go-sql-echo-sample](https://github.com/go-tutorials/go-sql-echo-sample)
+- Sample for [Gin](https://github.com/gin-gonic/gin) is at [go-sql-gin-sample](https://github.com/go-tutorials/go-sql-gin-sample)
+- Sample for [Gorilla mux](https://github.com/gorilla/mux) is at [go-sql-sample](https://github.com/go-tutorials/go-sql-sample)
+
+### Enable/Disable Logging
+#### Features
+- <b>Enable/Disable Logging</b>: Allow users to turn on or off logging for requests, responses, headers, and bodies independently.
+- <b>Logging Levels</b>: Support different logging levels (e.g., INFO, DEBUG, ERROR) to control the verbosity of logs.
+#### Benefits
+- <b>Flexibility</b>: Provides users with the flexibility to configure logging based on their needs and environment.
+- <b>Efficiency</b>: Reduces overhead by allowing selective logging, especially in production environments.
+
+### Asynchronous Logging
+#### Features
+- <b>Non-Blocking Logs</b>: Implement asynchronous logging to ensure that logging does not block request processing.
+- <b>Log Buffering</b>: Use buffering to improve logging performance and reduce latency.
+#### Benefits:
+- <b>Performance</b>: Improves the overall performance of the application by reducing logging overhead.
+- <b>Scalability</b>: Allows the application to handle high-throughput logging without performance degradation.
+
+
+## Use Cases of Sensitive Data Encryption
+### Financial Transactions
+- <b>Benefit</b>: Encrypting sensitive financial data, such as credit card numbers and transaction details, helps comply with PCI-DSS standards and secures financial transactions from exposure in logs.
+### Healthcare
+- <b>Benefit</b>: Encrypting patient data such as medical records and health information in logs ensures compliance with HIPAA regulations and protects patient privacy.
+### E-commerce
+- <b>Benefit</b>: Protecting customer information, such as addresses and payment details, enhances customer trust and protects the e-commerce platform from potential data breaches.
+
+## Benefits of Middleware Logging
+#### Debugging and Troubleshooting
+- Provides detailed logs that help developers debug and troubleshoot issues in the application by tracing the flow of requests and responses.
+#### Monitoring and Alerting
+- Enables monitoring of application performance and behavior, allowing for real-time alerting on errors, slow responses, and unusual activity.
+#### Performance Optimization
+- Logs performance metrics that can be analyzed to identify bottlenecks, optimize resource usage, and improve overall application performance.
+#### Security and Compliance
+- Helps in tracking access and usage patterns, detecting security incidents, and complying with regulatory requirements by logging relevant information.
+#### Auditing
+- Provides an audit trail of user actions and system operations, which is essential for security audits and forensic analysis.
+
+## Conclusion
+Middleware logging is a critical aspect of building robust, maintainable, and secure applications, providing valuable insights and aiding in the continuous improvement of the software.
 
 ## Installation
 Please make sure to initialize a Go module before installing core-go/middleware:
@@ -16,17 +125,7 @@ Import:
 import "github.com/core-go/middleware"
 ```
 
-## Features
-### log tracing at middleware
-#### Support to turn on, turn off
-- request
-- response
-- duration
-- http response status code
-- response content length
-#### Support to mask or encrypt fields
-- support to mask or encrypt fields, such as mobileNumber, creditCardNumber
-
+## Appendix
 ### Microservice Architect
 ![Microservice Architect](https://cdn-images-1.medium.com/max/800/1*vKeePO_UC73i7tfymSmYNA.png)
 

echo/config.go

@@ -3,6 +3,7 @@ package echo
 type LogConfig struct {
 	Separate       bool              `yaml:"separate" mapstructure:"separate" json:"separate,omitempty" gorm:"column:separate" bson:"separate,omitempty" dynamodbav:"separate,omitempty" firestore:"separate,omitempty"`
 	Build          bool              `yaml:"build" mapstructure:"build" json:"build,omitempty" gorm:"column:build" bson:"build,omitempty" dynamodbav:"build,omitempty" firestore:"build,omitempty"`
+	StringFormat   bool              `yaml:"string_format" mapstructure:"string_format" json:"stringFormat,omitempty" gorm:"column:string_format" bson:"stringFormat,omitempty" dynamodbav:"stringFormat,omitempty" firestore:"stringFormat,omitempty"`
 	Log            bool              `yaml:"log" mapstructure:"log" json:"log,omitempty" gorm:"column:log" bson:"log,omitempty" dynamodbav:"log,omitempty" firestore:"log,omitempty"`
 	Skips          string            `yaml:"skips" mapstructure:"skips" json:"skips,omitempty" gorm:"column:skips" bson:"skips,omitempty" dynamodbav:"skips,omitempty" firestore:"skips,omitempty"`
 	Ip             string            `yaml:"ip" mapstructure:"ip" json:"ip,omitempty" gorm:"column:ip" bson:"ip,omitempty" dynamodbav:"ip,omitempty" firestore:"ip,omitempty"`

echo/mask_logger.go

@@ -14,52 +14,57 @@ type MaskLogger struct {
 	RequestKey   string
 	MaskRequest  func(map[string]interface{})
 	MaskResponse func(map[string]interface{})
+	StringFormat bool
 }
 
-func NewMaskLogger(requestKey string, maskRequest func(map[string]interface{}), maskResponse func(map[string]interface{})) *MaskLogger {
-	return &MaskLogger{RequestKey: requestKey, MaskRequest: maskRequest, MaskResponse: maskResponse}
+func NewMaskLogger(requestKey string, maskRequest func(map[string]interface{}), maskResponse func(map[string]interface{}), stringFormat bool) *MaskLogger {
+	return &MaskLogger{RequestKey: requestKey, MaskRequest: maskRequest, MaskResponse: maskResponse, StringFormat: stringFormat}
 }
-func NewMaskLoggerWithSending(requestKey string, maskRequest func(map[string]interface{}), maskResponse func(map[string]interface{}), send func(context.Context, []byte, map[string]string) error, options ...map[string]string) *MaskLogger {
+func NewMaskLoggerWithSending(requestKey string, maskRequest func(map[string]interface{}), maskResponse func(map[string]interface{}), stringFormat bool, send func(context.Context, []byte, map[string]string) error, options ...map[string]string) *MaskLogger {
 	var keyMap map[string]string
 	if len(options) >= 1 {
 		keyMap = options[0]
 	}
-	return &MaskLogger{RequestKey: requestKey, MaskRequest: maskRequest, MaskResponse: maskResponse, send: send, KeyMap: keyMap}
+	return &MaskLogger{RequestKey: requestKey, MaskRequest: maskRequest, MaskResponse: maskResponse, StringFormat: stringFormat, send: send, KeyMap: keyMap}
 }
 func (l *MaskLogger) LogResponse(log func(context.Context, string, map[string]interface{}), r *http.Request, ww WrapResponseWriter,
 	c LogConfig, t1 time.Time, response string, fields map[string]interface{}, includeRequest bool) {
 	if includeRequest && len(c.Request) > 0 {
-		MaskRequest(c.Request, fields, l.MaskRequest)
+		MaskRequest(c.Request, fields, l.MaskRequest, l.StringFormat)
 	}
-	MaskResponse(ww, c, t1, response, fields, l.MaskResponse)
+	MaskResponse(ww, c, t1, response, fields, l.MaskResponse, l.StringFormat)
 	msg := r.Method + " " + r.RequestURI
 	log(r.Context(), msg, fields)
 	if l.send != nil {
 		go Send(r.Context(), l.send, msg, fields, l.KeyMap)
 	}
 }
 func (l *MaskLogger) LogRequest(log func(context.Context, string, map[string]interface{}), r *http.Request, fields map[string]interface{}) {
-	MaskRequest(l.RequestKey, fields, l.MaskRequest)
+	MaskRequest(l.RequestKey, fields, l.MaskRequest, l.StringFormat)
 	msg := "Request " + r.Method + " " + r.RequestURI
 	log(r.Context(), msg, fields)
 	if l.send != nil {
 		go Send(r.Context(), l.send, msg, fields, l.KeyMap)
 	}
 }
 
-func MaskResponse(ww WrapResponseWriter, c LogConfig, t1 time.Time, response string, fields map[string]interface{}, mask func(map[string]interface{})) {
+func MaskResponse(ww WrapResponseWriter, c LogConfig, t1 time.Time, response string, fields map[string]interface{}, mask func(map[string]interface{}), isStringFormat bool) {
 	if len(c.Response) > 0 {
 		fields[c.Response] = response
 		responseBody := response
 		responseMap := map[string]interface{}{}
 		json.Unmarshal([]byte(responseBody), &responseMap)
 		if len(responseMap) > 0 {
 			mask(responseMap)
-			responseString, err := json.Marshal(responseMap)
-			if err != nil {
-				fmt.Printf("Error: %s", err.Error())
+			if isStringFormat {
+				responseString, err := json.Marshal(responseMap)
+				if err != nil {
+					fmt.Printf("Error: %s", err.Error())
+				} else {
+					fields[c.Response] = string(responseString)
+				}
 			} else {
-				fields[c.Response] = string(responseString)
+				fields[c.Response] = responseMap
 			}
 		}
 	}
@@ -75,7 +80,7 @@ func MaskResponse(ww WrapResponseWriter, c LogConfig, t1 time.Time, response str
 		fields[c.Size] = ww.BytesWritten()
 	}
 }
-func MaskRequest(request string, fields map[string]interface{}, mask func(map[string]interface{})) {
+func MaskRequest(request string, fields map[string]interface{}, mask func(map[string]interface{}), isStringFormat bool) {
 	if len(request) > 0 {
 		req, ok := fields[request]
 		if ok {
@@ -85,11 +90,15 @@ func MaskRequest(request string, fields map[string]interface{}, mask func(map[st
 				json.Unmarshal([]byte(requestBody), &requestMap)
 				if len(requestMap) > 0 {
 					mask(requestMap)
-					requestString, err := json.Marshal(requestMap)
-					if err != nil {
-						fmt.Printf("Error: %s", err.Error())
+					if isStringFormat {
+						requestString, err := json.Marshal(requestMap)
+						if err != nil {
+							fmt.Printf("Error: %s", err.Error())
+						} else {
+							fields[request] = string(requestString)
+						}
 					} else {
-						fields[request] = string(requestString)
+						fields[request] = requestMap
 					}
 				}
 			}

echo/structured_logger.go

@@ -14,25 +14,30 @@ type Formatter interface {
 	LogResponse(log func(context.Context, string, map[string]interface{}), r *http.Request, ww WrapResponseWriter, c LogConfig, startTime time.Time, response string, fields map[string]interface{}, includeRequest bool)
 }
 type StructuredLogger struct {
-	send   func(context.Context, []byte, map[string]string) error
-	KeyMap map[string]string
+	send         func(context.Context, []byte, map[string]string) error
+	KeyMap       map[string]string
+	RequestKey   string
+	StringFormat bool
 }
 
 var fieldConfig FieldConfig
 
 func NewLogger() *StructuredLogger {
 	return &StructuredLogger{}
 }
-func NewLoggerWithSending(send func(context.Context, []byte, map[string]string) error, options ...map[string]string) *StructuredLogger {
+func NewLoggerWithStringFormat(requestKey string, stringFormat bool) *StructuredLogger {
+	return &StructuredLogger{RequestKey: requestKey, StringFormat: stringFormat}
+}
+func NewLoggerWithSending(requestKey string, stringFormat bool, send func(context.Context, []byte, map[string]string) error, options ...map[string]string) *StructuredLogger {
 	var keyMap map[string]string
 	if len(options) >= 1 {
 		keyMap = options[0]
 	}
-	return &StructuredLogger{send: send, KeyMap: keyMap}
+	return &StructuredLogger{RequestKey: requestKey, StringFormat: stringFormat, send: send, KeyMap: keyMap}
 }
 func (l *StructuredLogger) LogResponse(log func(context.Context, string, map[string]interface{}), r *http.Request, ww WrapResponseWriter,
-	c LogConfig, t1 time.Time, response string, fields map[string]interface{}, singleLog bool) {
-	BuildResponse(ww, c, t1, response, fields)
+	c LogConfig, t1 time.Time, response string, fields map[string]interface{}, includeRequest bool) {
+	BuildResponse(ww, c, t1, response, fields, l.StringFormat)
 	msg := r.Method + " " + r.RequestURI
 	log(r.Context(), msg, fields)
 	if l.send != nil {
@@ -48,15 +53,52 @@ func Send(ctx context.Context, send func(context.Context, []byte, map[string]str
 }
 func (l *StructuredLogger) LogRequest(log func(context.Context, string, map[string]interface{}), r *http.Request, fields map[string]interface{}) {
 	msg := "Request " + r.Method + " " + r.RequestURI
+	if !l.StringFormat && len(l.RequestKey) > 0 {
+		req, ok := fields[l.RequestKey]
+		if ok {
+			requestBody, ok2 := req.(string)
+			if ok2 {
+				requestMap := map[string]interface{}{}
+				json.Unmarshal([]byte(requestBody), &requestMap)
+				if len(requestMap) > 0 {
+					fields[l.RequestKey] = requestMap
+				}
+			}
+		}
+	}
 	log(r.Context(), msg, fields)
 	if l.send != nil {
 		go Send(r.Context(), l.send, msg, fields, l.KeyMap)
 	}
 }
 
-func BuildResponse(ww WrapResponseWriter, c LogConfig, t1 time.Time, response string, fields map[string]interface{}) {
+func BuildResponse(ww WrapResponseWriter, c LogConfig, t1 time.Time, response string, fields map[string]interface{}, isStringFormat bool) {
 	if len(c.Response) > 0 {
-		fields[c.Response] = response
+		if isStringFormat {
+			fields[c.Response] = response
+		} else {
+			responseBody := response
+			responseMap := map[string]interface{}{}
+			json.Unmarshal([]byte(responseBody), &responseMap)
+			if len(responseMap) > 0 {
+				fields[c.Response] = responseMap
+			} else {
+				fields[c.Response] = response
+			}
+		}
+	}
+	if !isStringFormat && len(c.Request) > 0 {
+		req, ok := fields[c.Request]
+		if ok {
+			requestBody, ok2 := req.(string)
+			if ok2 {
+				requestMap := map[string]interface{}{}
+				json.Unmarshal([]byte(requestBody), &requestMap)
+				if len(requestMap) > 0 {
+					fields[c.Request] = requestMap
+				}
+			}
+		}
 	}
 	if len(c.ResponseStatus) > 0 {
 		fields[c.ResponseStatus] = ww.Status()

echo/v3/config.go

@@ -3,6 +3,7 @@ package echo
 type LogConfig struct {
 	Separate       bool              `yaml:"separate" mapstructure:"separate" json:"separate,omitempty" gorm:"column:separate" bson:"separate,omitempty" dynamodbav:"separate,omitempty" firestore:"separate,omitempty"`
 	Build          bool              `yaml:"build" mapstructure:"build" json:"build,omitempty" gorm:"column:build" bson:"build,omitempty" dynamodbav:"build,omitempty" firestore:"build,omitempty"`
+	StringFormat   bool              `yaml:"string_format" mapstructure:"string_format" json:"stringFormat,omitempty" gorm:"column:string_format" bson:"stringFormat,omitempty" dynamodbav:"stringFormat,omitempty" firestore:"stringFormat,omitempty"`
 	Log            bool              `yaml:"log" mapstructure:"log" json:"log,omitempty" gorm:"column:log" bson:"log,omitempty" dynamodbav:"log,omitempty" firestore:"log,omitempty"`
 	Skips          string            `yaml:"skips" mapstructure:"skips" json:"skips,omitempty" gorm:"column:skips" bson:"skips,omitempty" dynamodbav:"skips,omitempty" firestore:"skips,omitempty"`
 	Ip             string            `yaml:"ip" mapstructure:"ip" json:"ip,omitempty" gorm:"column:ip" bson:"ip,omitempty" dynamodbav:"ip,omitempty" firestore:"ip,omitempty"`