Rename error handling to error handler:113 (#114)

* Fixes #113 - Refactored ErrorHandling to ErrorHandler and ErrorHandlingFilteringErrorRows to ErrorHandlerFilteringErrorRows

* Fixes #113 - Rolled back some changes

* Fixes #113 - Refactored ErrorHandlingFilteringErrorRows to ErrorHandlerFilteringErrorRows

* Fixes #113 - Refactored ErrorHandlingIgnoringErrors to ErrorHandlerIgnoringErrors

* Fixes #113 - Refactored ErrorMessageArray to ErrorHandlerErrorMessageIntoArray

* Fixes #113 - Refactored ErrorMessageArray Object to ErrorHandlerErrorMessageIntoArray

* Fixes #113 - Renamed errorHandling package to errorHandler

* Fixes #113 - Renamed DataFrameErrorHandlingImplicit to DataFrameErrorHandlerImplicit

* Fixes #113

* Fixes #113 - fixed some documentation errors

* Update spark-commons/src/test/scala/za/co/absa/spark/commons/errorhandler/DataFrameErrorHandlerImplicitTest.scala

Co-authored-by: David Benedeki <[email protected]>

* Update spark-commons/src/test/scala/za/co/absa/spark/commons/errorhandler/implementations/ErrorHandlerFilteringErrorRowsTest.scala

Co-authored-by: David Benedeki <[email protected]>

* Fixes #113 - fixed some documentation typo and added more information on ErrorHandler library in the README.md

---------

Co-authored-by: David Benedeki <[email protected]>

Author: TebaleloS

README.md

@@ -104,7 +104,7 @@ _Json Utils_ provides methods for working with Json, both on input and output.
 
 _ColumnImplicits_ provide implicit methods for transforming Spark Columns
 
-1. Transforms the column into a booleaan column, checking if values are negative or positive infinity
+1. Transforms the column into a boolean column, checking if values are negative or positive infinity
 
     ```scala
       column.isInfinite()
@@ -424,10 +424,18 @@ path even of nested fields. It also evaluates arrays and maps where the array in
        def nul_coll(dataType: DataType): Column
    ```
 
-## Error Handling
+## Error Handler
 
-A `trait` and a set of supporting classes and other traits to enable errrors channeling between libraries and 
-application during Spark data processing. 
+A `trait` and a set of supporting classes and other traits to enable errors channeling between libraries and 
+application during Spark data processing.
+
+1. It has an [implicit dataFrame](https://github.com/AbsaOSS/spark-commons/blob/113-Rename-ErrorHandling-to-ErrorHandler/spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/DataFrameErrorHandlerImplicit.scala) for easier usage of the methods provided by the error handler trait.
+
+2. It provides four basic implementations
+   * [ErrorHandlerErrorMessageIntoArray](https://github.com/AbsaOSS/spark-commons/blob/113-Rename-ErrorHandling-to-ErrorHandler/spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/implementations/ErrorHandlerErrorMessageIntoArray.scala) - An implementation of error handler trait that collects errors into columns of struct based on [za.co.absa.spark.commons.errorhandler.ErrorMessage ErrorMessage] case class.
+   * [ErrorHandlerFilteringErrorRows](https://github.com/AbsaOSS/spark-commons/blob/113-Rename-ErrorHandling-to-ErrorHandler/spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/implementations/ErrorHandlerFilteringErrorRows.scala) - An implementation of error handler that implements the functionality of filtering rows that have some error (any of the error columns is not NULL).
+   * [ErrorHandlerIgnoringErrors](https://github.com/AbsaOSS/spark-commons/blob/113-Rename-ErrorHandling-to-ErrorHandler/spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/implementations/ErrorHandlerIgnoringErrors.scala) -  An implementation of error handler trait that ignores the errors detected during the dataFrame error aggregation
+   * [ErrorHandlerThrowingException](https://github.com/AbsaOSS/spark-commons/blob/113-Rename-ErrorHandling-to-ErrorHandler/spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/implementations/ErrorHandlerThrowingException.scala) - An implementation of error handler trait that throws an exception on error detected.
 
 ## Spark Commons Test
 

…ing/DataFrameErrorHandlingImplicit.scala …dler/DataFrameErrorHandlerImplicit.scalaspark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/DataFrameErrorHandlingImplicit.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/DataFrameErrorHandlerImplicit.scala spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/DataFrameErrorHandlingImplicit.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/DataFrameErrorHandlerImplicit.scala

@@ -14,17 +14,17 @@
  * limitations under the License.
  */
 
-package za.co.absa.spark.commons.errorhandling
+package za.co.absa.spark.commons.errorhandler
 
 import org.apache.spark.sql.{Column, DataFrame}
-import za.co.absa.spark.commons.errorhandling.types.{AdditionalInfo, ErrCode, ErrMsg, ErrSourceColName, ErrType, ErrorColumn, ErrorWhen}
+import za.co.absa.spark.commons.errorhandler.types.{AdditionalInfo, ErrCode, ErrMsg, ErrSourceColName, ErrType, ErrorColumn, ErrorWhen}
 import scala.language.implicitConversions
 
 /**
- * Class implement the functionality of implicit ErrorHandling trait to DataFrame. This implementation take ErrorHandling trait as an
- * implicit which will allow easier usage for ErrorHandling trait.
+ * Class implement the functionality of implicit ErrorHandler trait to DataFrame. This implementation take ErrorHandler trait as an
+ * implicit which will allow easier usage for ErrorHandler trait.
  */
-object DataFrameErrorHandlingImplicit {
+object DataFrameErrorHandlerImplicit {
   /**
    * This method implicitly convert an errorColumn to a normal Column
    *
@@ -35,33 +35,33 @@ object DataFrameErrorHandlingImplicit {
     errorColumn.column
   }
 
-  implicit class DataFrameEnhancedWithErrorHandling(val dataFrame: DataFrame) extends AnyVal {
+  implicit class DataFrameEnhancedWithErrorHandler(val dataFrame: DataFrame) extends AnyVal {
 
     /**
      * Applies the earlier collected [[types.ErrorColumn ErrorColumns]] to the provided [[org.apache.spark.sql.DataFrame spark.DataFrame]].
      *
-     * @param errCols   - a list of [[types.ErrorColumn]] returned by previous calls of [[ErrorHandling!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandling\.ErrorMessageSubmit)* createErrorAsColumn]]
+     * @param errCols   - a list of [[types.ErrorColumn]] returned by previous calls of [[ErrorHandler!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandler\.ErrorMessageSubmit)* createErrorAsColumn]]
      * @return - the original data frame with the error detection applied
-     * @group Error Handling
+     * @group Error Handler
      * @since 0.6.0
      */
-    def applyErrorColumnsToDataFrame(errCols: ErrorColumn*)(implicit errorHandling: ErrorHandling): DataFrame = {
-      errorHandling.applyErrorColumnsToDataFrame(dataFrame)(errCols: _*)
+    def applyErrorColumnsToDataFrame(errCols: ErrorColumn*)(implicit errorHandler: ErrorHandler): DataFrame = {
+      errorHandler.applyErrorColumnsToDataFrame(dataFrame)(errCols: _*)
     }
 
     /**
      * The idea of this function is: "Put the error specified to the provided dataframe if the condition is true on the row."
-     * The error is transformed to a column using the [[ErrorHandling.transformErrorSubmitToColumn]] method and applied to the data frame
-     * if the "when" condition is true using the [[ErrorHandling.doApplyErrorColumnsToDataFrame]] method.
+     * The error is transformed to a column using the [[ErrorHandler.transformErrorSubmitToColumn]] method and applied to the data frame
+     * if the "when" condition is true using the [[ErrorHandler.doApplyErrorColumnsToDataFrame]] method.
      *
      * @param when               - the condition that defines the error occurred on the row
      * @param errorMessageSubmit - the detected error specification
      * @return - the original [[org.apache.spark.sql.DataFrame spark.DataFrame]] with the error detection applied
-     * @group Error Handling
+     * @group Error Handler
      * @since 0.6.0
      */
-    def putError(when: Column)(errorMessageSubmit: ErrorMessageSubmit)(implicit errorHandling: ErrorHandling): DataFrame = {
-      errorHandling.putError(dataFrame)(when)(errorMessageSubmit)
+    def putError(when: Column)(errorMessageSubmit: ErrorMessageSubmit)(implicit errorHandler: ErrorHandler): DataFrame = {
+      errorHandler.putError(dataFrame)(when)(errorMessageSubmit)
     }
 
     /**
@@ -71,11 +71,11 @@ object DataFrameErrorHandlingImplicit {
      *
      * @param errorsWhen - the list of condition-error pairs, the condition are grouped by the field of the error submissions
      * @return - the original data frame with the error detection applied
-     * @group Error Handling
+     * @group Error Handler
      * @since 0.6.0
      */
-    def putErrorsWithGrouping(errorsWhen: Seq[ErrorWhen])(implicit errorHandling: ErrorHandling): DataFrame = {
-      errorHandling.putErrorsWithGrouping(dataFrame)(errorsWhen)
+    def putErrorsWithGrouping(errorsWhen: Seq[ErrorWhen])(implicit errorHandler: ErrorHandler): DataFrame = {
+      errorHandler.putErrorsWithGrouping(dataFrame)(errorsWhen)
     }
 
     /**
@@ -85,15 +85,15 @@ object DataFrameErrorHandlingImplicit {
      *
      * @param errorMessageSubmit - the error specification
      * @return - [[types.ErrorColumn]] expression containing the error specification
-     * @group Error Handling
+     * @group Error Handler
      * @since 0.6.0
      */
-    def createErrorAsColumn(errorMessageSubmit: ErrorMessageSubmit)(implicit errorHandling: ErrorHandling): ErrorColumn = {
-      errorHandling.createErrorAsColumn(errorMessageSubmit)
+    def createErrorAsColumn(errorMessageSubmit: ErrorMessageSubmit)(implicit errorHandler: ErrorHandler): ErrorColumn = {
+      errorHandler.createErrorAsColumn(errorMessageSubmit)
     }
 
     /**
-     * Same as the other [[ErrorHandling!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandling\.ErrorMessageSubmit)* createErrorAsColumn(errorMessageSubmit: ErrorMessageSubmit)]], only providing the error specification
+     * Same as the other [[ErrorHandler!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandler\.ErrorMessageSubmit)* createErrorAsColumn(errorMessageSubmit: ErrorMessageSubmit)]], only providing the error specification
      * in decomposed state, not in the [[ErrorMessageSubmit]] trait form.
      *
      * @param errType          - word description of the type of the error
@@ -102,12 +102,12 @@ object DataFrameErrorHandlingImplicit {
      * @param errSourceColName - the name of the column the error happened at
      * @param additionalInfo   - any optional additional info in JSON format
      * @return - [[types.ErrorColumn]] expression containing the error specification
-     * @group Error Handling
+     * @group Error Handler
      * @since 0.6.0
      */
     def createErrorAsColumn(errType: ErrType, errCode: ErrCode, errMessage: ErrMsg, errSourceColName: Option[ErrSourceColName], additionalInfo: AdditionalInfo = None)
-                           (implicit errorHandling: ErrorHandling): ErrorColumn = {
-      errorHandling.createErrorAsColumn(errType, errCode, errMessage, errSourceColName, additionalInfo)
+                           (implicit errorHandler: ErrorHandler): ErrorColumn = {
+      errorHandler.createErrorAsColumn(errType, errCode, errMessage, errSourceColName, additionalInfo)
     }
   }
 

…ommons/errorhandling/ErrorHandling.scala …/commons/errorhandler/ErrorHandler.scalaspark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/ErrorHandling.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/ErrorHandler.scala spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/ErrorHandling.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/ErrorHandler.scala

@@ -14,31 +14,31 @@
  * limitations under the License.
  */
 
-package za.co.absa.spark.commons.errorhandling
+package za.co.absa.spark.commons.errorhandler
 
 import org.apache.spark.sql.catalyst.expressions.{CaseWhen, Expression}
 import org.apache.spark.sql.functions.when
 import org.apache.spark.sql.types.DataType
 import org.apache.spark.sql.{Column, DataFrame}
-import za.co.absa.spark.commons.errorhandling.implementations.submits.{ErrorMessageSubmitOnColumn, ErrorMessageSubmitWithoutColumn}
-import za.co.absa.spark.commons.errorhandling.types._
+import za.co.absa.spark.commons.errorhandler.implementations.submits.{ErrorMessageSubmitOnColumn, ErrorMessageSubmitWithoutColumn}
+import za.co.absa.spark.commons.errorhandler.types._
 
 /**
- * The basic class of error handling component. Every library that wants to use the component during Spark data
+ * The basic class of error handler component. Every library that wants to use the component during Spark data
  * processing should utilize this trait and its methods. The methods serve to record the errors and attach them to the
  * [[org.apache.spark.sql.DataFrame spark.DataFrame]]. The trait should be an input parameter for such library, perhaps as an implicit.
- * On the other side the end application provides concrete [[ErrorHandling]] implementation, that does the actual error
+ * On the other side the end application provides concrete [[ErrorHandler]] implementation, that does the actual error
  * handling by the application desire.
  * For easy to use and as examples, a few general implementations are provided in the implementations sub-folder.
  * Also for common, repeated implementations the folder `partials` offer some traits.
  */
-trait ErrorHandling {
+trait ErrorHandler {
   /**
    * First of the few methods that needs to be coded in the trait implementation
    * The purpose of this method is to convert the error specification into a [[org.apache.spark.sql.Column spark.Column]] expression
    * @param errorMessageSubmit - the error specification
    * @return - the error specification transformed into a column expression
-   * @group Error Handling
+   * @group Error Handler
    * @since 0.6.0
    */
   protected def transformErrorSubmitToColumn(errorMessageSubmit: ErrorMessageSubmit): Column
@@ -65,7 +65,7 @@ trait ErrorHandling {
    * @param when - the condition that defines the error occurred on the row
    * @param errorMessageSubmit - the detected error specification
    * @return - the original [[org.apache.spark.sql.DataFrame spark.DataFrame]] with the error detection applied
-   * @group Error Handling
+   * @group Error Handler
    * @since 0.6.0
    */
   def putError(dataFrame: DataFrame)(when: Column)(errorMessageSubmit: ErrorMessageSubmit): DataFrame = {
@@ -79,7 +79,7 @@ trait ErrorHandling {
    * @param dataFrame - the [[org.apache.spark.sql.DataFrame spark.DataFrame]] to operate on
    * @param errorsWhen - the list of condition-error pairs, the condition are grouped by the field of the error submissions
    * @return - the original data frame with the error detection applied
-   * @group Error Handling
+   * @group Error Handler
    * @since 0.6.0
    */
   def putErrorsWithGrouping(dataFrame: DataFrame)(errorsWhen: Seq[ErrorWhen]): DataFrame = {
@@ -104,23 +104,24 @@ trait ErrorHandling {
    * The returned [[types.ErrorColumn]] should then be used in [[applyErrorColumnsToDataFrame]].
    * @param errorMessageSubmit - the error specification
    * @return - [[types.ErrorColumn]] expression containing the error specification
-   * @group Error Handling
+   * @group Error Handler
    * @since 0.6.0
    */
   def createErrorAsColumn(errorMessageSubmit: ErrorMessageSubmit): ErrorColumn = {
     ErrorColumn(transformErrorSubmitToColumn(errorMessageSubmit))
   }
 
   /**
-   * Same as the other [[ErrorHandling!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandling\.ErrorMessageSubmit)* createErrorAsColumn(errorMessageSubmit: ErrorMessageSubmit)]], only providing the error specification
+   * Same as the other [[ErrorHandler!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandler\.ErrorMessageSubmit)* createErrorAsColumn(errorMessageSubmit: ErrorMessageSubmit)]], only providing the error specification
    * in decomposed state, not in the [[ErrorMessageSubmit]] trait form.
+   *
    * @param errType - word description of the type of the error
    * @param errCode - number designation of the type of the error
    * @param errMessage - human friendly description of the error
    * @param errSourceColName - the name of the column the error happened at
    * @param additionalInfo - any optional additional info in JSON format
    * @return - [[types.ErrorColumn]] expression containing the error specification
-   * @group Error Handling
+   * @group Error Handler
    * @since 0.6.0
    */
   def createErrorAsColumn(errType: ErrType, errCode: ErrCode, errMessage: ErrMsg, errSourceColName: Option[ErrSourceColName], additionalInfo: AdditionalInfo = None): ErrorColumn = {
@@ -133,26 +134,29 @@ trait ErrorHandling {
   /**
    * Applies the earlier collected [[types.ErrorColumn ErrorColumns]] to the provided [[org.apache.spark.sql.DataFrame spark.DataFrame]].
    * See [[doApplyErrorColumnsToDataFrame]] for detailed functional explanation.
+   *
    * @param dataFrame - the [[org.apache.spark.sql.DataFrame spark.DataFrame]] to operate on
-   * @param errCols - a list of [[types.ErrorColumn]] returned by previous calls of [[ErrorHandling!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandling\.ErrorMessageSubmit)* createErrorAsColumn]]
+   * @param errCols   - a list of [[types.ErrorColumn]] returned by previous calls of [[ErrorHandler!.createErrorAsColumn(errorMessageSubmit:za\.co\.absa\.spark\.commons\.errorhandler\.ErrorMessageSubmit)* createErrorAsColumn]]
    * @return - the original data frame with the error detection applied
-   * @group Error Handling
+   * @group Error Handler
    * @since 0.6.0
    */
   def applyErrorColumnsToDataFrame(dataFrame: DataFrame)(errCols: ErrorColumn*): DataFrame = {
     doApplyErrorColumnsToDataFrame(dataFrame, errCols.map(_.column): _*)
   }
 
   /**
-   * Provides the library some information about how the actual implementation of [[ErrorHandling]] is structured.
+   * Provides the library some information about how the actual implementation of [[ErrorHandler]] is structured.
    * This function provides the information on the structure of single error column
+   *
    * @return -  the DataType of the column returned from `createErrorAsColumn` function
    */
   def errorColumnType: DataType
 
   /**
-   * Provides the library some information about how the actual implementation of [[ErrorHandling]] is structured.
+   * Provides the library some information about how the actual implementation of [[ErrorHandler]] is structured.
    * This function describes what is the type of the column attached (if it didn't exists before) to the [[org.apache.spark.sql.DataFrame DataFrame]]
+   *
    * @return - the DataType of the column containing the error info that is attached to the [[org.apache.spark.sql.DataFrame DataFrame]].
    */
   def dataFrameColumnType: Option[DataType]

…commons/errorhandling/ErrorMessage.scala …/commons/errorhandler/ErrorMessage.scalaspark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/ErrorMessage.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/ErrorMessage.scala spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/ErrorMessage.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/ErrorMessage.scala

@@ -14,9 +14,9 @@
  * limitations under the License.
  */
 
-package za.co.absa.spark.commons.errorhandling
+package za.co.absa.spark.commons.errorhandler
 
-import za.co.absa.spark.commons.errorhandling.types._
+import za.co.absa.spark.commons.errorhandler.types._
 
 /**
  * Case class to represent an error message

…s/errorhandling/ErrorMessageSubmit.scala …ns/errorhandler/ErrorMessageSubmit.scalaspark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/ErrorMessageSubmit.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/ErrorMessageSubmit.scala spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/ErrorMessageSubmit.scala renamed to spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandler/ErrorMessageSubmit.scala

@@ -14,9 +14,9 @@
  * limitations under the License.
  */
 
-package za.co.absa.spark.commons.errorhandling
+package za.co.absa.spark.commons.errorhandler
 
-import za.co.absa.spark.commons.errorhandling.types._
+import za.co.absa.spark.commons.errorhandler.types._
 
 /**
  * Trait collecting error definition in a format usable during Spark data processing