Merge pull request #2 from AIML4OS/Austria

merge Austria chapter 1 - data to main

Author: ninanhm

chapters/chapter1.qmd

@@ -3,6 +3,7 @@ title: "Data"
 format: html # This qmd file will be compiled as an HTML web page
 ---
 
+
 # Data storage
 
 ## Germany
@@ -12,6 +13,10 @@ For programming and data processing, we use Cloudera Machine Learning (CML) with
 We store our data in the Parquet format, which is ideal for big data and in addition, to make it easier for users to handle and cross-check the data, we use Hue (Hadoop User Experience), an open-source SQL-based cloud editor.
 For rights management, we use Ranger, which provides a big variety of access control to ensure data security.   
 
+## Austria
+Training data is stored as csv-files. New files are added quarterly by the subject matter experts (between 300-500 data entries), which are then used as to retrain the model. 
+
+
 # Data cleaning
 
 ## Germany
@@ -21,11 +26,17 @@ First, data augmentation is performed by adding new text entries (e.g. text like
 Adding a variety of new textual entries helps the model to generelize better. 
 Secondly, we clean the data by removing punctation and handling missing values. 
 
+## Austria
+Duplicated entries are removed from the data. Text inputs are transformed into all lower-case letters. Further, we remove stop words, umlaut-charaters (ä,ö,ü), special characters (e.g. -,+,#,), gender-specific words endings (e.g. "-in", ":innen"), and numbers.
+Each categorical variable has a predefined set of valid input classes, since the model can only handle known classes. All known inputs are translated into this set of classes. Unknown inputs are set to their respective "unknown" category. 
+
+
 # Data versioning
 
 ## Germany 
 
 None
 
-
+## Austria 
+None