Feature/improvements confirmit forsta (#59)

* Add 'string_value' parameter
* Encode string values into numbers
* Upload data to forsta API & add tests
* Add venv folder to gitignore
* Change methods and variables names
* Install dotenv in pipeline
* Change replace by apply method
* Use loc instead of iloc in crosstab

Author: Pablincho

.github/workflows/python-package.yml

@@ -28,7 +28,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install pytest
+        pip install pytest python-dotenv
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
     - name: Test with pytest
       run: |

.gitignore

@@ -28,8 +28,9 @@ tests/Example Data (A).cache
 *.eot
 *.svg
 *.woff
-
+*.env
 *.js
+*-venv/
 
 docs/API/_static/
 docs/API/_templates/

docs/API/_build/_sources/index.rst.txt

@@ -35,7 +35,7 @@ and `numpy <http://www.numpy.org/>`_ libraries. Aimed at social and marketing re
 Key features
 """"""""""""
 
-* Reads plain .csv, converts from Dimensions, SPSS, Decipher, Ascribe and Confirmit
+* Reads plain .csv, converts from Dimensions, SPSS, Decipher, Ascribe and Forsta
 * Open metadata format to describe and manage datasets
 * Powerful, metadata-driven cleaning, editing, recoding and transformation of datasets
 * Computation and assessment of data weights

docs/API/_build/_sources/sites/lib_doc/dataprocessing/02_io.rst.txt

@@ -214,42 +214,42 @@ Confirmit
 
 **Reading (from files):**
 
->>> from quantipy.core.tools.dp.io import read_confirmit_from_files
->>> meta, data = read_confirmit_from_files(path_meta_json, path_data_json, verbose = default False)
+>>> from quantipy.core.tools.dp.io import read_forsta_from_files
+>>> meta, data = read_forsta_from_files(path_meta_json, path_data_json, verbose = default False)
 
-The verbose parameter will store all of the metadata from the Confirmit read
+The verbose parameter will store all of the metadata from the Forsta read
 within a custom element within the Quantipy metadata, this will impact performance
 so only set to true if you intend to write data back via the write method.
 
 **Reading (from API):**
 
->>> from quantipy.core.tools.dp.io import read_confirmit_api
->>> meta, data = read_confirmit_api(projectid, public_url, idp_url, client_id, client_secret, schema_vars, schema_filter, verbose)
+>>> from quantipy.core.tools.dp.io import read_forsta_api
+>>> meta, data = read_forsta_api(projectid, public_url, idp_url, client_id, client_secret, schema_vars, schema_filter, verbose)
 
 By default the verbose parameter will store all of the metadata
-from the Confirmit read within a custom element within the Quantipy metadata.
+from the Forsta read within a custom element within the Quantipy metadata.
 
 * projectid: survey id e.g. p12345
 * public_url: e.g. "https://ws.euro.confirmit.com/"
 * idp_url: e.g. "https://idp.euro.confirmit.com/"
-* client_id: Confirmit clientid
-* client_secret: Confirmit client secret
+* client_id: Forsta clientid
+* client_secret: Forsta client secret
 * schema_vars: variables you wish to download as comma delimited string
 * schema_filter: e.g."response:status='complete'"
 * verbose: default = False
 
-The verbose parameter will store all of the metadata from the Confirmit read
+The verbose parameter will store all of the metadata from the Forsta read
 within a custom element within the Quantipy metadata, this will impact performance
 so only set to true if you intend to write data back via the write method.
 
 **Writing:**
 
->>> from quantipy.core.tools.dp.io import write_confirmit_api
->>> write_confirmit_api(projectid, public_url, idp_url, client_id, client_secret, schema_vars)
+>>> from quantipy.core.tools.dp.io import write_forsta_api
+>>> write_forsta_api(projectid, public_url, idp_url, client_id, client_secret, schema_vars)
 
 * projectid: survey id e.g. p12345
 * public_url: e.g. "https://ws.euro.confirmit.com/"
 * idp_url: e.g. "https://idp.euro.confirmit.com/"
-* client_id: Confirmit clientid
-* client_secret: Confirmit client secret
+* client_id: Forsta clientid
+* client_secret: Forsta client secret
 * schema_vars: variables you wish to upload as comma delimited string

docs/API/_build/index.html

@@ -420,7 +420,7 @@ <h1>Quantipy: Python survey data toolkit<a class="headerlink" href="#quantipy-py
 <div class="section" id="key-features">
 <h2>Key features<a class="headerlink" href="#key-features" title="Permalink to this headline">¶</a></h2>
 <ul class="simple">
-<li><p>Reads plain .csv, converts from Dimensions, SPSS, Decipher, Ascribe and Confirmit</p></li>
+<li><p>Reads plain .csv, converts from Dimensions, SPSS, Decipher, Ascribe and Forsta</p></li>
 <li><p>Open metadata format to describe and manage datasets</p></li>
 <li><p>Powerful, metadata-driven cleaning, editing, recoding and transformation of datasets</p></li>
 <li><p>Computation and assessment of data weights</p></li>

docs/API/index.rst

@@ -35,7 +35,7 @@ and `numpy <http://www.numpy.org/>`_ libraries. Aimed at social and marketing re
 Key features
 """"""""""""
 
-* Reads plain .csv, converts from Dimensions, SPSS, Decipher, Ascribe and Confirmit
+* Reads plain .csv, converts from Dimensions, SPSS, Decipher, Ascribe and Forsta
 * Open metadata format to describe and manage datasets
 * Powerful, metadata-driven cleaning, editing, recoding and transformation of datasets
 * Computation and assessment of data weights

quantipy/core/dataset.py

@@ -10,9 +10,9 @@
     read_decipher as r_decipher,
     read_spss as r_spss,
     read_ascribe as r_ascribe,
-    read_confirmit_from_files as r_confirmit_from_files,
-    read_confirmit_api as r_confirmit_api,
-    write_confirmit_api as w_confirmit_api,
+    read_forsta_from_files as r_forsta_from_files,
+    read_forsta_api as r_forsta_api,
+    write_forsta_api as w_forsta_api,
     write_spss as w_spss,
     write_quantipy as w_quantipy,
     write_dimensions as w_dimensions)
@@ -585,8 +585,8 @@ def read_ascribe(self, path_meta, path_data, text_key):
         self._rename_blacklist_vars()
         return None
 
-    def read_confirmit_from_files(self, path_meta, path_data, reset=True, verbose=False):
-        """Read confirmit data
+    def read_forsta_from_files(self, path_meta, path_data, reset=True, verbose=False):
+        """Read forsta data
 
         Parameters
         ----------
@@ -601,11 +601,11 @@ def read_confirmit_from_files(self, path_meta, path_data, reset=True, verbose=Fa
         """
         if verbose:
             self.write_allowed = True
-        self._meta, self._data = r_confirmit_from_files(path_meta, path_data, verbose)
+        self._meta, self._data = r_forsta_from_files(self, path_meta, path_data, verbose)
         self._set_file_info(path_data, path_meta, reset=reset)
 
-    def read_confirmit_api(self, projectid, public_url, idp_url=None, client_id=None, client_secret=None, reset=True, schema_vars=None, schema_filter=None, verbose=False):
-        """Read confirmit data from confirmit api
+    def read_forsta_api(self, projectid, public_url, idp_url=None, client_id=None, client_secret=None, reset=True, schema_vars=None, schema_filter=None, verbose=False):
+        """Read forsta data from forsta api
 
         Parameters
         ----------
@@ -624,12 +624,13 @@ def read_confirmit_api(self, projectid, public_url, idp_url=None, client_id=None
             client_id = os.getenv('CLIENT_ID')
         if not client_secret:
             client_secret = os.getenv('CLIENT_SECRET')
-
-        self._meta, self._data = r_confirmit_api(projectid, public_url, idp_url, client_id, client_secret, schema_vars, schema_filter, verbose)
+        self._original_meta, self._original_data, self._meta, self._data = \
+            r_forsta_api(self, projectid, public_url, idp_url, client_id, \
+                            client_secret, schema_vars, schema_filter, verbose)
         self._set_file_info('', reset=reset)
 
-    def write_confirmit(self, path_meta, path_data, schema_vars=None, verbose=False):
-        """Converts quantipy dataset into Confirmit format"""
+    def write_forsta(self, path_meta, path_data, schema_vars=None, verbose=False):
+        """Converts quantipy dataset into forsta format"""
         try:
             if self.write_allowed:
                 res_meta_string = json.dumps(self._meta)
@@ -644,9 +645,9 @@ def write_confirmit(self, path_meta, path_data, schema_vars=None, verbose=False)
         except AttributeError:
             raise Exception("Must set has_external parameter in read method first")
 
-    def write_confirmit_api(self, projectid, public_url, idp_url, client_id, client_secret, schema_vars):
-        """Converts quantipy dataset into Confirmit format and uploads it to the confirmit API"""
-        return w_confirmit_api(projectid, public_url, idp_url, client_id, client_secret, schema_vars)
+    def write_forsta_api(self, projectid, public_url, idp_url, client_id, client_secret, schema_vars):
+        """Converts quantipy dataset into forsta format and uploads it to the forsta API"""
+        return w_forsta_api(self, projectid, public_url, idp_url, client_id, client_secret, schema_vars)
 
     def read_spss(self, path_sav, **kwargs):
         """
@@ -1931,7 +1932,7 @@ def _rounding(x, dec):
             idx = self.manifest_filter(f)
         else:
             idx = self.take(f)
-        data = self._data.copy().iloc[idx]
+        data = self._data.copy().loc[idx]
         stack = qp.Stack(name='ct', add_data={'ct': (data, self._meta)})
         if xtotal or not y:
             y = ['@'] + self.unroll(y)