Skip to content

sum results in concatenation rather than addition #166

@jbothma

Description

@jbothma

In order to submit an issue, please ensure you can check the following. Thanks!

  • Declare which version of Python you are using (python --version)
  • Declare which operating system you are using
tableschema-sql==0.10.4
tabulate==0.8.2
tabulator==1.19.0
dataflows==0.0.32
datapackage==1.5.1
datapackage-pipelines==2.0.0

when I run a pipeline including the following steps, excluding the string_to_int processor, I get a cast error at the end of the pipeline because the value 100000-15297008027871520 is not an integer. It looks like the values from the csv are strings, and I suspect the sum operation does += resulting in concatenation.

    - run: load
      parameters:
        from: 'http://datastore.openspending.org/b9d2af843f3a7ca223eea07fb608e62a/estimates-of-provincial-expenditure-2018-19-uploaded-2019-05-01t1328/final/data/epre-2018-19.csv'
      cache: true

      # because otherwise string "addition" happens which looks like
      # '7871520' + '9700802' + '-152' + '100000' = '100000-15297008027871520'
    - run: string_to_int
      parameters:
        column_name: 'value'

    - run: filter
      parameters:
        resources: epre-2018-19
        in:
          - finyear: '2018'

    - run: delete_fields
      parameters:
        resources: epre-2018-19
        fields:
          - voteno
          - econ5
          - sprogno
          - subprogramme

    - run: join
      parameters:
        source:
          name: epre-2018-19
          key:
            - "province"
            - "department"
            - "progno"
            - "programme"
            - "fg1"
            - "fg2"
            - "econ1"
            - "econ2"
            - "econ3"
            - "econ4"
            - "finyear"
            - "budget_phase"
          delete: yes
        target:
          name: epre-2018-19
          key: null
        fields:
          "province": {}
          "department": {}
          "progno": {}
          "programme": {}
          "fg1": {}
          "fg2": {}
          "econ1": {}
          "econ2": {}
          "econ3": {}
          "econ4": {}
          "finyear": {}
          "budget_phase": {}
          "value":
            aggregate: sum
        full: true
| ERROR   :ROW: {'Amount Kind': 'Total', 'Budget Phase': 'Main appropriation', 'Department': 'Education', 'Econ1': 'Capital', 'Econ2': 'Payments for capital assets', 'Econ3': 'Machinery and equipment', 'Econ4': 'Other machinery and equipment', 'Financial Year': 2018, 'FunctionGroup1': 'Learning and culture', 'FunctionGroup2': 'Basic education', 'Government': 'Eastern Cape', 'Programme': 'Public Ordinary School Education', 'Programme Number': 2, 'Value': '100000-15297008027871520'}

Screenshot_2019-05-01_15-02-15

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions