Uniformise scales and parameters in the Web API

[fpagnoux]

Connected to #677

In YAML parameters, scales are defined this way:

description: Social security contribution tax scale
brackets:
- rate:
    2013-01-01:
      value: 0.03
    2015-01-01:
      value: 0.04
    2017-01-01:
      value: 0.02
  threshold:
    2013-01-01:
      value: 0.0
- rate:
    2013-01-01:
      value: 0.1
    2015-01-01:
      value: 0.12
    2017-01-01:
      value: 0.06
  threshold:
    2013-01-01:
      value: 12000.0
    2014-01-01:
      value: 12100.0
    2015-01-01:
      value: 12200.0
    2016-01-01:
      value: 12300.0
    2017-01-01:
      value: 6000.0
- rate:
    2017-01-01:
      value: 0.12
  threshold:
    2017-01-01:
      value: 12400.0

In the web API, they are represented in a different way:

{
  "brackets": {
    "2013-01-01": {
      "0.0": 0.03, 
      "12000.0": 0.1
    }, 
    "2014-01-01": {
      "0.0": 0.03, 
      "12100.0": 0.1
    }, 
    "2015-01-01": {
      "0.0": 0.04, 
      "12200.0": 0.12
    }, 
    "2016-01-01": {
      "0.0": 0.04, 
      "12300.0": 0.12
    }, 
    "2017-01-01": {
      "0.0": 0.02, 
      "6000.0": 0.06, 
      "12400.0": 0.12
    }
  }, 
  "description": "Social security contribution tax scale", 
  "id": "taxes.social_security_contribution", 
  "metadata": {
    "rate_unit": "/1", 
    "threshold_unit": "currency-EUR"
  }, 
  "source": "https://github.com/openfisca/openfisca-country-template/blob/3.2.2rc0/openfisca_country_template/parameters/taxes/social_security_contribution.yaml"
}

The latter representation is "too compact" to expose the metadata defined in the YAML file.

[fpagnoux]

We could make the format evolve just enough to fit these metadata.

However, after spending some time working on parameters in different context (implementing fancy indexing, building a web client that renders table of parameters), I am convinced that scales is a legislative concept that is right now implemented way too deep in OpenFisca.

Some arguments:

• Each time we are doing some works on parameters, we often have to do it twice to handle scales.
• This doesn't scale (no pun intended): we handle 3 types of scales, but there may be other legitimate common calculation patterns to share. Should we keep complexify our parameter taxonomy?
• This makes P = tax_benefit_system.parameters(2018) a very inhomogeneous object:
  • P.taxes.housing_tax.rate is a float
  • P.taxes.housing_tax is basically a dict wrapper, which keys are rate and minimal_amount. Makes sense so far.
  • P.taxes.social_security_contribution is a MarginalRateTaxScale, a complex object that does very domain-specific calculations. This feels clumsy.

I feel like we should ideally move towards a different approach, already mentioned in #657 (comment):

• Keep parameters really simple. They are just (dated) data structured in a tree.
• On top of that, build reusable patterns. These patterns may be provided by Core, or built by users, but they should be a separate layer.

Now, this whole reflexion goes far beyond this issue, but I believe that if we want to go in that direction, and if we are to publish a breaking change on the scale route anyway, it's a good place to start.

[fpagnoux]

I'd therefore suggest to start considering scales as just parameter nodes with a specific structure.

Based on the #694 implementation, that would give us:

/parameter/taxes/social_security_contribution:

{
  "description": "Social security contribution tax scale", 
  "id": "taxes.social_security_contribution", 
  "subparams": {
    "brackets": {
      "description": null
    }
  }
}

/parameter/taxes/social_security_contribution/brackets:

{
  "description": null, 
  "id": "taxes.social_security_contribution.brackets", 
  "subparams": {
    "0": {
      "description": null
    }, 
    "1": {
      "description": null
    }, 
    "2": {
      "description": null
    }
  }
}

parameter/taxes/social_security_contribution/brackets/1:

{
  "description": null, 
  "id": "taxes.social_security_contribution.xbrackets.1", 
  "subparams": {
    "rate": {
      "description": null
    }, 
    "threshold": {
      "description": null
    }
  }
}

and finally parameter/taxes/social_security_contribution/brackets/1/rate:

{
  "description": null,
  "id": "taxes.social_security_contribution.xbrackets.1.rate",
  "values": {
    "2013-01-01": 0.1,
    "2015-01-01": 0.12,
    "2017-01-01": 0.06
  }
}

[fpagnoux]

An obvious issue is that a client who wants to get all the data about a scale would have to do at least two requests by bracket, which would be a pain. But that's easily avoidable by allowing a depth parameter to the query, to allow render nested parameter nodes.

parameter/taxes/social_security_contribution?depth=3:

{
  "description": "Social security contribution tax scale",
  "metadata": {
    "threshold_unit": "currency-EUR",
    "rate_unit": "/1"
  },
  "subparams": {
    "brackets": {
      "subparams": {
        "0": {
          "subparams": {
            "rate": {
              "2013-01-01": {
                "value": 0.03
              },
              "2015-01-01": {
                "value": 0.04
              },
              "2017-01-01": {
                "value": 0.02
              }
            },
            "threshold": {
              "2013-01-01": {
                "value": 0
              }
            }
          }
        },
        "1": {
          "subparams": {
            "rate": {
              "2013-01-01": {
                "value": 0.1
              },
              "2015-01-01": {
                "value": 0.12
              },
              "2017-01-01": {
                "value": 0.06
              }
            },
            "threshold": {
              "2013-01-01": {
                "value": 12000
              },
              "2014-01-01": {
                "value": 12100
              },
              "2015-01-01": {
                "value": 12200
              },
              "2016-01-01": {
                "value": 12300
              },
              "2017-01-01": {
                "value": 6000
              }
            }
          }
        },
        "2": {
          "subparams": {
            "rate": {
              "2017-01-01": {
                "value": 0.12
              }
            },
            "threshold": {
              "2017-01-01": {
                "value": 12400
              }
            }
          }
        }
      }
    }
  }
}

With that in place, a client exploring the legislation would only end up sending max 2 requests to get the whole scale:

• One to parameter/taxes/social_security_contribution that will let them know that this node represents a scale (either because a parameter node that only has one bracket subparam is a scale by convention, or thanks to a tag in metadata).
• One to parameter/taxes/social_security_contribution?depth=3 to get all the data

[fpagnoux]

@Anna-Livia @maukoquiroga @sandcha @MattiSG (and others), your input are very welcome both on the general principle exposed in my second comment, and in the strategy to start going towards it in the web API 🙂.

[MattiSG]

What are our use cases? I feel that any reuser will want to get all brackets, so asking for multiple requests there seems a bad constraint, just like adding a depth parameter that will almost always be needed to do anything meaningful with the API. Also, adding parameters opens a whole new set of complexity that I'd try to avoid as much as possible.

The initial post in this issue doesn't make it very clear which metadata can't be exposed exactly (I read that the sample JSON already contains more information than the sample YAML!). Could you please make that a bit more explicit? 🙂

---

I am concerned that this discussion would progress independently from #696 (comment), while I believe they should really be taken together as a global API design topic.

[bonjourmauko]

What are our use cases? I feel that any re-user will want to get all brackets, so asking for multiple requests there seems a bad constraint, just like adding a depth parameter that will almost always be needed to do anything meaningful with the API.

I feel DDD's entity and value-object concepts could help here: is there a use case where a user needs to interact with brackets in a parameter-independent way?

If the answer is yes, then brackets themselves should be queryable independently, like for the scenario _"I need to know the rates for the Social Security Contribution Tax Scale, between 2015 and 2017".

In that case, we could consider brackets as entities, having a value by themselves. Then a compound option like depth would make sense, as we're talking about different ressources.

Otherwise, if the answer is no, then the default behaviour should be to retrieve the entire node of the ressource, optionally providing a filter, like for the scenario _"I need to know the Social Security Contribution Tax Scale, between 2015 and 2017".

In that case, we could consider brackets as value-objects, having no value by themselves that is parameter-independent.

It all depends on actual use cases IMHO.

[MattiSG]

This issue has seen no discussion in 3 years, and has no documented real-life use case. I could see a link with #983, in that additional metadata could be exposed, but #983 aims at providing a generic way to expose arbitrary metadata.

Closing as stale.