Validators ignored for fields with repeating subfields

[themowski]

Environment

CKAN version: 2.10.1
ckanext-scheming version: release-3.0.0

Description

Our CKAN implementation is being used to index datasets across a variety of domains. We have a schema that defines a core set of fields that all datasets need to specify. To support flexibility for dataset authors to add additional key-value metadata that is outside of the required core set of fields, our schema also has a field called custom_fields that uses repeating_subfields to allow users to specify additional key-value pairs to associate with their dataset. The key subfield has a validator that makes sure that the key's name meets certain requirements:

- field_name: custom_fields
  label: Custom Fields
  repeating_label: Custom Field
  repeating_subfields:
  - field_name: key
    label: Key
    validators: key_validator
    required: true
  - field_name: value
    label: Value
    required: true

We want to block users from specifying the same key value in multiple entries in custom_fields. The obvious way to do this is to define a validator function, keys_are_unique(), that can be applied to custom_fields to verify that the keys are unique.

- field_name: custom_fields
  label: Custom Fields
  validators: keys_are_unique
  repeating_label: Custom Field
  repeating_subfields:
  - field_name: key
    label: Key
    validators: key_validator
    required: true
  - field_name: value
    label: Value
    required: true

After trying this & spending some time in my debugger, I discovered that the extension initially saves the top-level field's validator, but then replaces it with the subfields' validator(s). When this happens, the data type of the validators list also changes from list[str] to dict. I haven't had time yet to dig into why/how this works in the grand scheme of CKAN dataset creation, but that's probably my next step.

One example of this behavior is in the _field_validators() function in ckanext/scheming/plugins.py:

ckanext-scheming/ckanext/scheming/plugins.py

Lines 582 to 608 in 8646a9d

	def _field_validators(f, schema, convert_extras):
	"""
	Return the validators for a scheming field f
	"""
	if 'validators' in f:
	validators = validation.validators_from_string(
	f['validators'],
	f,
	schema
	)
	elif helpers.scheming_field_required(f):
	validators = [not_empty]
	else:
	validators = [ignore_missing]
	if convert_extras:
	validators.append(convert_to_extras)
	# If this field contains children, we need a special validator to handle
	# them.
	if 'repeating_subfields' in f:
	validators = {
	sf['field_name']: _field_validators(sf, schema, False)
	for sf in f['repeating_subfields']
	}
	return validators

As a user, this behavior is unexpected -- I would expect that validators can be applied to both the top-level & sub-level fields. I think that it's worth updating the documentation for repeating_subfields and/or validators to explain that if a field has repeating_subfields, then validators for that field are ignored in favor of validators defined for the subfields.

I also have two specific questions, any insight would be appreciated:

1. We are trying to allow users to add an arbitrary number of "extra" key-value pairs to a dataset that is defined by a ckanext-scheming schema. Is our current approach (custom_fields field with repeating_subfields) the "correct" way to do what we are trying to do, or is there another method that is better supported, either by ckanext-scheming or by CKAN itself?

2. Assuming that what we are doing is correct, roughly how difficult would it be to modify ckanext-scheming to support validation of both the subfields and the top-level field? Any tips on what we would need to do to support that? It almost seems like this would require changes to the way the data is modelled, which I assume would be nontrivial.

[Skitionek]

Workaround:

@validator
@scheming_validator
def validate_repeating_subfields_before(field, schema):
    _check_schema_context(field)
    get_self_field_name = toolkit.h.get_self_field_name
    repeating_subfields_self_mapping = {}
    # Create schema validating only fields with 'repeating_subfields' property
    repeating_subfields_validation_schema = {}
    for field in schema['fields']:
        if 'repeating_subfields' in field:
            repeating_subfields_self_mapping[field['field_name']] = get_self_field_name(field)
            # caveat field validators will be called with key = get_get_self_field_name(field)
            repeating_subfields_validation_schema[get_self_field_name(field)] = field_validators(field, schema)
            # Cannot figure why subfields are not validated by scheming
            # Adding validation in here, in worst case will check twice
            nested_validator = repeating_subfields_validation_schema.setdefault(field['field_name'], {})
            for subfield in field['repeating_subfields']:
                nested_validator[subfield['field_name']] = field_validators(subfield, schema)

    def validator(key, data, errors, context):
        # By this point data is already in __extras
        extras = data.get(('__extras',), {})
        expand_form_composite(extras, repeating_subfields_self_mapping.keys())
        # Move from extras to normal scope
        for field_name, get_self_field_name in repeating_subfields_self_mapping.items():
            if field_name in extras:
                # Add as object for self validator
                data[(get_self_field_name,)] = extras[field_name]
                # As well as flatten version (allows build-in validation)
                for index, row in enumerate(extras[field_name]):
                    for subfield_name, subfield_value in row.items():
                        data[(field_name, index, subfield_name)] = subfield_value

                # deleted moved content
                del extras[field_name]

        # Allow passing through for all entities in data
        unflatten_data = unflatten(data)
        repeating_subfields_validation_schema_pass_data = {
            # Add empty validators for each existing data value
            **{key: [] for key in unflatten_data.keys()},
            **repeating_subfields_validation_schema
        }

        converted_data, field_errors = navl_validate(
            unflatten_data,
            repeating_subfields_validation_schema_pass_data,
            context
        )
        for get_self_field_name, field_error in field_errors.items():
            errors[(get_self_field_name,)] = field_error

    return validator

Meant to be added as:

before_validators: validate_repeating_subfields_before

This one uses additional key (defined with toolkit.h.get_self_field_name) to denote reference for root field.

Please be aware that you would need to modify templates as they do not support it out of the box.