MDE UI - issue flagging

[stufraser1]

Issues to address after new user was working with the tool:

• Ensure all mandatory fields are clearly marked as mandatory in UI with '*' - some fields not marked. – e.g. attribution (requires all three?) – name, address, role.

• Saving json file – give option to define filename - currently, user creating three hazard files in a row will save as same file named with number suffix.

• Flag more clearly the version mismatch if an uploaded file is not matching the version being used in the app - currnetly user receives a blank screen in middle data entry pane, and an unclear error message, but the file validation shows the file contents:

[matamadio]

See:

• [Proposal] best location for spatial and temporal attributes #360
• https://github.com/GFDRR/rdl-datapipeline/issues/28#issuecomment-3885159520

[bennyistanto]

• Saving json file – give option to define filename - currently, user creating three hazard files in a row will save as same file named with number suffix.

Related to json filename, in HDX script I used below rules for the integration of each (General Metadata + HEVL) components

Determine risk_data_type and Filename Prefix

Naming convention priority (highest → lowest):
  loss > vulnerability > exposure > hazard

Examples:
  H only              → rdls_hzd
  E only              → rdls_exp
  H + E               → rdls_exp   (exposure > hazard)
  H + L               → rdls_lss   (loss > hazard)
  E + L               → rdls_lss   (loss > exposure)
  E + V               → rdls_vln   (vulnerability > exposure)
  H + V               → rdls_vln   (vulnerability > hazard)
  V + L               → rdls_lss   (loss > vulnerability)
  H + E + V           → rdls_vln   (vulnerability > exposure > hazard)
  H + E + L           → rdls_lss   (loss > exposure > hazard)
  E + V + L           → rdls_lss   (loss > vulnerability > exposure)
  H + V + L           → rdls_lss   (loss > vulnerability > hazard)
  H + E + V + L       → rdls_lss   (loss > vulnerability > exposure > hazard)

Standalone constraint (configurable):
  V only (no H or E)  → INVALID — vulnerability needs hazard or exposure context
  L only (no H or E)  → INVALID — loss needs hazard or exposure context
  V + L (no H or E)   → INVALID — same reason
  Set REQUIRE_HE_FOR_VL = False to disable this constraint.

As this is inside JKAN repo, I would suggest move this (MDE) issue to #342

[matamadio]

Related to json filename, in HDX script I used below rules for the integration of each (General Metadata + HEVL) components

Determine risk_data_type and Filename Prefix

Naming convention priority (highest → lowest):
  loss > vulnerability > exposure > hazard

Examples:
  H only              → rdls_hzd
  E only              → rdls_exp
  H + E               → rdls_exp   (exposure > hazard)
  H + L               → rdls_lss   (loss > hazard)
  E + L               → rdls_lss   (loss > exposure)
  E + V               → rdls_vln   (vulnerability > exposure)
  H + V               → rdls_vln   (vulnerability > hazard)
  V + L               → rdls_lss   (loss > vulnerability)
  H + E + V           → rdls_vln   (vulnerability > exposure > hazard)
  H + E + L           → rdls_lss   (loss > exposure > hazard)
  E + V + L           → rdls_lss   (loss > vulnerability > exposure)
  H + V + L           → rdls_lss   (loss > vulnerability > hazard)
  H + E + V + L       → rdls_lss   (loss > vulnerability > exposure > hazard)

Standalone constraint (configurable):
  V only (no H or E)  → INVALID — vulnerability needs hazard or exposure context
  L only (no H or E)  → INVALID — loss needs hazard or exposure context
  V + L (no H or E)   → INVALID — same reason
  Set REQUIRE_HE_FOR_VL = False to disable this constraint.

As this is inside JKAN repo, I would suggest move this (MDE) issue to GFDRR/rdl-standard#342

Why not keeping all information about included components? I am proposing this:

rdls_{component1}.{component2}-title.json

Example:

rdls_hzd.exp-gfdrr_eth_climate.json

I choose . over + as safer character for parsers. Could be , as well.

[bennyistanto]

Why not keeping all information about included components? I am proposing this:

rdls_{component1}.{component2}-title.json

Example:

rdls_hzd.exp-gfdrr_eth_climate.json

I choose . over + as safer character for parsers. Could be , as well.

I don't mind using what you are proposing above.

Currently, the script still uses the initial naming convention from you generic_identifier-specific_identifier.json

Does the . and , are bad for filename? They cause parsing headaches.

I would prefer using:
A. Underscore. rdls_hzd_exp_vln_lss- as the general_identifier for complete metadata with HEVL components
B. We change hzd, exp, vln and lss with single letter h, e, v and l. Then the general_identifier will be:

• rdls_hevl-
• rdls_hvl-
• rdls_evl-
• rdls_he-
• ....

[matamadio]

Does the . and , are bad for filename? They cause parsing headaches.

I don't know for sure. They might ¯\_(ツ)_/¯

A. Underscore. rdls_hzd_exp_vln_lss- as the general_identifier for complete metadata with HEVL components
B. We change hzd, exp, vln and lss with single letter h, e, v and l:

Your solutions are also eligible. I like B better.

[bennyistanto]

Ok, should I change the script for option B? Or keep it as is for now

One more clarification, is my assumption correct for the below?

Standalone constraint (configurable):
  V only (no H or E)  → INVALID — vulnerability needs hazard or exposure context
  L only (no H or E)  → INVALID — loss needs hazard or exposure context
  V + L (no H or E)   → INVALID — same reason

[stufraser1]

On below, No - user should be able to record any single component without filling the other related components.
There is value in having them in one dataset but equally user can choose to link to another dataset entirely to provide the H and E into V, for example.

One more clarification, is my assumption correct for the below?

Standalone constraint (configurable):
  V only (no H or E)  → INVALID — vulnerability needs hazard or exposure context
  L only (no H or E)  → INVALID — loss needs hazard or exposure context
  V + L (no H or E)   → INVALID — same reason

[stufraser1]

Ok, should I change the script for option B? Or keep it as is for now

Option B will enable shorter filename and as long as we explain it in docs, then it can make sense.

Go ahead with option B for naming components.

What is the solution for the rest of the filename, e.g. parsing the dataset id or name, especially very long names?

[matamadio]

What is the solution for the rest of the filename, e.g. parsing the dataset id or name, especially very long names?

Updating GFDRR/rdl-jkan#114 with detailed approach for id and filenaming.

[bennyistanto]

On below, No - user should be able to record any single component without filling the other related components. There is value in having them in one dataset but equally user can choose to link to another dataset entirely to provide the H and E into V, for example.

One more clarification, is my assumption correct for the below?

Standalone constraint (configurable):
  V only (no H or E)  → INVALID — vulnerability needs hazard or exposure context
  L only (no H or E)  → INVALID — loss needs hazard or exposure context
  V + L (no H or E)   → INVALID — same reason

I just check the schema (line 2865), Loss only is not possible due to hazard_type is required. So technically only "V only" which is possible.

...
...
    "Losses": {
      "title": "Losses",
      "type": "object",
      "description": "Losses resulting from the occurrence of a hazard.",
      "required": [
        "id",
        "hazard_type",
        "asset_category",
        "asset_dimension",
        "impact_and_losses"
      ],
...
...

The schema says hazard_type is required in a loss entry. This makes sense, the schema describes "Losses resulting from the occurrence of a hazard." So every loss entry must specify which hazard caused the loss.

But this does NOT mean the dataset needs a separate hazard component block. It means:

• A rdls_lss (loss-only) record is valid as long as each loss entry has a hazard_type field filled (e.g., "flood", "earthquake")
• The dataset doesn't need a full hazard event_set / analysis, it just needs to state what type of hazard caused the reported losses
• Example: "This dataset reports flood damage costs" → hazard_type: "flood" in the loss entry, but no hazard block needed

Real Problem

The 4,950 skipped datasets have loss signals (displacement, casualties, economic damage) but Notebook 11 couldn't determine what hazard caused them. The filter correctly skips these because writing hazard_type: null would violate the schema.

@matamadio Does this issue also similar with what you posted in #306 (comment) ?

[matamadio]

The 4,950 skipped datasets have loss signals (displacement, casualties, economic damage) but Notebook 11 couldn't determine what hazard caused them. The filter correctly skips these because writing hazard_type: null would violate the schema.

Can we look at random examples? If those losses are not tied to one or more natural hazards but else, then they shouldn't be included.

[bennyistanto]

The 4,950 skipped datasets have loss signals (displacement, casualties, economic damage) but Notebook 11 couldn't determine what hazard caused them. The filter correctly skips these because writing hazard_type: null would violate the schema.

Can we look at random examples? If those losses are not tied to one or more natural hazards but else, then they shouldn't be included.

What Are These 4,950 Datasets?
The breakdown of loss signal types among the skipped records:

Signal Type	Count	% of 4,950
human_loss	3,040	61.4%
displacement	1,657	33.5%
affected_population	1,018	20.6%
structural_damage	32	0.6%
general_loss	16	0.3%

These 4,950 datasets are overwhelmingly NOT natural-hazard losses. They are:

• Humanitarian/refugee datasets (~45%), UNHCR, IOM, IDMC tracking displacement, internally displaced persons, refugee flows (no natural hazard trigger mentioned)
• Health & socioeconomic indicators (~35%), World Bank, WHO, UNESCO development indicators with mortality/morbidity statistics
• Conflict-related data (~10%), ACLED, armed violence impact tracking
• Miscellaneous development data, demographic, institutional datasets

Examples: "Uganda - Internal Displacements", "Somalia - Health", "AOAV 2011-2020 Explosive Violence Data", "Chile - Health Indicators"

The pipeline is working correctly, it detects loss-related keywords (casualties, displacement, structural damage) but appropriately excludes these records because they lack natural hazard context. Only 821 loss-detected datasets (14.2%) successfully matched to specific hazard types and were integrated into RDLS.

Here are 6 example JSON files, one per loss signal type:

• human_loss, Health indicators with mortality keywords
  • File: https://data.humdata.org/dataset/002cf4e5-2a6a-42d5-8079-b47c4fffef2d
  • Title: Korea, Dem. People's Rep. - Health
  • Org: World Bank Group
  • Trigger: "Mortality rate", "Death rate", "Number of infant deaths"
  • Why skipped: Development health statistics, mortality terms refer to demographic baselines, not hazard casualties
• displacement, Conflict-driven refugee survey
  • File: https://data.humdata.org/dataset/009b36b7-9983-4fe5-84c7-fcce283e118c
  • Title: South Sudan - Forced Displacement Survey, 2023
  • Org: UNHCR
  • Trigger: "Forced Displacement", "displaced", "refugee households"
  • Why skipped: Conflict-driven displacement (civil war), not natural hazard
• affected_population, Conflict humanitarian needs
  • File: https://data.humdata.org/dataset/019de7ad-8043-4755-b1fb-f84ce4b961a5
  • Title: Ukraine: Humanitarian Needs
  • Org: OCHA HPC Tools
  • Trigger: Tags "affected population", "people in need-pin"
  • Why skipped: War/conflict-affected populations, not natural disaster impacts
• structural_damage, Armed conflict infrastructure damage
  • File: https://data.humdata.org/dataset/31428563-4820-4725-96a0-1d1e6786fc8d
  • Title: Yemen Impact on Civilian Infrastructure
  • Org: Civilian Impact Monitoring Project
  • Trigger: "damage assessment", "Civilian Infrastructure damaged"
  • Why skipped: Airstrikes/shelling damage, not natural hazard damage
• general_loss, Multi-hazard inventory without specific hazard type
  • File: https://data.humdata.org/dataset/19ed73b5-9f91-4568-a0f3-48b7fa8c1796
  • Title: Disaster Loss Data for Mali
  • Org: UNDRR (DesInventar)
  • Trigger: "Disaster Loss", "Losses and Damages"
  • Why skipped: This one is genuinely hazard-related (DesInventar), but the metadata doesn't name a specific hazard type (flood, earthquake, etc.) — it's a multi-hazard loss inventory. The pipeline requires an explicit hazard_type match.
• catastrophe_model, Composite risk index, not actual losses
  • File: https://data.humdata.org/dataset/17582fbe-00b1-4077-b921-4890bfe525e5
  • Title: Kenya Children's Climate Risk Index-Disaster Risk Model (CCRI-DRM)
  • Org: INFORM / UNICEF
  • Trigger: "Disaster Risk Model", "risk assessment"
  • Why skipped: A risk index scoring framework, not actual observed or modeled losses