Helper functions to format trait data from external databases (e.g. TRY) for PEcAn.MA

[AritraDey-Dev]

I'd rephase this. The MA runs on tabular data in a specific format. One way to easily get data in that format is to query BETYdb, but you can also generate that format manually if you have other trait data. Indeed, a great new Issue for first time PEcAn developers would be create a helper function(s) that reformats trait data from common trait databases (e.g. TRY) into the tabular format this module is expecting. Given that no one is actively updating BETY, this approach is probably going to be the de facto norm for most users, and in the future there should be an update to this demo once we have functions that enable this

Originally posted by @mdietze in #3707 (comment)

Create helper functions to reformat trait data from common external databases (like TRY) into the tabular format expected by the meta-analysis module. This would allow users to easily use data sources other than BETYdb.

[mdietze]

I'll add that #3601 provides more info about how the flat files need to be formatted for the MA. A good test case would be the crop C and N trait data that @mattykim06 has been working with for PR #3683, as there's a very real need to set new SIPNET N cycle parameters

[Mayanknishad9]

Hi @mdietze @AritraDey-Dev, I'm interested in working on this TRY database helper function.

From the description, I understand we need:

1. A function to convert TRY database format to PEcAn MA tabular format
2. Standard trait name mappings
3. Documentation and examples

Before I start implementing, could you clarify:

• Are there any existing examples of the expected PEcAn MA format?
• Should the function handle specific edge cases in TRY data (ranges, qualifiers)?
• Are there any existing tests for data formatting I should follow?

Can you assign this issue to me?

[AritraDey-Dev]

Are there any existing examples of the expected PEcAn MA format?

pecan/base/db/R/query.data.R

Lines 25 to 37 in 09ea6c9

	query <- paste("select
	traits.id, traits.citation_id, traits.site_id, traits.treatment_id,
	treatments.name, traits.date, traits.time, traits.cultivar_id, traits.specie_id,
	traits.mean, traits.statname, traits.stat, traits.n, variables.name as vname,
	extract(month from traits.date) as month,",
	extra.columns,
	"treatments.control, sites.greenhouse
	from traits
	left join treatments on (traits.treatment_id = treatments.id)
	left join sites on (traits.site_id = sites.id)
	left join variables on (traits.variable_id = variables.id)
	where ", id_type, " in (", spstr,")
	and variables.name in ('", trait,"');", sep = "")

This shows the exact SQL columns returned from BETYdb.You can also refer to this test also for an example of the expected data format.

Should the function handle specific edge cases in TRY data (ranges, qualifiers)?

yes,see this https://github.com/PecanProject/pecan/blob/develop/modules/meta.analysis/R/jagify.R#L46-L55, https://github.com/PecanProject/pecan/blob/develop/modules/meta.analysis/R/jagify.R#L70-L88 for example.

Are there any existing tests for data formatting I should follow?

You can refer to the this files test.jagify.R, test.query.data.R

[Mayanknishad9]

Done Started working on it.

[Mayanknishad9]

@AritraDey-Dev
I've created the helper function format_try_for_ma() to convert TRY database data into PEcAn's meta-analysis format.

The function will be located at:
pecan/modules/data.remote/R/format_try.R

Function overview:

Maps TRY columns (StdValue, TraitName, Replicates, ErrorRisk) to PEcAn MA format

Handles basic column renaming and structure

Includes parameters for trait name mapping and citation ID

Usage example:

try_data <- data.frame(
TraitName = c("Leaf nitrogen content per leaf dry mass"),
StdValue = c(2.5),
ErrorRisk = c(0.5),
Replicates = c(10)
)

trait_map <- c("Leaf nitrogen content per leaf dry mass" = "leaf_N_concentration")
result <- format_try_for_ma(try_data, trait_map, citation_id = 999)

Next steps I need help with:

Could you provide a sample of actual TRY database export data to test with real values?
How should I handle species mapping from TRY's AccSpeciesName to PEcAn's specie_id?
Are there specific edge cases in TRY data (like ranges, qualifiers) I should prioritize handling?

I'm ready to refine this based on feedback and add more functionality once I can test with real TRY data.