No way to get specimen information from a media query

[mjcollin]

When running idig_search_media(), an rq can be passed but you can't get specimen fields back since media and specimens are n to n. Also, in the data.frame format, the media field indexTerms.records is a list so it is dropped by idig_field_indexes(). So how do you get specimen data about a media record?

[richiehodel]

To follow up on this, is there any way to merge/subset a set of media objects obtained via idig_search_media with a set of records obtained via idig_search_records? I would like to bulk download media images but have columns for lat/lon as an option in 'fields'

[mgaynor1]

Hi @richiehodel -you can download the fields "data.dwc:decimalLatitude" and "data.dwc:decimalLongitude" with your media download.

If you want to connect a media and record download, the "records" field in the media download should match the UUID in the record download. We have some documentation for this here: https://idigbio.github.io/ridigbio/articles/BasicUsage.html#download-media-records

[jbennettufl]

@richiehodel Additionally, I decided to test this out with a large set of data and didn't find any issues doing this and combining the results. Here is an example of the code which I have been working with:

library(ridigbio)
library(future)
library(future.apply)

# Set up parallel processing
plan(multisession, workers = 4) 

# Get your initial records
results <- idig_search_media(
  rq = list(
    recordset = "6b8d6d23-308a-4886-a3e4-00afbfc2960b"
  ),
  fields = c("all"),
  limit = 1000
)

# Check the structure to understand what we're dealing with
print("Structure of results:")
str(results, max.level = 2)

# Create a clean list of records with proper structure
records <- list()
for (i in 1:length(results$records)) {
  rs <- results$records[[i]]
  if (!is.null(rs)) {
    # Check if the record is atomic (simple value) or a complex structure
    if (!is.list(rs) && !is.data.frame(rs)) {
      # It's an atomic value, so we create a list structure
      record_item <- list(uuid = rs)
    } else {
      # It's already a list or data frame
      record_item <- rs
    }
    records <- c(records, list(record_item))
  }
}

# Verify our records structure
print("First few records:")
str(head(records))

# Process in parallel but with controlled batch sizes
batch_size <- 10
num_batches <- ceiling(length(records) / batch_size)

search_results_by_uuid <- list()

for (batch in 1:num_batches) {
  start_idx <- (batch - 1) * batch_size + 1
  end_idx <- min(batch * batch_size, length(records))
  batch_records <- records[start_idx:end_idx]
  
  batch_results <- future_lapply(batch_records, function(record) {
    uuid <- if (is.list(record) && !is.null(record$uuid)) record$uuid else record
    result <- idig_search_records(rq = list(uuid = uuid), fields = c("all"), limit = 1000)
    return(list(uuid = uuid, search_data = result))
  })
  
  # Add to our collection indexed by UUID
  for (result in batch_results) {
    if (!is.null(result$uuid)) {
      search_results_by_uuid[[result$uuid]] <- result$search_data
    }
  }
  
  Sys.sleep(1)
  cat(sprintf("Completed batch %d of %d\n", batch, num_batches))
}

plan(sequential)

# Now create a new combined structure with both original data and search results
combined_results <- results
for (i in 1:length(results$records)) {
  # Extract the UUID (handling both list and atomic cases)
  record <- results$records[[i]]
  if (is.list(record)) {
    uuid <- record$uuid
  } else {
    # If record is atomic (just the UUID itself)
    uuid <- record
  }
  
  # Get the corresponding search result
  search_data <- search_results_by_uuid[[uuid]]
  
  # Add it to the structure
  if (is.list(combined_results$records[[i]])) {
    combined_results$records[[i]]$search_data <- search_data
  } else {
    # If the original was just a UUID, create a proper structure
    combined_results$records[[i]] <- list(
      uuid = uuid,
      search_data = search_data
    )
  }
}

# Now combined_results has both the original data and the search results

[richiehodel]

Hi @mgaynor1 and @jbennettufl, thanks for the helpful feedback!
@jbennettufl, I ran through your code and it worked just fine.

However, I'm still running into issues when I use some 'messier' datasets. For instance, if I run the below code, I run into problems because some of the entries in the 'records' column have multiple values (e.g., row 3, row 7).
I can of course find a workaround by removing problematic rows, but I am curious if you have a simple solution for this problem.
Thanks!

results <- idig_search_media(rq = list(genus = "Sphaeralcea", specificepithet= "ambigua"), fields = c("all"), limit = 1000)

[jbennettufl]

@richiehodel I had a look at this as well, and after our most recent re-index, you will be able to find examples of these records now as the 4th and 8th records. It is possible to modify the script to look also for multiple records entries and it will go through and add them to another list additionally.

library(ridigbio)
library(future)
library(future.apply)

# Set up parallel processing
plan(multisession, workers = 4)

# Get your initial records
results <- idig_search_media(
    rq = list(
        genus = "Sphaeralcea", specificepithet = "ambigua"
    ),
    fields = c("all"),
    # limit = 1000
    limit = 10
)

# Check the structure to understand what we're dealing with
print("Structure of results:")
str(results, max.level = 2)

# Create a clean list of records with proper structure
records <- list()
for (i in 1:length(results$records)) {
    rs <- results$records[[i]]
    if (!is.null(rs)) {
        # Check if the record is atomic (simple value) or a complex structure
        if (!is.list(rs) && !is.data.frame(rs)) {
            # It's an atomic value, so we create a list structure
            record_item <- list(uuid = rs)
        } else {
            # It's already a list or data frame
            record_item <- rs
        }
        records <- c(records, list(record_item))
    }
}

# Verify our records structure
print("First few records:")
str(head(records))

# Process in parallel but with controlled batch sizes
batch_size <- 10
num_batches <- ceiling(length(records) / batch_size)

search_results_by_uuid <- list()

for (batch in 1:num_batches) {
    start_idx <- (batch - 1) * batch_size + 1
    end_idx <- min(batch * batch_size, length(records))
    batch_records <- records[start_idx:end_idx]

    batch_results <- future_lapply(batch_records, function(record) {
        result <- list()
        uuid <- if (is.list(record) && !is.null(record$uuid)) record$uuid else record

        # Check if uuid is a list/vector or a single value
        if (length(uuid) > 1) {
            # If it's a list/vector, iterate through each uuid
            for (id in uuid) {
                ures <- idig_search_records(rq = list(uuid = id), fields = c("all"), limit = 1000)

                result <- c(result, list(ures))
            }
        } else {
            # If it's a single value, just do one query
            ures <- idig_search_records(rq = list(uuid = uuid), fields = c("all"), limit = 1000)
            result <- list(ures)
        }

        return(list(uuid = uuid, search_data = result))
    })

    # Add to our collection indexed by UUID
    # In the batch processing loop:
    for (result in batch_results) {
        if (!is.null(result$uuid)) {
            # If uuid is a vector with multiple UUIDs
            if (length(result$uuid) > 1) {
                # Store each complete UUID as a separate entry
                for (i in seq_along(result$uuid)) {
                    single_uuid <- result$uuid[i]
                    search_results_by_uuid[[single_uuid]] <- result$search_data[[i]]
                }
            } else {
                # Single UUID case
                search_results_by_uuid[[result$uuid]] <- result$search_data
            }
        }
    }

    Sys.sleep(1)
    cat(sprintf("Completed batch %d of %d\n", batch, num_batches))
}

plan(sequential)

# Now create a new combined structure with both original data and search results
combined_results <- results
for (i in 1:length(results$records)) {
    record <- results$records[[i]]
    if (is.list(record)) {
        uuid <- record$uuid
    } else {
        uuid <- record
    }

    # Handle multiple UUIDs case
    search_data <- NULL
    if (length(uuid) > 1) {
        search_data <- list()
        for (j in seq_along(uuid)) {
            single_uuid <- uuid[j]
            if (!is.null(search_results_by_uuid[[single_uuid]])) {
                search_data[[j]] <- search_results_by_uuid[[single_uuid]]
            }
        }
    } else {
        # Single UUID case
        search_data <- search_results_by_uuid[[uuid]]
    }

    # Add it to the structure
    if (is.list(combined_results$records[[i]])) {
        combined_results$records[[i]]$search_data <- search_data
    } else {
        # If the original was just a UUID, create a proper structure
        combined_results$records[[i]] <- list(
            uuid = uuid,
            search_data = search_data
        )
    }
}

[richiehodel]

Hi @jbennettufl
I've just had a chance to go through this example and it's very helpful; thank you!!