- Improved observability and import handling by adding debug logging to zip/model imports and extending ACCEPT validation to support multiple return types (model/zip/list).
- Fixed an issue where the ACCEPT button did not properly validate imported zips or lists, resulting in those objects not being imported.
- Refactored model and zip import workflows to use
import_bundle_zip(), replacing manual unzip logic - Added
extract_model_import()andextract_model_notes()helpers - Added support for loading models from directories via
from_dir - Deprecated
extractNotes()andextractObjectFromFile() - Added support for legacy
model.RDatabundles
- Reverted pushes to the false branch
- Reverted parameter name from
compression_levelto its originalcompressionLevelin function signatures of already exported functions.
-
Extracted and refactored export/import logic for model bundles by introducing two new classes:
DownloadBundleandZipImport. -
These classes handle file bundling and zip extraction operations, improving code organization and reusability by moving bundle creation logic out of server functions.
-
Refactored
downloadModelServerto use the newbuild_download_zipfunction instead of inline bundle creation. -
Standardized parameter naming across the codebase (e.g.,
compressionLevel→compression_level,subFolder→sub_model). -
Introduced a new
DataProcessItemclass structure to modernize and organize data processing workflows. -
Added S3 classes:
DataProcessItemandDataProcessLinkto represent data in different processing states.
- Improved file import handling with automatic file type detection, filtering of allowed types, and clearer error messages.
- Data preparation in Query with SQL, Prepare, and Merge is now automatically enabled once a file is successfully loaded.
- Renamed
mergeListtodataProcessListthroughout the codebase for improved clarity. - Deprecated the
fileExtensionparameter in favor of usingckanFileTypes.
- align test to changes on CKAN (#116)
- fixed issue in previous
llmModuleversion
- importModule - "Query with SQL" tab:
- integrated the
llm_generate_promptmodule from our newllmModulepackage - when using AI PEITHO data operations to create AI-assisted SQL queries, users can now choose between the OpenAI and DeepSeek providers
- removed all dependencies on the rgpt3 package which was previously used for OpenAI API calls.
- integrated the
- fixed issue in the UI when using
batch = TRUEwithimportDataServer
- importModule: remove warning when no internet connection is available
- importModule: add link to the repository to the help text
- reduce package size by including test files into the
.Rbuildignore - importModule: update help text if remote files are not available
- importModule: cleaner UI to display tables and column names in the 'Query' tab
- importModule: fixed an issue with the update of data processing in the 'Prepare' tab
- fixed an issue where the 'Only data and user inputs' checkbox was not functioning correctly during model download (#108)
- remove the tab specific 'accept buttons' in the import module, use the tab id instead to identify the data to be accepted
- fix issue that prevented to (optionally) process data before import if data was not valid
- combine buttons 'Create Query from file' and 'Prepare / Merge file(s)' and the logic behind into a single button 'Process data'
- fix issue with missing filename for file imports (#104)
- prevent duplicated check of internet connection
- importModule: enable import of
.binfiles forimportType = "list"(requirement forInquiryRapp)
- new wrapper module
dataUIanddataServeraround theimportDatamodule that allows for loading example data
- fix an issue where the Pandora API being unreachable caused the import module to stop responding (#101)
- now the connection is checked only once when the pop-up is opened
- the time out for the connection check is set to 10 instead of 60 seconds
- a warning message is displayed if the connection fails
- Query with SQL:
- automatically load available gpt models directly from openai and filter those applicable to query creation (#78)
- option to select the gpt model for the creation of SQL queries (#78)
- when importing models:
- additionally to the display of warning and error messages, a new message displays the version number under which a model was saved
- the version is now also part of the returned import object
- update 'deprecated' warnings for several deprecated arguments
- maintenance update for the selectData module, split of the module into two modules to reduce the complexity
- Configure Data: configure data from clv, xlsx imports
- Configure File: configure file from model, zip, list imports
- maintenance update for the import module, preparations for the split of the import module into
two modules to reduce the complexity of the import data module (which will increase the speed)
- Import Data: import data from local files, url, or using the pandora api
- Import Model/Zip: import model objects, zip from local files, url, or using the pandora api
- part 1: extraction of the import of model objects and zips from the importData module into a separate module
- Import Data:
- option to import a list from a json file from url, Pandora or file (#91)
- option to import user inputs together with a json file in order to load a previous session (#91)
- Download Model: new optional parameter
pathToOtherZipto add the content of a zip file to the the zip file to be downloaded (which contains also model, data and inputs objects). This is needed for MapR where pictures are stored in zip files and needed for plots (instead of model outputs) (#89)
- export of functions to extract notes and inputs from an uploaded file (#89)
- Import Data module:
- updated paths to find example models locally to enable compatibility to external apps (e.g. CausalR) (#86)
- tryCatchWithWarningsAndErrors():
- shifted function to the package
shinyToolsand renamed toshinyTryCatch() - improved error handling for the function
shinyTools::shinyTryCatch(), now all errors and warnings of an expression are caught and forwarded to the UI of an app - deprecated the function
tryCatchWithWarningsAndErrors()in the packageDataTools, added a warning message to use the functionshinyTools::shinyTryCatch()instead
- shifted function to the package
- Down- and Upload of Models module:
- optionally, specify a filename for model download
- store the filename of an uploaded model when reading the path
- export of module previewData for the app 'Data search and Spatiotemporal modeling' (MpiIsoApp)
- fix import of model zips from url (#80)
- fix issue when trying to end an open rgpt session
- fix config of path to the zip examples
- integration of the most recent version of rgpt functions into data tools package
- option to download 'data queries' as .json file if data loading was successful in advance (#55)
- stores only user inputs and file path information, not the data itself
- save data queries that were created under 'Query with SQL'
- option to upload 'data queries' containing user inputs for data imports from a .json file
- option to upload 'data queries' containing user inputs for data imports from github from .json file
- Integration of rgpt3 functions into data tools package
- Note: The functions were copied from https://github.com/ben-aaron188/rgpt3 & https://github.com/gasparl/rgpt3/tree/main Please refer to the original R-Packages for more details.
- Import of models from Pandora:
- an error message occured when trying to load a model from pandora.
- fix: adding the missing download of the zip file from the url before unpacking the zip
- Import Data module: display of "About" information that is associated to a selected Pandora Repository (#62)
- removing functions for "Pandora" API, updated functionality was moved to the package
Pandora
- removing functions for "IsoMemo" API, updated functionality was moved to the package
IsoMemo
- export of functions
fillIsoData(),handleDescription(),has_internet()which will be needed for the Isomemo app
- Import Data module: A modification that is required to apply the import module within the app
MapR.- new value "zip" for parameter
importType - new optional parameter
expectedFileInZip: This parameter is ignored if importType != "zip". Name files that must be contained in the zip upload.
- new value "zip" for parameter
- Import Data module: new optional parameter
fileExtensionwhich filters files for the import of model objects for exactly the specified type, e.g. "zip", "resources", "bpred", "bmsc"
- Import Data module: Modal kept closing when trying to import data a second time
- Import Data module: functionality to import model objects by setting a new parameter
importType = "model"(#50)
- Add a new parameter "mappingId" to all functions that are used to reach the isomemo API (#36)
- Import of Pandora datasets:
- fix and optimize checks for internet connection
- fix issue with sheet selection if no internet
- set 'file' as initial source if no internet
- Import of Pandora datasets: rename "Pandora group" to "Pandora network", and "Pandora dataset" to "Pandora repository" (#18)
- contains fixes regarding issues with loading data and updates of the lists of user inputs (#47)
- Import of Pandora datasets: info icon next to the meta data filter with examples how to filter for several keywords (#18)
- catch api connection errors and output them in UI
- catch errors for wrong meta filters and output them in UI
- Import of Pandora datasets:
- option to filter the list of Pandora datasets for a string in their metadata (#18)
- option to select a ckan group to filter the list of Pandora datasets (#20)
- Import of Pandora datasets:
- alphabetical order of datasets and resources (#18)
- select resource types to be displayed (xls, xlsx, csv, odt, txt, all) (#18)
- fixes issues when trying to reach the isomemo, github and pandorra API without internet connection
- Import Data module
- in the tab Prepare: option to select the file that should be updated
- in the tab Merge: option to merge several files
- merge two files, save with a new file name, select the new file and another table, merge, repeat, ...
- in the tab Query with SQL: option to keep a new file and do some data preparation in other tabs
- new function to update names of lists if the names occur in another list,
see
updateListNamesIfDuplicate()
- new functions to reach the isomemo API that were extracted from the iso-app
- module downUploadButton: update of titles, progress bar for upload, close pop up after download
- new module downUploadButton: A button that opens a pop up with the download and upload UI for models. The upload contains upload from local files (3) or upload from a remote github folder. If there is no access to the github folder, a note is displayed.
remoteModelsServer: new parameter to set the folder for remote models on github if it differs from the default folder
- Down- and Upload of Models module: UI and server functions for down and upload of user data, user inputs and model output
- Remote Models module: UI and server functions for selecting a remote (or local) model and obtaining the remote path on github (or the local path) to the file of the selected model
- Import Data module in the tab Query with SQL: before using the GPT-3 feature a confirmation is required (#15)
- Import Data module in the tab Query with SQL: option to use GPT-3 for creation of SQL queries (#15)
- New function to catch and forward errors and warnings to the UI of the app. If a warning is triggered the result of the evaluated expression is kept and returned.
- factors are converted to character in cutAllLongStrings (#25)
- apply output format before applying custom checks
- UI for batch mode (requirement for ReSources app)
- option to format output as matrix (requirement for ReSources app)
- update logic for assignment of colnames and rownames because of matrix outputs
- import of Pandora dataset: keep string in input even if not found in choices (#19, PR #21)
- import of Pandora dataset: empty input as default input (#19)
- disable "Accept Merged" / "Accept Query" buttons if merge / query failed
- wrong warnings when switching on/of 'First column contains rownames'
- new custom check to ensure that all columns are numeric
- renaming package to DataTools
- export of function
cutAllLongStrings() - new parameter
defaultSourceinimportDataServer()
- fix typo in Merge Data
- fix disappearance of buttons after opening pop-up a second time
- reset fields in Prepare Data when switching source to file or url
- custom checks for warnings and errors added that can be applied in the importData module
- module
importData- import from local files, url, or using the pandora api
- prepare data
- merge data
- query data
importDatacan be integrated into different apps- already present in Pandora & IsoMemo spatiotemporal modeling (
MpiIsoApp)
- already present in Pandora & IsoMemo spatiotemporal modeling (
- app to test modules locally: Test App