Skip to content

2. Azure Synchronization

Jump to bottom
Danny Collier edited this page May 26, 2025 · 25 revisions

Overview

The ultimate deliverables for our application will be a set of xml, txt, csv, and media files delivered to Azure file storage. (Note that this is Azure file storage and not an Azure storage blob.) SkillRX will store and manage its media files in its own file storage. A process within SkillRX will manage synchronization with Azure.

The Azure storage includes folders for delivery to:

  • Raspberry devices (CMES-PI)
  • Mini computers
  • USB

As of this writing, we are not delivering files for USB. That could change.

Storage Paths and Filenames

Documents are stored in a folder in the Azure file storage based on their topic's language. The path comes down to:

  • "[environment-specific route]/[file share folder determined by language]/assets/[topic_id]_[filename_with_extension]"

For now, these requirements are only concerned with the "for CMES-PI" files, both current and legacy, and not the legacy USB app files.

We will generate and deliver one set of files for each language, delivered to the root storage path for the environment plus the paths and filenames specified here:

  • Legacy XML for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix]Server_XML.xml
  • Provider XML for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix][provider name].xml for every provider
  • New topics for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix]New_Uploads.xml
  • We will not generate Top topics for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix]Top_Topics.xml as these depend on collecting stats from remote devices.

These file names and locations follow the patterns described in the XML Generation application.

From the XML File Generation app documentation: image

Filenames will have topic.id and an underscore prepended to prevent filename collisions. It is less likely but still possible for an editor to upload two files with the same name to the same topic.

When a topic is archived, we will move its related Training Materials documents to an archival folder. Archival folders follow this naming convention:

  • [language.file_storage_prefix]CMES-Pi_Archive/[their unarchived location following the conventions described above.

Media Files

Media storage follows this convention:

  • [language.file_storage_prefix]CMES-Pi/assets/Content/[Topic_id]_[file name with underscores instead of spaces and no special characters].[file extension]

XML Files

See the XML Generation application for details.

"Provider XML" structure from the XML Generation app documentation: image

"Legacy XML" structure from the XML Generation app documentation: image

Tag Files

In addition to the XML and the media files, we will deliver .txt files containing tag information.

We will generate and deliver one set of files for each language, delivered to the root storage path for the environment plus the paths and filenames specified here:

  • Tag file for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/Tags/[language.file_storage_prefix]tags.txt
  • Tags and title file for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/Tags/[language.file_storage_prefix]tagsAndTitle.txt

CSV Files

The mini computers rely on a set of .csv files which they import to their local database. [full requirements for those files]

Azure Client

Our Azure interface uses Azure File Shares, a gem written by Ruby for Good volunteer Dmitry Trager.

We have not found a viable way to set up Azure file storage for local development environments, so we are using a shared Azure file storage created by our stakeholders. We can provide the name of this environment and access details to developers who take on work related to Azure storage. Since this is a shared environment, local development environments will all be writing to the same storage, so developers will need to coordinate.

The Azure client will be used to:

  • Authenticate
  • Add individual media, csv, txt, and xml files to Azure.
  • Archive files when a topic is archived. There is no api for moving files, so we will delete the file and upload it to the archive location * Delete files when a topic is deleted

Synchronization

Synchronization between SkillRX and the Azure File Storage has two main aspects: training material media and generated files. The media can be synchronized in real time but file generation and synchronization will need to be handled asynchronously.

Training Materials

With training materials, we are maintaining two separate repositories with the same content but with different organizational structures. Files are added, replaced, moved, or--in rare cases and only by admins--they can be deleted. We will handle these updates in real time.

Generated Files

With the generated xml, csv, and txt files, each change within SkillRX can result in changes in multiple files. A series of changes during an editing session can result in a cascading set of updates. We will handle file generation as a scheduled task, executed at intervals (TBD). We will also provide a way for admins to trigger a file generation/sync through the user interface.

Clone this wiki locally