Google Drive

Data organization

We store all of the data collected by IMACS in a Team Drive, organized in the following way:

ACCESS
└── IMACS
    ├── <TARGET NAME>
    │   └── <UT DATE>
    ├── <...>
    │   └── <...>
    └── <...>
        └── <...>

For example, a dataset for HAT-P-26b collected on the night of 23 Mar 2021 would be stored as: ACCESS/IMACS/HATP26/ut210323.

Data access

The folder can be accessed as a regular cloud directory after access is granted by one of the admins. There are also handy tools for interacting with the data outside of the web browser if desired. Three that have worked well for us in the past are Backup and Sync, Google Drive File Stream (now called Google Drive for desktop, comparison here), and rclone. Rclone is especially useful when working directly from the terminal or on Linux, so we will outline basic usage instructions for this tool in the next section.

Data transfer

As of this writing, the current LCO policy is to upload data collected by the remote observer at Magellan to a secured Google Drive folder. From there, we can transfer the data using whichever manner we prefer. Here, we will step through an example using rclone, which has the capability to transfer, download, and mount files/directories between cloud storage providers in a parallel and efficient manner. More information about configuring rclone to work with Google Drive can be found here. The instruction we post below are for a *nix system.

Create a new remote

The following commands in the terminal will allow rclone to communicate with our Team Drive:

> rclone config
> n (Select new remote)
> (Enter a name to call the remote)
> (Select cloud storage provider from list. This can be selected by name or number)
> (The client id and secret will be asked for next. If you are comfortable with using
   Google Cloud Platform (https://rclone.org/drive/#making-your-own-client-id),
   go ahead and enter them here. Otherwise, just press enter to skip these prompts)
> (Select the option for full access to the drive)
> (Press enter to skip the root_folder_id and service_account_file prompts)
> (Select n to skip the advanced configuration prompt)
> (Select y to complete the authetication in your browser)
> (Return back to the terminal)
> (Select y to configure as a team drive)
> (Select ACCESS from the list)
> (Select y to complete setup and q to exit the config wizard)