Explorer

Introduction

Explorer is a command-line tool for system admins to access any cloud storage. It integrates multi-cloud storage into a unified namespace to provide simple, path-based access to any content via interactive shell or CLI.

Access Everywhere
Access cloud storage from any terminal.

Access Everything
Access any content from any storage in one place.

Access Automatically
Access cloud storage from shell scripts.

Start Cloud Storage Gateway

Explorer uses Cloud Storage Gateway to access cloud storage. Cloud Storage Gateway runs as a daemon process and listens for requests from Explorer and other valid Gateway clients. Cloud Storage Gateway is included with the toolkit. Start Cloud Storage Gateway before using Explorer.

user@ubuntu-64:~/toolkit/$ ./startcloud.sh
Starting Cloud Gateway
Started on port 27504

user@ubuntu-64:~/toolkit/$

C:\toolkit\> startcloud.bat
Starting Cloud Gateway
Started on port 27504

C:\toolkit\>

mbp:toolkit user$ ./startcloud.sh
Starting Cloud Gateway
Started on port 27504

mbp:toolkit user$

Start Interactive Explorer Shell

Explorer is an interactive shell and command-line interface. It talks to Gateways to connect and access storage. Run explore in Explorer Toolkit to start the interactive shell.

user@ubuntu-64:~/toolkit/$ ./explore
explorer>

C:\toolkit\> explore
explorer>

mbp:toolkit user$ ./explore
explorer>

Connect Cloud Storage

Use the interactive Explorer Shell to connect cloud storage accounts. To register a storage account, use the explore subcommand named after the storage provider and follow the command instructions to authorize access. Once connected, Explorer maintains the connection until explicitly deauthorized.

$ explore
explorer> onedrive authorize

What do you want to call this OneDrive account?
Account Name: work

Please open a web browser to authorize access:
https://oauth.grid.oxygencloud.com/v1/signin/abc123
After authorizing access, press enter to continue. 

Confirmed. /onedrive/work authorized.
explorer> ls /onedrive/work --long
FOLDER    General
FOLDER    Engineering
FOLDER    Sales
FOLDER    Random
FILE      2306916 Oct 05 10:10 readme.txt
explorer>

Browse Storage (ls, cd, pwd)

Use the ls subcommand to list the current directory. To change the current working directory, use the cd subcommand. To see the current working directory, use the pwd subcommand.

$ explore
explorer> pwd
/
explorer> ls -l
FOLDER    amazon_drive
FOLDER    box
FOLDER    google_drive
FOLDER    onedrive
FOLDER    oxygen
explorer> cd dropbox
explorer> pwd
/dropbox
explorer> ls
FOLDER    personal
explorer> ls -l personal
FOLDER    Apr 22 15:56 backup
FOLDER    Dec 19 18:01 junk
FOLDER    Dec 18 14:42 media
FILE      2306916 Oct 05 10:10 mixed_tape.mp4
explorer> 
$ explore ls -l personal
FOLDER    Apr 22 15:56 backup
FOLDER    Dec 19 18:01 junk
FOLDER    Dec 18 14:42 media
FILE      2306916 Oct 05 10:10 mixed_tape.mp4
$

Make sure to read the note below about the --refresh option if you aren't seeing the latest changes made on the remote storage side when using the ls command. Without the -r option, the explorer will display the last cached view of the files in that folder.

📘

Use ls with the --refresh option to get a fresh directory listing

By default, the ls command reads from a local cache for faster performance. If you need to make sure the latest is returned, you can use the --refresh option (which will do what is necessary to get the latest view, including triggering a network call to the remote storage to get the latest listing).

In practice, this means that if you are doing a bunch of local changes, the cache gets updated regularly so there may be little need to do a full refresh. But if you absolutely need to have a fresh directory listing including any remote changes made since the cache was last updated, then use the --refresh option.

Download Files

Use the explore download subcommand to download files or folder.

C:\toolkit\> explore
explorer> download Goals.docx C:\Users\user\Downloads\
Saved to: C:\Users\user\Downloads\Goals.docx
explorer> download media C:\Users\user\Downloads --recursive

Processed 10 files with 0 errors.
explorer>

Upload Files

Use the explore upload subcommand to upload local files or folders to storage.

$ explore upload ~/temp/media /dropbox/work/project/x --recursive --overwrite

Processed 10 files with 0 errors.
$ explore
explorer> upload ~/temp/media2/* /dropbox/work/project/x/media --recursive --skip

Copy Files

Use the explore copy subcommand to copy files across cloud storage accounts.

$ explore
explorer> copy /google_drive/work/project/x /dropbox/work/project/x --recursive --merge

Processed 10 files with 0 errors.
$

Reorganize Files (mv, rename)

Use the explore mv and explorer rename subcommands to reorganize files.

$ explore
explorer> rename /google_drive/work/projects/x moonshot
explorer> ls -l /google_drive/work/projects
FOLDER .   moonshot
explorer> mv /google_drive/work/projects/moonshot /google_drive/work/investments
explroer>

Shell Scripting

By default, Explorer Shell launches in interactive mode. If a caller starts Explorer Shell with a subcommand as an argument, Explorer Shell immediately executes the subcommand and exits with a return code.

$ explore download /dropbox/personal/media/movie.mp4 .
$ ls
movie.mp4
$

You can use Explorer Shell in command-line mode to incorporate cloud storage into your shell scripts.

Example: Backup directory to cloud storage

#!/bin/bash
set -e

# Usage: ./backup_dir.sh [source directory path] [target directory path]

# path to Explorer Toolkit
TOOLKIT="$HOME/Downloads/explorer"

SOURCEPATH="$1"
if [[ ! -d $SOURCEPATH ]]; then
     echo
     echo "Source path must be an existing directory"
     echo
     exit 1
else
    # get the absolute path, in case we were given a relative path
    SOURCEPATH=$(cd $SOURCEPATH 2> /dev/null && pwd -P)
fi

# Check that cloudd is running. If not, start it
if pgrep -f "$TOOLKIT/cloudd" >/dev/null || pgrep -f "\./cloudd" >/dev/null ; then
    echo "cloudd already running"
else
    (cd $TOOLKIT && "$TOOLKIT/start.sh")
fi

DESTPATH="$2"
echo "Uploading contents of $SOURCEPATH ..."
(cd $TOOLKIT && ./explore upload "$SOURCEPATH/*" "$DESTPATH" --merge --recursive)
echo "Done!"

📘

Return Code

In command-line mode, Explorer Shell executes the command and exits with a return code. The return code is 0 for normal execution. If there is an error, the return code is 1. Use return code in scripts to determine if the command executed as expected.