Explorer Toolkit

Universal command line access to cloud storage

Suggest Edits

Introduction

Explorer Toolkit provides command-line access to cloud storage. It includes a gateway server called CloudD for cloud storage integration, and an interactive command-line shell called explore for accessing cloud storage.

Explorer Toolkit is available for Linux, Windows, and Mac. It currently supports Amazon Drive, Box, Dropbox, Google Drive, Microsoft OneDrive, and Procore. It also supports private file servers on Linux, Windows, and Mac through the File Server Toolkit.

To access cloud storage from any machine:

  1. Install Explorer Toolkit
  2. Start CloudD
  3. Start the explore shell to connect one or more cloud storage accounts
  4. Use explore shell to access content

Install Explorer

Download toolkit and extract the download to your preferred location.

user@ubuntu-64:~/toolkit$ tar -xvzf linux64-explorer-toolkit-4.tar.gz
./
./explorer/
./explorer/explore
./explorer/start.sh
./explorer/stop.sh
./explorer/config.json
./explorer/cloudd
./explorer/TOS.txt
./explorer/dist.dat

user@ubuntu-64:~/toolkit$

C:\toolkit> unzip win-explorer-toolkit-5.zip
Archive:  win-explorer-toolkit-5.zip
   creating: explorer/
  inflating: explorer/cloudd.exe
 extracting: explorer/config.json
  inflating: explorer/dist.dat
  inflating: explorer/explore.exe
  inflating: explorer/start.bat
  inflating: explorer/stop.bat
  inflating: explorer/TOS.txt

C:\toolkit>

mbp:toolkit user$ tar -xvzf mac-explorer-toolkit-5.tar.gz
./
./explorer/
./explorer/explore
./explorer/start.sh
./explorer/stop.sh
./explorer/config.json
./explorer/cloudd
./explorer/TOS.txt
./explorer/dist.dat

mbp:toolkit user$

If you are updating an existing install to a newer version, see the [Upgrade Toolkit Version] section for more information.

Start Cloudd

Cloudd is a gateway server. It provides universal access to cloud storage. Run the start script in Explorer Toolkit to begin running CloudD in the background.

user@ubuntu-64:~/toolkit$ cd explorer
user@ubuntu-64:~/toolkit/explorer$ ./start.sh
Starting Cloud Gateway
Started on port 26504

user@ubuntu-64:~/toolkit/explorer$

C:\toolkit> cd explorer
C:\toolkit\explorer> start.bat
Starting Cloud Gateway
Started on port 26504

C:\toolkit\explorer>

mbp:toolkit user$ cd explorer
mbp:explorer user$ ./start.sh
Starting Cloud Gateway
Started on port 26504

mbp:explorer user$

Storage Explorer Shell

Explorer is an interactive shell and command-line interface. It talks to CloudD to connect and access storage. Run explore in Explorer Toolkit to start the interactive shell.

user@ubuntu-64:~/toolkit/explorer$ ./explore
explorer>

C:\toolkit\explorer> explore
explorer>

mbp:explorer user$ ./explore
explorer>

📘

Explorer Shell CLI Reference

Please refer to the Explorer Shell CLI Reference for details on available explorer subcommands.

Connect Cloud Storage

Use Explorer Shell to connect cloud storage accounts. To authorize access, use the explore subcommand named after the storage provider. Follow the command instructions to register access. 

$ explore
explorer> onedrive authorize

What do you want to call this OneDrive account?
Account Name: work

Please open a web browser to authorize access:
https://oauth.grid.oxygencloud.com/v1/signin/abc123
After authorizing access, press enter to continue. 

Confirmed. /onedrive/work authorized.
explorer> ls /onedrive/work --long
FOLDER    General
FOLDER    Engineering
FOLDER    Sales
FOLDER    Random
FILE      2306916 Oct 05 10:10 readme.txt
explorer>

Browsing Storage (ls, cd, pwd)

Use the ls subcommand to list the current directory. To change the current working directory, use the cd subcommand. To see the current working directory, use the pwd subcommand.

$ explore
explorer> pwd
/
explorer> ls -l
FOLDER    amazon_drive
FOLDER    box
FOLDER    google_drive
FOLDER    onedrive
FOLDER    oxygen
explorer> cd dropbox
explorer> pwd
/dropbox
explorer> ls
FOLDER    personal
explorer> ls -l personal
FOLDER    Apr 22 15:56 backup
FOLDER    Dec 19 18:01 junk
FOLDER    Dec 18 14:42 media
FILE      2306916 Oct 05 10:10 mixed_tape.mp4
explorer> 
$ explore ls -l personal
FOLDER    Apr 22 15:56 backup
FOLDER    Dec 19 18:01 junk
FOLDER    Dec 18 14:42 media
FILE      2306916 Oct 05 10:10 mixed_tape.mp4
$

Make sure to read the note below about the --refresh option if you aren't seeing the latest changes made on the remote storage side when using the ls command. Without the -r option, the explorer will display the last cached view of the files in that folder.

📘

Use ls with the --refresh option to get a fresh directory listing

By default, the ls command reads from a local cache for faster performance. If you need to make sure the latest is returned, you can use the --refresh option (which will do what is necessary to get the latest view, including triggering a network call to the remote storage to get the latest listing).

In practice, this means that if you are doing a bunch of local changes, the cache gets updated regularly so there may be little need to do a full refresh. But if you absolutely need to have a fresh directory listing including any remote changes made since the cache was last updated, then use the --refresh option.

Download Files

Use the explore download subcommand to download files or folder.

C:\toolkit\explorer> explorer
explorer> download Goals.docx C:\Users\user\Downloads\
Saved to: C:\Users\user\Downloads\Goals.docx
explorer> download media C:\Users\user\Downloads --recursive

Processed 10 files with 0 errors.
explorer>

Upload Files

Use the explore upload subcommand to upload local files or folders to storage.

$ explore upload ~/temp/media /dropbox/work/project/x --recursive --overwrite

Processed 10 files with 0 errors.
$ explore
explorer> upload ~/temp/media2/* /dropbox/work/project/x/media --recursive --skip

Copy Files

Use the explore copy subcommand to copy files across cloud storage accounts.

$ explore
explorer> copy /google_drive/work/project/x /dropbox/work/project/x --recursive --merge

Processed 10 files with 0 errors.
$

Reorganize Files (mv, rename)

Use the explore mv and explorer rename subcommands to reorganize files.

$ explore
explorer> rename /google_drive/work/projects/x moonshot
explorer> ls -l /google_drive/work/projects
FOLDER .   moonshot
explorer> mv /google_drive/work/projects/moonshot /google_drive/work/investments
explroer>

Shell Scripting

By default, Explorer Shell launches in interactive mode. If a caller starts Explorer Shell with a subcommand as an argument, Explorer Shell immediately executes the subcommand and exits with a return code. 

$ explore download /dropbox/personal/media/movie.mp4 .
$ ls
movie.mp4
$

You can use Explorer Shell in command-line mode to incorporate cloud storage into your shell scripts.

Example: Backup directory to cloud storage

#!/bin/bash
set -e

# Usage: ./backup_dir.sh [source directory path] [target directory path]

# path to Explorer Toolkit
TOOLKIT="$HOME/Downloads/explorer"

SOURCEPATH="$1"
if [[ ! -d $SOURCEPATH ]]; then
     echo
     echo "Source path must be an existing directory"
     echo
     exit 1
else
    # get the absolute path, in case we were given a relative path
    SOURCEPATH=$(cd $SOURCEPATH 2> /dev/null && pwd -P)
fi

# Check that cloudd is running. If not, start it
if pgrep -f "$TOOLKIT/cloudd" >/dev/null || pgrep -f "\./cloudd" >/dev/null ; then
    echo "cloudd already running"
else
    (cd $TOOLKIT && "$TOOLKIT/start.sh")
fi

DESTPATH="$2"
echo "Uploading contents of $SOURCEPATH ..."
(cd $TOOLKIT && ./explore upload "$SOURCEPATH/*" "$DESTPATH" --merge --recursive)
echo "Done!"

📘

Return Code

In command-line mode, Explorer Shell executes the command and exits with a return code. The return code is 0 for normal execution. If there is an error, the return code is 1. Use return code in scripts to determine if the command executed as expected.

Upgrade Versions

From time to time, we will release new versions of the Explorer Toolkit on the download page. The instructions below will help you update your build. The zip package conveniently places all of the explorer toolkit files into a parent explorer folder, so this is not a difficult process:

  1. Stop CloudD using the stop script from your prior install.
  2. Unpack the zip package to a temporary location.
  3. Delete the parent explorer folder containing your prior toolkit version's files.
  4. Copy the new explorer folder from the new unzipped package to the same location as before.

At this point, you can use the start script to start the new CloudD gateway. All of the storage you linked before should still be active, and your command history for tab autocompletion in the interactive mode explorer> prompt should be preserved.

Reset Configuration

In the event you would like to clear your existing configuration, simply shutdown CloudD with the stop script and then delete the .oxygen and .gateway folders from the current user's home directory. When you start CloudD again and try to list your storage with the explore utility, you'll find that all of the prior state has been cleared.

Updated over 4 years ago