Use Rclone To Mount Azure Blob Storage into Local Windows File System - NETSEC

Latest

Learning, Sharing, Creating

Cybersecurity Memo

Saturday, November 27, 2021

Use Rclone To Mount Azure Blob Storage into Local Windows File System

This post is going to show you how to mount Azure Blob Storage Account into your local windows file system, which can make the management of Blob's files much easier. 

It can be much easier to mount Azure Storage File system. You can check some related posts from following list:





Prerequisites

1 Storage Account Information

You will need to have your Azure Storage Account created. You should be able to use Storage Explorer to check your files and folder under your Blob Containers.



2 Storage Account Access key

Get your storage account access keys using AZ Cli command in Azure Cloud Shell or directly from Azure Portal.

From Azure Cloud Shell:

PS /home/jon-netsec> az storage account keys list --account-name test4blob --resource-group test-storage --out table
CreationTime                      KeyName    Permissions    Value
--------------------------------  ---------  -------------  ----------------------------------------------------------------------------------------
2021-11-27T18:29:44.581445+00:00  key1       FULL           K3WrYU+MTPY7uUrnY/ou2iXyanSdw6oiKrLBa9WbHUCrKnWu4H4rjDsxMVZBwvHH7/JY7wCXT0OuVpvXOEfg==
2021-11-27T18:29:44.581445+00:00  key2       FULL           SEy3o9+QLXicMe0ysTOn0VhF7Sg1CyimUvmtWRs2sgenx9LHv9eL/fcNYpS7YC0OnM37NYQq1T9S6e36Iqjg==
PS /home/johnyan_ca>
From Azure Portal - Storage Account:



3 Windows preparation

Install following software :

  • Install Chocolatey - a  software management solution, which makes other packages/software installation much easier

Download Rclone and Directly Run it in Windows

Here is an example to download and run latest Windows Rclone v1.60.1.


Get latest Windows Rclone version on your machine (https://rclone.org/install/): 
Unzip it to a folder then from the command line to run rclone command directly:

D:\Tools\rclone-current-windows-amd64\rclone-v1.60.1-windows-amd64>rclone version
rclone v1.60.1
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.2364 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: cmount
D:\Tools\rclone-current-windows-amd64\rclone-v1.60.1-windows-amd64>rclone config
2022/12/20 11:55:34 NOTICE: Config file "C:\\Users\\admin\\AppData\\Roaming\\rclone\\rclone.conf" not found - using defaults
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
Enter name for new remote.
name> yyyjjjj
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 1 / 1Fichier
   \ (fichier)
 2 / Akamai NetStorage
   \ (netstorage)
 3 / Alias for an existing remote
   \ (alias)
 4 / Amazon Drive
   \ (amazon cloud drive)
 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
   \ (s3)
 6 / Backblaze B2
   \ (b2)
 7 / Better checksums for other remotes
   \ (hasher)
 8 / Box
   \ (box)
 9 / Cache a remote
   \ (cache)
10 / Citrix Sharefile
   \ (sharefile)
11 / Combine several remotes into one
   \ (combine)
12 / Compress a remote
   \ (compress)
13 / Dropbox
   \ (dropbox)
14 / Encrypt/Decrypt a remote
   \ (crypt)
15 / Enterprise File Fabric
   \ (filefabric)
16 / FTP
   \ (ftp)
17 / Google Cloud Storage (this is not Google Drive)
   \ (google cloud storage)
18 / Google Drive
   \ (drive)
19 / Google Photos
   \ (google photos)
20 / HTTP
   \ (http)
21 / Hadoop distributed file system
   \ (hdfs)
22 / HiDrive
   \ (hidrive)
23 / In memory object storage system.
   \ (memory)
24 / Internet Archive
   \ (internetarchive)
25 / Jottacloud
   \ (jottacloud)
26 / Koofr, Digi Storage and other Koofr-compatible storage providers
   \ (koofr)
27 / Local Disk
   \ (local)
28 / Mail.ru Cloud
   \ (mailru)
29 / Mega
   \ (mega)
30 / Microsoft Azure Blob Storage
   \ (azureblob)
31 / Microsoft OneDrive
   \ (onedrive)
32 / OpenDrive
   \ (opendrive)
33 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ (swift)
34 / Oracle Cloud Infrastructure Object Storage
   \ (oracleobjectstorage)
35 / Pcloud
   \ (pcloud)
36 / Put.io
   \ (putio)
37 / QingCloud Object Storage
   \ (qingstor)
38 / SMB / CIFS
   \ (smb)
39 / SSH/SFTP
   \ (sftp)
40 / Sia Decentralized Cloud
   \ (sia)
41 / Storj Decentralized Cloud Storage
   \ (storj)
42 / Sugarsync
   \ (sugarsync)
43 / Transparently chunk/split large files
   \ (chunker)
44 / Union merges the contents of several upstream fs
   \ (union)
45 / Uptobox
   \ (uptobox)
46 / WebDAV
   \ (webdav)
47 / Yandex Disk
   \ (yandex)
48 / Zoho
   \ (zoho)
49 / premiumize.me
   \ (premiumizeme)
50 / seafile
   \ (seafile)
Storage> 18
Option client_id.
Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a value. Press Enter to leave empty.
client_id> 1036860883959-oon5ud5p5itpjne7uakrktlhg72sda4t.apps.googleusercontent.com
Option client_secret.
OAuth Client Secret.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_secret> 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
Option scope.
Scope that rclone should use when requesting access from drive.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Full access all files, excluding Application Data Folder.
   \ (drive)
 2 / Read-only access to file metadata and file contents.
   \ (drive.readonly)
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ (drive.file)
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ (drive.appfolder)
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ (drive.metadata.readonly)
scope> 1
Option service_account_file.
Service Account Credentials JSON file path.
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a value. Press Enter to leave empty.
service_account_file>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n
Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
        rclone authorize "drive" "eyJjbGllbnRfaWQiOiIxMDM2ODYwODgzOTU5LW9vbjV1ZDVwNWl0cGpuZTd1YWtya3RsaGc3MnNkYTR0LmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwiY2xpZW50X3NlY3JldCI6IkdPQ1NQWC0"
Then paste the result.

Enter a value.
config_token> eyJ0b2tlbiI6IntcImFjY2Vzc190b2tlblwiOlwieWEyOS5hMEFYOUdCZFZUlkxcGEzSkpjMjlVMS16V0ItSFJEX0UzMUNxNEZTNEVFSWdqbXJPczkyeXRPbl9uYzVzSGxQSWpX0FVR0tWYTNnSkw4MXFHOXZSM01qREVfMlVZS0pDWU9ETlpjRjJpWWhrdlA1OGdwWWhWRHJQRZpTHZCWXZydHk3OEdZbXdoaDhTNUtIOF9PVzk0WWcxNElsSWFDZ1lLQVJzU0FSQVNGUUhVQ3Ni0QzMUNMUldvaGdUOFFCZFhfRnU1UlEwMTYzXCIsXCJ0b2tlbl90eXBlXCI6XCJCZWFyZXJcIixInJlZnJlc2hfdG9rZW5cIjpcIjEvLzAxc1Ryc3N2TEV5Ui1DZ1lJQVJBQUdBRVNOd0YtTDlJcl2ZER1M0hrX2t0S0t1dVNLMHFFNTg1VGs4bW5aNXZGSzJPVEFTZEo4empWX2E2eVN5alYyZVdyTRkV0NGVmI1ZGdcIixcImV4cGlyeVwiOlwiMjAyMi0xMi0yMFQxMjo1NzoyMS42OTk4OTE5LTA1jAwXCJ9In0
Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No (default)
y/n>
Configuration complete.
Options:
- type: drive
- client_id: 1032860283919-oov5ud5p5itpjne7ualrktlhg12sda4t.apps.googleusercontent.com
- client_secret: GOCSQX-6qJC5VE-dcHhPIlFh_O_1d1ew6ru
- scope: drive
- token: {"access_token":"ya29.a0ABdVRRY1pa3JJc29U1-zWB-HRD_E31Cq4F4EEIgjmrOs92ytOn_nc5sHlPIgi_AUGKVa3gJqG9vR3MjDE_2UYKJCYODNZcF2iYhkvP58gpYhVDrPGFiLvBYvrty78GYmwhh8S5KH8_OW94Yg1IaCgYKARsSARASFQHUCsbCD31CLRohgT8QBdX_Fu5RQ0163","token_type":"Bearer","refresh_token":"1//01sTrssvLER-CgYIARAAGAESNwF-L9IrXvdDu3Hk_ktKKuuSK0qE585mnZ5vFK2OTASdJ8zjV_a6ySyjV2eWLddWCFVb5dg","expiry":"2022-12-20T12:57:21.6998919-05:00"}
- team_drive:
Keep this "yyyjjjj" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name                 Type
====                 ====
yyyjjjj              drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q


Note: Rclone conf file is at "C:\Users\admin\AppData\Roaming\rclone\rclone.conf"

You will get refresh_token from above outputs and that will need to put it into your cloudflare workers. 



Also, you might need to get your own Google client ID and Secret based on this document:


Install Rclone on Windows

There are two ways to install Rclone:

1 Using Chocolatey to install Rclone automatically

C:\tools>choco
Chocolatey v0.11.3
Please run 'choco -?' or 'choco <command> -?' for help menu.

C:\tools>choco install rclone -y
Chocolatey v0.11.3
Installing the following packages:
rclone
By installing, you accept licenses for the packages.
Progress: Downloading rclone.portable 1.57.0... 100%
Progress: Downloading rclone 1.57.0... 100%

rclone.portable v1.57.0 [Approved]
rclone.portable package files install completed. Performing other installation steps.
The package rclone.portable wants to run 'chocolateyInstall.ps1'.
Note: If you don't run this script, the installation will fail.
Note: To confirm automatically next time, use '-y' or consider:
choco feature enable -n allowGlobalConfirmation
Do you want to run the script?([Y]es/[A]ll - yes to all/[N]o/[P]rint):

2 Manual installation


There are many ways to do manual installation. Here is a way to use Powershell:

Make sure you run following command first


PS C:\Windows\system32> Set-ExecutionPolicy AllSigned

Then you can use Powershell script to get rclone file downloaded and unzipped, moved to the folder you want it to run, which is c:\tools.

$ [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$ Invoke-WebRequest https://downloads.rclone.org/rclone-current-windows-amd64.zip -Outfile rclone.zip
$ Expand-Archive rclone.zip
$ Move-Item -Path .\rclone\**\rclone.exe -Destination C:\tools\rclone.exe


Configure Rclone to Mount Azure Blob Container

Run "rclone config" command  to configure your new remote. Only thing you will need is the storage account name and access key.

C:\tools>rclone config
Current remotes:

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> test
Option Storage.
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value.
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, SeaweedFS, and Tencent COS
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Better checksums for other remotes
   \ "hasher"
 7 / Box
   \ "box"
 8 / Cache a remote
   \ "cache"
 9 / Citrix Sharefile
   \ "sharefile"
10 / Compress a remote
   \ "compress"
11 / Dropbox
   \ "dropbox"
12 / Encrypt/Decrypt a remote
   \ "crypt"
13 / Enterprise File Fabric
   \ "filefabric"
14 / FTP Connection
   \ "ftp"
15 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
16 / Google Drive
   \ "drive"
17 / Google Photos
   \ "google photos"
18 / Hadoop distributed file system
   \ "hdfs"
19 / Hubic
   \ "hubic"
20 / In memory object storage system.
   \ "memory"
21 / Jottacloud
   \ "jottacloud"
22 / Koofr
   \ "koofr"
23 / Local Disk
   \ "local"
24 / Mail.ru Cloud
   \ "mailru"
25 / Mega
   \ "mega"
26 / Microsoft Azure Blob Storage
   \ "azureblob"
27 / Microsoft OneDrive
   \ "onedrive"
28 / OpenDrive
   \ "opendrive"
29 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
30 / Pcloud
   \ "pcloud"
31 / Put.io
   \ "putio"
32 / QingCloud Object Storage
   \ "qingstor"
33 / SSH/SFTP Connection
   \ "sftp"
34 / Sia Decentralized Cloud
   \ "sia"
35 / Sugarsync
   \ "sugarsync"
36 / Tardigrade Decentralized Cloud Storage
   \ "tardigrade"
37 / Transparently chunk/split large files
   \ "chunker"
38 / Union merges the contents of several upstream fs
   \ "union"
39 / Uptobox
   \ "uptobox"
40 / Webdav
   \ "webdav"
41 / Yandex Disk
   \ "yandex"
42 / Zoho
   \ "zoho"
43 / http Connection
   \ "http"
44 / premiumize.me
   \ "premiumizeme"
45 / seafile
   \ "seafile"
Storage> 26
Option account.
Storage Account Name.
Leave blank to use SAS URL or Emulator.
Enter a string value. Press Enter for the default ("").
account> test4blob
Option service_principal_file.
Path to file containing credentials for use with a service principal.
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
    $ az ad sp create-for-rbac --name "<name>" \
      --role "Storage Blob Data Owner" \
      --scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
      > azure-principal.json
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
Enter a string value. Press Enter for the default ("").
service_principal_file>
Option key.
Storage Account Key.
Leave blank to use SAS URL or Emulator.
Enter a string value. Press Enter for the default ("").
key> SEy3o9+QLXicMe0ysTOn0VhF7Sg1CyimUvmtWRs2sgenx9LHv9eL/fcNYpS7YC0OnM37NYQq1T9S6e36Iqjg==
Option sas_url.
SAS URL for container level access only.
Leave blank if using account/key or Emulator.
Enter a string value. Press Enter for the default ("").
sas_url>
Option use_msi.
Use a managed service identity to authenticate (only works in Azure).
When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
to authenticate to Azure Storage instead of a SAS token or account key.
If the VM(SS) on which this program is running has a system-assigned identity, it will
be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
the user-assigned identity will be used by default. If the resource has multiple user-assigned
identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
msi_client_id, or msi_mi_res_id parameters.
Enter a boolean value (true or false). Press Enter for the default ("false").
use_msi>
Option use_emulator.
Uses local storage emulator if provided as 'true'.
Leave blank if using real azure storage endpoint.
Enter a boolean value (true or false). Press Enter for the default ("false").
use_emulator>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
[test]
type = azureblob
account = test1
key = SEy3o9+QLXicMe0ysTOn0VhF7Sg1CyimUvmtWRs2sgenx9LHv9eL/fcNYpS7YC0OnM37NYQq1T9S6e36Iqjg==
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d>y

Now let’s see if we can access the storage:

C:\tools>rclone lsd blob:
          -1 2021-11-27 18:58:52        -1 rclone

C:\tools>

Mount Rclone Remote to Your Local Windows System

The basics of Rclone is now working on your system. If you want to mount it on your local Windows, we will need to install WinFsp which provides a filesystem proxy for Windows much like Fuse for Linux.

We will use choco to help installation (Make sure your powershell is running under your administrator account):

PS C:\Windows\system32> choco install winfsp -y
Chocolatey v0.11.3
Installing the following packages:
winfsp
By installing, you accept licenses for the packages.
winfsp v1.9.21096 already installed.
 Use --force to reinstall, specify a version to install, or try upgrade.

Chocolatey installed 0/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Warnings:
 - winfsp - winfsp v1.9.21096 already installed.
 Use --force to reinstall, specify a version to install, or try upgrade.
PS C:\Windows\system32> choco install winfsp -y --force
Chocolatey v0.11.3
Installing the following packages:
winfsp
By installing, you accept licenses for the packages.
winfsp v1.9.21096 already installed. Forcing reinstall of version '1.9.21096'.
 Please use upgrade if you meant to upgrade to a new version.
Progress: Downloading winfsp 1.9.21096... 100%

winfsp v1.9.21096 (forced) [Approved]
winfsp package files install completed. Performing other installation steps.
Uninstalling winfsp...
0
winfsp has been uninstalled.
Installing winfsp...
winfsp has been installed.
 The install of winfsp was successful.
  Software installed as 'msi', install location is likely default.

Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
PS C:\Windows\system32>


Last step is to mount to a local folder:


C:\tools>rclone mount blob:rclone C:\tools\rclone-mount
The service rclone has been started.
Unfortunately, the mount will not survive a reboot, even not closing this window. We will need to get this command running as a service. 


Install NSSM (the Non-Sucking Service Manager) Service Manager


PS C:\Windows\system32> choco install nssm -y
Chocolatey v0.11.3
Installing the following packages:
nssm
By installing, you accept licenses for the packages.
Progress: Downloading NSSM 2.24.101.20180116... 100%

NSSM v2.24.101.20180116 [Approved]
nssm package files install completed. Performing other installation steps.
Installing 64 bit version
Extracting C:\ProgramData\chocolatey\lib\NSSM\tools\nssm-2.24-101-g897c7ad.zip to C:\ProgramData\chocolatey\lib\NSSM\tools...
C:\ProgramData\chocolatey\lib\NSSM\tools
 ShimGen has successfully created a shim for nssm.exe
 The install of nssm was successful.
  Software installed to 'C:\ProgramData\chocolatey\lib\NSSM\tools'

Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).



Configure a new service:


PS C:\Windows\system32> nssm install rclone-azure-blob

This will launch a NSSM service installer Window to configure your service:

To modify the service, you also can use "nssm edit rclone-azure-blob" command to modify it, such as log on:

It is important to make sure this service running under an admin privilege account, else, you will get an error.



Once it has done the installation, it will shows service installed successfully screen.


Now this new service should show in the services.msc :





It also works if you put following command into a bat file, then let NSSM launch that bat file as a service:


C:\tools\rclone.exe mount blob:rclone C:\tools\rclone-mount --vfs-cache-mode full
In this setting, the NSSM service window will look like this:




References


No comments:

Post a Comment