Use Aria2+AriaNg Docker to download and Use Rclone to Sync To Cloud Drives

This post is to record what i did to create a process to auto mount Google Drive and sync aria2's download files to Google drive's specific folder.

Topology

Run Aria2+AriaNG Docker

Regarding docker commands and usage, please visit post: https://blog.51sec.org/2020/04/docker-usage.html

Github Project : https://github.com/wahyd4/aria2-ariang-docker

Commands to install Docker on CentOS, Ubuntu.

root@linux:/# curl -sSL https://get.docker.com/ | sh

systemctl start docker
systemctl enable docker

Commands to run Aria2-ariang docker

docker run -d -i --restart=always --name ariang -p 8080:80  -p 6800:6800 -v ~/data/:/data -v /home/gdrive/:/gdrive wahyd4/aria2-ariang

Please make sure your VPS has those two ports opened: 8080 and 6800 opened to Internet.

Mapping more ports out:
docker run -d -i --restart=always --name ariang -p 8080:80 -p 6800:6800 -p 6881-6999:6881-6999 -p 6881-6999:6881-6999/udp -v ~/data/:/data -v /home/gdrive/:/gdrive wahyd4/aria2-ariang

Mapping conf folder to host:

[root@cenos1-download ~]# docker exec -it ariang /bin/bash
bash-4.3# cp -R conf /data
bash-4.3# ls
conf
bash-4.3# exit
exit
[root@cenos1-download ~]# docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                                   NAMES
d82418f696a3        wahyd4/aria2-ariang   "/bin/sh /root/conf/…"   3 minutes ago       Up 3 minutes        443/tcp, 0.0.0.0:6800->6800/tcp, 0.0.0.0:8000->80/tcp   ariang
[root@cenos1-download ~]# docker kill d82418f696a3
d82418f696a3
[root@cenos1-download ~]# docker rm d82418f696a3
d82418f696a3
[root@cenos1-download ~]# ls
data
[root@cenos1-download ~]# cd data
[root@cenos1-download data]# ls
conf
[root@cenos1-download data]# mv conf ~/
[root@cenos1-download data]# cd ..
[root@cenos1-download ~]# ls
conf  data
[root@cenos1-download ~]# docker run -d -i --restart=always --name ariang -p 8000:80  -p 6800:6800 -v ~/conf/:/root/conf -v ~/data/:/data -v /home/gdrive/:/gdrive wahyd4/aria2-ariang
f6711085e4209239f075c9f915ffee14b04e853b376f6ac5d54b3a615129d367
[root@cenos1-download ~]#

Enter into docker to make configuration change for aria2 service.

docker exec -it ariang /bin/bash

Inside the docker, create a shell script , rcloneupload.sh，with following code：

bash-4.3#vi rcloneupload.sh

#!/bin/bash

GID="$1";
FileNum="$2";
File="$3";
MinSize="5"  #限制最低上传大小，默认5k
MaxSize="157286400"  #限制最高文件大小(单位k)，默认15G
RemoteDIR="/gdrive/";  #rclone挂载的本地文件夹，最后面保留/
LocalDIR="/data/";  #Aria2下载目录，最后面保留/

if [[ -z $(echo "$FileNum" |grep -o '[0-9]*' |head -n1) ]]; then FileNum='0'; fi
if [[ "$FileNum" -le '0' ]]; then exit 0; fi
if [[ "$#" != '3' ]]; then exit 0; fi

function LoadFile(){
  IFS_BAK=$IFS
  IFS=$'\n'
  if [[ ! -d "$LocalDIR" ]]; then return; fi
  if [[ -e "$File" ]]; then
    FileLoad="${File/#$LocalDIR}"
    while true
      do
        if [[ "$FileLoad" == '/' ]]; then return; fi
        echo "$FileLoad" |grep -q '/';
        if [[ "$?" == "0" ]]; then
          FileLoad=$(dirname "$FileLoad");
        else
          break;
        fi;
      done;
    if [[ "$FileLoad" == "$LocalDIR" ]]; then return; fi
    EXEC="$(command -v mv)"
    if [[ -z "$EXEC" ]]; then return; fi
    Option=" -f";
    cd "$LocalDIR";
    if [[ -e "$FileLoad" ]]; then
      ItemSize=$(du -s "$FileLoad" |cut -f1 |grep -o '[0-9]*' |head -n1)
      if [[ -z "$ItemSize" ]]; then return; fi
      if [[ "$ItemSize" -le "$MinSize" ]]; then
        echo -ne "\033[33m$FileLoad \033[0mtoo small to spik.\n";
        return;
      fi
      if [[ "$ItemSize" -ge "$MaxSize" ]]; then
        echo -ne "\033[33m$FileLoad \033[0mtoo large to spik.\n";
        return;
      fi
      eval "${EXEC}${Option}" \'"${FileLoad}"\' "${RemoteDIR}";
    fi
  fi
  IFS=$IFS_BAK
}
LoadFile;

make file become executable: chmod +x rcloneupload.sh

Edit Aria2 configuration file (/root/conf/aria2.conf) to add one line at the file end:配置文件中加上一行on-download-complete=/root/rcloneupload.sh即可，后面为脚本的路径。最后重启Aria2生效。

bash-4.3# cd /root/conf/
bash-4.3# ls
aria2.conf aria2.session aria2c.sh key
bash-4.3# vi aria2.conf

# Bit Torrent: The amount of time and the upload-to-download ratio you wish to
# seed to. If either the time limit ( seconds ) or the seed ratio is reached,
# torrent seeding will stop. You can set seed-time to zero(0) to disable
# seeding completely.
seed-ratio=0.01
seed-time=1

https://gist.github.com/rudylacrete/dcd9779175fe18731852

Exit from docker to host server. Reboot docker ariang:

Docker restart ariang

Install Rclone

First to install epel source


yum -y install epel-release

Install some components


yum -y install wget unzip screen fuse fuse-devel

Install rclone


[root@centos7-test1 data]# curl https://rclone.org/install.sh | sudo bash

configure rclone


rclone config

first step is to choose n, then pick a name, which is google-drive in my case


No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> google-drive

choose 13 for your storage


Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Citrix Sharefile
   \ "sharefile"
 9 / Dropbox
   \ "dropbox"
10 / Encrypt/Decrypt a remote
   \ "crypt"
11 / FTP Connection
   \ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
13 / Google Drive
   \ "drive"
14 / Google Photos
   \ "google photos"

Storage> 13

just press enter for Google Application Client ID and client_secret


Google Application Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id> 
Google Application Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>

chosse 1 for your access to your drive


Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ "drive.metadata.readonly"
scope> 1

as to your folder id and json service account file, press enter to use default vaule.


ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders. (see docs).
Enter a string value. Press Enter for the default ("").
root_folder_id> 
Service Account Credentials JSON file path 
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Enter a string value. Press Enter for the default ("").
service_account_file>

N for advanced config.


Edit advanced config? (y/n)
y) Yes
n) No
y/n> n

Since we are working on a remote, n for auto config.


Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n

You will have to copy the link to log in to your Google account to get auth verification code:


If your browser doesn't open automatically go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=55663e7e07382e3dddb9025c86de4f
Log in and authorize rclone for access
Enter verification code> 4/UQGiRz375eb-OixO5EUtZMxBhJwAQ4zOyvA1wtJWKq2Ocmzh3zNYE

n for team drive.


Configure this as a team drive?
y) Yes
n) No
y/n> n

y to confirm your config


--------------------
[google-drive]
type = drive
scope = drive
token = {"access_token":"ya29.GlsQByNiBURlXoPpe-bDpa2kF99Jo4rrmjicBXdWIT6loPUhS7S_zpUwHn6no0Y_LBbXYFZvyVf0gRthepF2VuPFdhBFEKY7XYJaelt","token_type":"Bearer","refresh_token":"1/ry1JGhRiqqE6-PqRN-S2icZ_Oz9uOTXfSNxWA85zUnj-6TejL6o-hjyuY","expiry":"2019-05-21T04:36:23.300542043-04:00"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

q to exit.


Current remotes: 
Name                 Type
====                 ====
google-drive         drive
 
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

We finished the basic rclone config.

Now we will need to mount Google Drive to VPS and even it rebooted, it will still auto-mount Google Drive

create a new folder at /home/gdrive


mkdir -p /home/gdrive

mount system


rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes

google-drive is the Rclone configuration name.

You also can define a sub folder name:


rclone mount google-drive:backup /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes

google-drive:backup google-drive 为 Rclone 的配置名称：backup 为网盘里的目录名

unmount Google Drive - easiest way in a ssh session is to ctrl+c


fusermount -qzu /home/gdrive

[root@centos7-test1 data]# rclone ls google-drive:/

33196 3916278.html

1036266 69bbca83ly1gdr8plweo5g209e09yx6b.gif

42 test.test

Mount action usually take a couple seconds. You can open a second ssh session to check.


[opc@centos7-test1 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        462M     0  462M   0% /dev
tmpfs           494M     0  494M   0% /dev/shm
tmpfs           494M   14M  481M   3% /run
tmpfs           494M     0  494M   0% /sys/fs/cgroup
/dev/sda3        39G  3.2G   35G   9% /
/dev/sda1       512M   12M  501M   3% /boot/efi
tmpfs            99M     0   99M   0% /run/user/1000
google-drive:    15G  1.2G   14G   8% /home/gdrive

To unmount, simply press “CTRL+c” to stop the mount.

To make rclone mount the google drive even after rebooted the vps, create /usr/lib/systemd/system/rclone.service with following information:
vi /usr/lib/systemd/system/rclone.service


[Unit]
Description=rclone 
[Service]
User=root
ExecStart=/usr/bin/rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes
Restart=on-abort
 
[Install]
WantedBy=multi-user.target

systemctl start rclone
systemctl enable rclone

You also can only mount to Google Drive certain folder by changing root_folder_id at the file /root/.config/rclone/rclone.conf

[root@centos7-test1 rclone]# cat rclone.conf

[google-drive]

type = drive

scope = drive

token = {"access_token":"ya29.a0Ae4lvC3NjPpY5EmltR-TaIh2_I3et3TqsXxbr73XnS0Gj19zalB4xke2FXosB8kThTG8akkjqc94ZjOoY8sLOokoxZf26YrUf04F14abUGRaGZTWznSGB","token_type":"Bearer","refresh_token":"1//0fqzXgAuJUWU2CgYIARAGA8SNwF-L9IrsitYDMEfAeJgOutDZV7OyqaWLjeoaj9mDw-e-4beZlwucBY8Br32tQM5cXFl7BnyLo","expiry":"2020-05-08T20:00:53.118819799Z"}

root_folder_id = 1lMF61yERYbDDjV86VmxQk0mJjwBwCH

For root_folder_id, you can find it from the drive folder's url:

Notes for Rclone Usage

把文件上传到 Google Drive

为什么不直接把文件目录设置成挂载目录？如果直接把文件目录指定到挂载目录，会出现各种莫名其妙的错误，比如：文件无法写入、读取、保存到 Googlr Drive 的文件不完整等等奇葩的问题。这里可以用同步命令，本地目录/home/backup 同步到网盘的 backup 目录


rclone sync /home/backup gdrive:backup

相反，把目录调整下，就是把网盘 backup 目录同步到 VPS 目录/home/backup


rclone sync gdrive:backup /home/backup

通过添加此参数 --ignore-existing 可以忽略在网盘上已备份的文件，这相当于增量备份


rclone copy --ignore-existing /home/backup gdrive:backup

如果挂载 2 个网盘的话，可以同步配置名 gd2 的网盘里的 backup 目录，到配置名为 gd 的网盘的 backup 目录，反之亦然


rclone sync gdrive2:backup gdrive:backup

Command outputs


[root@centos7-test1 data]# curl https://rclone.org/install.sh | sudo bash
....
rclone v1.51.0 has successfully installed.
Now run "rclone config" for setup. Check https://rclone.org/docs/ for more details.

[root@centos7-test1 data]# rclone config
2020/05/05 22:02:56 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> google-drive
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
....
11 / FTP Connection
   \ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
13 / Google Drive
   \ "drive"
14 / Google Photos
   \ "google photos"
15 / Hubic
   \ "hubic"
....
Storage> 13
** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
Google Application Client Secret
Setting your own is recommended.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ "drive.metadata.readonly"
scope> 1
ID of the root folder
Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.

Note that if this is blank, the first time rclone runs it will fill it
in with the ID of the root folder.

Enter a string value. Press Enter for the default ("").
root_folder_id>
Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Enter a string value. Press Enter for the default ("").
service_account_file>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n
Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoath%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=wxylrgDHf7QnNy5zetieg
Log in and authorize rclone for access
Enter verification code> 4/zQEAebNYt68Rj1k2ckWuffBaL35WqVd-rJhf4AiCSU5MTqn-O_4ao                                                                                                                                                Configure this as a team drive?
y) Yes
n) No (default)
y/n> n
--------------------
[google-drive]
type = drive
scope = drive
token = {"access_token":"ya29.a0Ae4lvC1R7463egrz5gGDGFAwjA2elNfSs0T325er5U0FBHgw_B2pZSipzP9CXBjidavMM_cOHSm1MzRS5jxIL3wlFoXzj3eC2xcAWzJvlchzM95wOxoO6YRzu8j175S1DthpJwr6Zt_tR7Dg","token_type":"Bearer","refresh_token":"1//0fqzXgAuJUWU2CgYIRAAGA8SNwF-L9IrsitYDMEfAeJgOutDV7OyqaWLjeoaj9mDw-e-4beZlwucBY8Br32tQM5cXFl7enyLo","expiry":"2020-05-05T23:14:12.28655445Z"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
google-drive         drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
[root@centos7-test1 /]# mkdir -p /home/gdrive

[root@centos7-test1 /]# rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes
2020/05/05 22:15:48 Fatal error: failed to mount FUSE fs: fusermount: exec: "fusermount": executable file not found in $PATH
[root@centos7-test1 /]# yum install fuse

Install  1 Package

Total download size: 86 k
Installed size: 218 k
Is this ok [y/d/N]: y
Downloading packages:
fuse-2.9.2-11.el7.x86_64.rpm                                                                                                                                                                               |  86 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : fuse-2.9.2-11.el7.x86_64                                                                                                                                                                                       1/1
  Verifying  : fuse-2.9.2-11.el7.x86_64                                                                                                                                                                                       1/1

Installed:
  fuse.x86_64 0:2.9.2-11.el7

Complete!
[root@centos7-test1 /]# rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes

Latest

NETSEC

Sunday, June 14, 2020

Use Aria2+AriaNg Docker to download and Use Rclone to Sync To Cloud Drives

Topology

Run Aria2+AriaNG Docker

Install Rclone

Notes for Rclone Usage

Command outputs

No comments:

Post a Comment

Search

About + Social

NETSEC Blog

Categories

Popular

Recent Comments

51SEC Sites

Blog Archive

Translate

Pages

Contact Us

YouTube

Latest

NETSEC

Sunday, June 14, 2020

Use Aria2+AriaNg Docker to download and Use Rclone to Sync To Cloud Drives

Topology

Run Aria2+AriaNG Docker

Install Rclone

Notes for Rclone Usage

Command outputs

Subscribe via email

No comments:

Post a Comment

Search

About + Social

NETSEC Blog

Categories

Popular

Recent Comments

51SEC Sites

Blog Archive

Translate

Pages

Contact Us

YouTube