Use Aria2+AriaNg Docker to download and Use Rclone to Sync To Cloud Drives - NETSEC

Latest

Learning, Sharing, Creating

Cybersecurity Memo

Sunday, June 14, 2020

Use Aria2+AriaNg Docker to download and Use Rclone to Sync To Cloud Drives

This post is to record what i did to create a process to auto mount Google Drive and sync aria2's download files to Google drive's specific folder.





Run Aria2+AriaNG Docker

Regarding docker commands and usage, please visit post: https://blog.51sec.org/2020/04/docker-usage.html

Github Project : https://github.com/wahyd4/aria2-ariang-docker

Commands to install Docker on CentOS, Ubuntu.

root@linux:/# curl -sSL https://get.docker.com/ | sh
systemctl start docker
systemctl enable docker
Commands to run Aria2-ariang docker
docker run -d -i --restart=always --name ariang -p 8080:80  -p 6800:6800 -v ~/data/:/data -v /home/gdrive/:/gdrive wahyd4/aria2-ariang
Please make sure your VPS has those two ports opened: 8080 and 6800 opened to Internet.

Mapping more ports out:
docker run -d -i --restart=always --name ariang -p 8080:80  -p 6800:6800 -p 6881-6999:6881-6999 -p 6881-6999:6881-6999/udp -v ~/data/:/data -v /home/gdrive/:/gdrive wahyd4/aria2-ariang





Mapping conf folder to host:
[root@cenos1-download ~]# docker exec -it ariang /bin/bash
bash-4.3# cp -R conf /data
bash-4.3# ls
conf
bash-4.3# exit
exit
[root@cenos1-download ~]# docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                                   NAMES
d82418f696a3        wahyd4/aria2-ariang   "/bin/sh /root/conf/…"   3 minutes ago       Up 3 minutes        443/tcp, 0.0.0.0:6800->6800/tcp, 0.0.0.0:8000->80/tcp   ariang
[root@cenos1-download ~]# docker kill d82418f696a3
d82418f696a3
[root@cenos1-download ~]# docker rm d82418f696a3
d82418f696a3
[root@cenos1-download ~]# ls
data
[root@cenos1-download ~]# cd data
[root@cenos1-download data]# ls
conf
[root@cenos1-download data]# mv conf ~/
[root@cenos1-download data]# cd ..
[root@cenos1-download ~]# ls
conf  data
[root@cenos1-download ~]# docker run -d -i --restart=always --name ariang -p 8000:80  -p 6800:6800 -v ~/conf/:/root/conf -v ~/data/:/data -v /home/gdrive/:/gdrive wahyd4/aria2-ariang
f6711085e4209239f075c9f915ffee14b04e853b376f6ac5d54b3a615129d367
[root@cenos1-download ~]#


Enter into docker to make configuration change for aria2 service.
docker exec -it ariang /bin/bash

Inside the docker, create a shell script , rcloneupload.sh,with following code:
bash-4.3#vi rcloneupload.sh
#!/bin/bash

GID="$1";
FileNum="$2";
File="$3";
MinSize="5"  #限制最低上传大小,默认5k
MaxSize="157286400"  #限制最高文件大小(单位k),默认15G
RemoteDIR="/gdrive/";  #rclone挂载的本地文件夹,最后面保留/
LocalDIR="/data/";  #Aria2下载目录,最后面保留/

if [[ -z $(echo "$FileNum" |grep -o '[0-9]*' |head -n1) ]]; then FileNum='0'; fi
if [[ "$FileNum" -le '0' ]]; then exit 0; fi
if [[ "$#" != '3' ]]; then exit 0; fi

function LoadFile(){
  IFS_BAK=$IFS
  IFS=$'\n'
  if [[ ! -d "$LocalDIR" ]]; then return; fi
  if [[ -e "$File" ]]; then
    FileLoad="${File/#$LocalDIR}"
    while true
      do
        if [[ "$FileLoad" == '/' ]]; then return; fi
        echo "$FileLoad" |grep -q '/';
        if [[ "$?" == "0" ]]; then
          FileLoad=$(dirname "$FileLoad");
        else
          break;
        fi;
      done;
    if [[ "$FileLoad" == "$LocalDIR" ]]; then return; fi
    EXEC="$(command -v mv)"
    if [[ -z "$EXEC" ]]; then return; fi
    Option=" -f";
    cd "$LocalDIR";
    if [[ -e "$FileLoad" ]]; then
      ItemSize=$(du -s "$FileLoad" |cut -f1 |grep -o '[0-9]*' |head -n1)
      if [[ -z "$ItemSize" ]]; then return; fi
      if [[ "$ItemSize" -le "$MinSize" ]]; then
        echo -ne "\033[33m$FileLoad \033[0mtoo small to spik.\n";
        return;
      fi
      if [[ "$ItemSize" -ge "$MaxSize" ]]; then
        echo -ne "\033[33m$FileLoad \033[0mtoo large to spik.\n";
        return;
      fi
      eval "${EXEC}${Option}" \'"${FileLoad}"\' "${RemoteDIR}";
    fi
  fi
  IFS=$IFS_BAK
}
LoadFile;
make file become executable: chmod +x rcloneupload.sh
Edit Aria2 configuration file (/root/conf/aria2.conf) to add one line at the file end:配置文件中加上一行on-download-complete=/root/rcloneupload.sh即可,后面为脚本的路径。最后重启Aria2生效。

bash-4.3# cd /root/conf/
bash-4.3# ls
aria2.conf      aria2.session      aria2c.sh      key
bash-4.3# vi aria2.conf
# Bit Torrent: The amount of time and the upload-to-download ratio you wish to
# seed to. If either the time limit ( seconds ) or the seed ratio is reached,
# torrent seeding will stop. You can set seed-time to zero(0) to disable
# seeding completely.
 seed-ratio=0.01
 seed-time=1

https://gist.github.com/rudylacrete/dcd9779175fe18731852


Exit from docker to host server. Reboot docker ariang:
Docker restart ariang


Install Rclone

First to install epel source
  1. yum -y install epel-release
Install some components
  1. yum -y install wget unzip screen fuse fuse-devel
Install rclone
  1. [root@centos7-test1 data]# curl https://rclone.org/install.sh | sudo bash


configure rclone
  1. rclone config
first step is to choose n, then pick a name, which is google-drive in my case
  1. No remotes found - make a new one
  2. n) New remote
  3. s) Set configuration password
  4. q) Quit config
  5. n/s/q> n
  6. name> google-drive
choose 13 for your storage 
  1. Type of storage to configure.
  2. Enter a string value. Press Enter for the default ("").
  3. Choose a number from below, or type in your own value
  4. 1 / 1Fichier \ "fichier" 2 / Alias for an existing remote \ "alias" 3 / Amazon Drive \ "amazon cloud drive" 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) \ "s3" 5 / Backblaze B2 \ "b2" 6 / Box \ "box" 7 / Cache a remote \ "cache" 8 / Citrix Sharefile \ "sharefile" 9 / Dropbox \ "dropbox" 10 / Encrypt/Decrypt a remote \ "crypt" 11 / FTP Connection \ "ftp" 12 / Google Cloud Storage (this is not Google Drive) \ "google cloud storage" 13 / Google Drive \ "drive" 14 / Google Photos \ "google photos"
  5. Storage> 13
just press enter for Google Application Client ID and client_secret
  1. Google Application Client Id
  2. Leave blank normally.
  3. Enter a string value. Press Enter for the default ("").
  4. client_id>
  5. Google Application Client Secret
  6. Leave blank normally.
  7. Enter a string value. Press Enter for the default ("").
  8. client_secret>
chosse 1 for your access to your drive
  1. Scope that rclone should use when requesting access from drive.
  2. Enter a string value. Press Enter for the default ("").
  3. Choose a number from below, or type in your own value
  4. 1 / Full access all files, excluding Application Data Folder.
  5. \ "drive"
  6. 2 / Read-only access to file metadata and file contents.
  7. \ "drive.readonly"
  8. / Access to files created by rclone only.
  9. 3 | These are visible in the drive website.
  10. | File authorization is revoked when the user deauthorizes the app.
  11. \ "drive.file"
  12. / Allows read and write access to the Application Data folder.
  13. 4 | This is not visible in the drive website.
  14. \ "drive.appfolder"
  15. / Allows read-only access to file metadata but
  16. 5 | does not allow any access to read or download file content.
  17. \ "drive.metadata.readonly"
  18. scope> 1
as to your folder id and json service account file, press enter to use default vaule. 
  1. ID of the root folder
  2. Leave blank normally.
  3. Fill in to access "Computers" folders. (see docs).
  4. Enter a string value. Press Enter for the default ("").
  5. root_folder_id>
  6. Service Account Credentials JSON file path
  7. Leave blank normally.
  8. Needed only if you want use SA instead of interactive login.
  9. Enter a string value. Press Enter for the default ("").
  10. service_account_file>
 N for advanced config.
  1. Edit advanced config? (y/n)
  2. y) Yes
  3. n) No
  4. y/n> n
Since we are working on a remote, n for auto config.
  1. Remote config
  2. Use auto config?
  3. * Say Y if not sure
  4. * Say N if you are working on a remote or headless machine
  5. y) Yes
  6. n) No
  7. y/n> n
You will have to copy the link to log in to your Google account to get auth verification code:
  1. If your browser doesn't open automatically go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=55663e7e07382e3dddb9025c86de4f
  2. Log in and authorize rclone for access
  3. Enter verification code> 4/UQGiRz375eb-OixO5EUtZMxBhJwAQ4zOyvA1wtJWKq2Ocmzh3zNYE
n for team drive. 
  1. Configure this as a team drive?
  2. y) Yes
  3. n) No
  4. y/n> n
 y to confirm your config
  1. --------------------
  2. [google-drive]
  3. type = drive
  4. scope = drive
  5. token = {"access_token":"ya29.GlsQByNiBURlXoPpe-bDpa2kF99Jo4rrmjicBXdWIT6loPUhS7S_zpUwHn6no0Y_LBbXYFZvyVf0gRthepF2VuPFdhBFEKY7XYJaelt","token_type":"Bearer","refresh_token":"1/ry1JGhRiqqE6-PqRN-S2icZ_Oz9uOTXfSNxWA85zUnj-6TejL6o-hjyuY","expiry":"2019-05-21T04:36:23.300542043-04:00"}
  6. --------------------
  7. y) Yes this is OK
  8. e) Edit this remote
  9. d) Delete this remote
  10. y/e/d> y
q to exit.
  1. Current remotes: 
  2. Name Type
  3. ==== ====
  4. google-drive drive
  5.  
  6. e) Edit existing remote
  7. n) New remote
  8. d) Delete remote
  9. r) Rename remote
  10. c) Copy remote
  11. s) Set configuration password
  12. q) Quit config
  13. e/n/d/r/c/s/q> q
We finished the basic rclone config. 

Now we will need to mount  Google Drive to VPS and even it rebooted, it will still auto-mount Google Drive
create a new folder at /home/gdrive
  1. mkdir -p /home/gdrive
mount system
  1. rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes
google-drive is the Rclone configuration name.
You also can define a sub folder name:
  1. rclone mount google-drive:backup /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes
google-drive:backup google-drive 为 Rclone 的配置名称:backup 为网盘里的目录名
unmount Google Drive  - easiest way in a ssh session is to ctrl+c
  1. fusermount -qzu /home/gdrive

[root@centos7-test1 data]# rclone ls google-drive:/
    33196 3916278.html
  1036266 69bbca83ly1gdr8plweo5g209e09yx6b.gif
       42 test.test


Mount action usually take a couple seconds. You can open a second ssh session to check. 
  1. [opc@centos7-test1 ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 462M 0 462M 0% /dev tmpfs 494M 0 494M 0% /dev/shm tmpfs 494M 14M 481M 3% /run tmpfs 494M 0 494M 0% /sys/fs/cgroup /dev/sda3 39G 3.2G 35G 9% / /dev/sda1 512M 12M 501M 3% /boot/efi tmpfs 99M 0 99M 0% /run/user/1000 google-drive: 15G 1.2G 14G 8% /home/gdrive

To unmount, simply press “CTRL+c” to stop the mount.

To make rclone mount the google drive even after rebooted the vps, create /usr/lib/systemd/system/rclone.service with following information:
vi /usr/lib/systemd/system/rclone.service
  1. [Unit]
  2. Description=rclone 
  3. [Service]
  4. User=root
  5. ExecStart=/usr/bin/rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes
  6. Restart=on-abort
  7.  
  8. [Install]
  9. WantedBy=multi-user.target

systemctl start rclone
systemctl enable rclone

You also can only mount to Google Drive certain folder by changing root_folder_id at the file /root/.config/rclone/rclone.conf

[root@centos7-test1 rclone]# cat rclone.conf
[google-drive]
type = drive
scope = drive
token = {"access_token":"ya29.a0Ae4lvC3NjPpY5EmltR-TaIh2_I3et3TqsXxbr73XnS0Gj19zalB4xke2FXosB8kThTG8akkjqc94ZjOoY8sLOokoxZf26YrUf04F14abUGRaGZTWznSGB","token_type":"Bearer","refresh_token":"1//0fqzXgAuJUWU2CgYIARAGA8SNwF-L9IrsitYDMEfAeJgOutDZV7OyqaWLjeoaj9mDw-e-4beZlwucBY8Br32tQM5cXFl7BnyLo","expiry":"2020-05-08T20:00:53.118819799Z"}
root_folder_id = 1lMF61yERYbDDjV86VmxQk0mJjwBwCH
For root_folder_id, you can find it from the drive folder's url:






Notes for Rclone Usage


    把文件上传到 Google Drive
    为什么不直接把文件目录设置成挂载目录?如果直接把文件目录指定到挂载目录,会出现各种莫名其妙的错误,比如:文件无法写入、读取、保存到 Googlr Drive 的文件不完整等等奇葩的问题。这里可以用同步命令,本地目录/home/backup 同步到网盘的 backup 目录
    1. rclone sync /home/backup gdrive:backup
    相反,把目录调整下,就是把网盘 backup 目录同步到 VPS 目录/home/backup
    1. rclone sync gdrive:backup /home/backup
    通过添加此参数 --ignore-existing 可以忽略在网盘上已备份的文件,这相当于增量备份
    1. rclone copy --ignore-existing /home/backup gdrive:backup
    如果挂载 2 个网盘的话,可以同步配置名 gd2 的网盘里的 backup 目录,到配置名为 gd 的网盘的 backup 目录,反之亦然
    1. rclone sync gdrive2:backup gdrive:backup


    Command outputs


      1. [root@centos7-test1 data]# curl https://rclone.org/install.sh | sudo bash .... rclone v1.51.0 has successfully installed. Now run "rclone config" for setup. Check https://rclone.org/docs/ for more details. [root@centos7-test1 data]# rclone config 2020/05/05 22:02:56 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> google-drive Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / 1Fichier \ "fichier" .... 11 / FTP Connection \ "ftp" 12 / Google Cloud Storage (this is not Google Drive) \ "google cloud storage" 13 / Google Drive \ "drive" 14 / Google Photos \ "google photos" 15 / Hubic \ "hubic" .... Storage> 13 ** See help for drive backend at: https://rclone.org/drive/ ** Google Application Client Id Setting your own is recommended. See https://rclone.org/drive/#making-your-own-client-id for how to create your own. If you leave this blank, it will use an internal key which is low performance. Enter a string value. Press Enter for the default (""). client_id> Google Application Client Secret Setting your own is recommended. Enter a string value. Press Enter for the default (""). client_secret> Scope that rclone should use when requesting access from drive. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Full access all files, excluding Application Data Folder. \ "drive" 2 / Read-only access to file metadata and file contents. \ "drive.readonly" / Access to files created by rclone only. 3 | These are visible in the drive website. | File authorization is revoked when the user deauthorizes the app. \ "drive.file" / Allows read and write access to the Application Data folder. 4 | This is not visible in the drive website. \ "drive.appfolder" / Allows read-only access to file metadata but 5 | does not allow any access to read or download file content. \ "drive.metadata.readonly" scope> 1 ID of the root folder Leave blank normally. Fill in to access "Computers" folders (see docs), or for rclone to use a non root folder as its starting point. Note that if this is blank, the first time rclone runs it will fill it in with the ID of the root folder. Enter a string value. Press Enter for the default (""). root_folder_id> Service Account Credentials JSON file path Leave blank normally. Needed only if you want use SA instead of interactive login. Enter a string value. Press Enter for the default (""). service_account_file> Edit advanced config? (y/n) y) Yes n) No (default) y/n> n Remote config Use auto config? * Say Y if not sure * Say N if you are working on a remote or headless machine y) Yes (default) n) No y/n> n Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoath%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=wxylrgDHf7QnNy5zetieg Log in and authorize rclone for access Enter verification code> 4/zQEAebNYt68Rj1k2ckWuffBaL35WqVd-rJhf4AiCSU5MTqn-O_4ao Configure this as a team drive? y) Yes n) No (default) y/n> n -------------------- [google-drive] type = drive scope = drive token = {"access_token":"ya29.a0Ae4lvC1R7463egrz5gGDGFAwjA2elNfSs0T325er5U0FBHgw_B2pZSipzP9CXBjidavMM_cOHSm1MzRS5jxIL3wlFoXzj3eC2xcAWzJvlchzM95wOxoO6YRzu8j175S1DthpJwr6Zt_tR7Dg","token_type":"Bearer","refresh_token":"1//0fqzXgAuJUWU2CgYIRAAGA8SNwF-L9IrsitYDMEfAeJgOutDV7OyqaWLjeoaj9mDw-e-4beZlwucBY8Br32tQM5cXFl7enyLo","expiry":"2020-05-05T23:14:12.28655445Z"} -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current remotes: Name Type ==== ==== google-drive drive e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q [root@centos7-test1 /]# mkdir -p /home/gdrive [root@centos7-test1 /]# rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes 2020/05/05 22:15:48 Fatal error: failed to mount FUSE fs: fusermount: exec: "fusermount": executable file not found in $PATH [root@centos7-test1 /]# yum install fuse Install 1 Package Total download size: 86 k Installed size: 218 k Is this ok [y/d/N]: y Downloading packages: fuse-2.9.2-11.el7.x86_64.rpm | 86 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : fuse-2.9.2-11.el7.x86_64 1/1 Verifying : fuse-2.9.2-11.el7.x86_64 1/1 Installed: fuse.x86_64 0:2.9.2-11.el7 Complete! [root@centos7-test1 /]# rclone mount google-drive: /home/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes




      No comments:

      Post a Comment