Comments

Latest Posts

Build Cloud File Download & Manage Site using FileBrowser+Aria2+AriaNg+Rclone+Caddy+Google Drive

This post is regarding how to use wahyd4/aria2-ui's docker image to achieve a downloading and file managing / sharing center. In just one docker image, you can do downloading (Aria2), Web GUI management for your downloading task (AriaNG) , File Managing (FileBrowser), supporting multi-type file view, and saving to Google Drive. 



Topology










Pre-requirement:

Install Docker, Portainer, Nginx:


Open Firewall port at cloud firewall and local firewall if enabled:
  • for tcp 8000
#CentOS 6
iptables -A INPUT -p tcp --dport 8000 -j ACCEPT
service iptables save
service iptables restart

#CentOS 7
firewall-cmd --zone=public --add-port=8000/tcp --permanent
firewall-cmd --reload


Github Project and Docker Image Information


Github project : 

Docker images:

Other related GitHub Sites:




Features

  • Aria2 (SSL support)
  • AriaNg
  • Rclone
  • File Browser: Files mangement and videos playing
  • Auto HTTPS (Let's Encrypt)
  • Bind non root user into container, so non root user can also manage downloaded files.
  • Basic Auth
  • Support ARM CPUs as well, all supported CPU platforms can be found here
  • Cloud Storage platforms synchronization



Quick Run for Testing



docker run -d --name aria2-ui -p 80:80 wahyd4/aria2-ui
Here are default URL and user name / password. 
  • Aria2: http://yourip
  • FileManger: http://yourip/files
  • Rclone: http://yourip/rclone
  • Please use admin/admin as username and password to login Filebrowser for the first time. And if rclone enabled, please use user/password to login Rclone if you didn't update ARIA2_USER and ARIA2_PWD
  • ENABLE_RCLONE set to false to disable Rclone in the docker


Modified Docker Run Command Based Requirements

Quick run is not exactly what we want. For my configuration, I would like to put this docker into different port, 8000, also I will need to map my host folders into docker. No need rclone to run in the docker, I will get it running on my host to mount my Google Drive. Also I will need some kind of authentication to be enabled for AriaNG web gui. 

  1. Change Docker Port to 8000 since Nginx using port 80 & 443
  2. Enable Basic Authentication
  3. After completed downloading, file will be moved to Cloud Drive, such as Google Drive
  4. Nginx will be used as reverse porxy for domain access downloading site

docker run -d --name aria2-ui \
  --restart=unless-stopped \
  -p 8000:80 \
  -e ENABLE_AUTH=true \
  -e ARIA2_SSL=false \
  -e ARIA2_USER=user \
  -e ARIA2_PWD=password \
  -e ARIA2_EXTERNAL_PORT=8000 \
  -e ENABLE_RCLONE=False \
  -v ~/data:/data \
  -v ~/gdrive:/gdrive \
  wahyd4/aria2-ui


Notes:
  • Both data and gdrive folders will be automatically created under host's /root folder
  • /data folder is aria2 docker's downloading folder. After finished downloading, file will be transferred to /gdrive folder automatically with a script.
  • /gdrive folder is host's Rclone mounted Google Drive folder, which is mapped to ~/gdrive folder (/root/gdrive). 


Configure FileBrowser to Manage Gdrive folder (Optional)

This is an option to do if you would like FileBrowser web gui to manage your mounted google drive. Default it is managing download folder /data. 

Change File Browser's folder. vi /app/Procfile
Change default /data folder to /gdrive. So File Browser will open /gdrive when you log in. You can mange your Google Drive from File Browser directly. It is not necessary to do this step if you does not want to manage your Google Drive folder from FileBrowser page. 

[[email protected] ~]docker exec -it aria2-ui /bin/bash

bash-5.0# ls
Procfile aria2c.sh caddy.sh conf filebrowser filebrowser.db forego init.sh start.sh
bash-5.0# vi Procfile
filebrowser: /app/filebrowser -p 8080 -d /app/filebrowser.db -r /gdrive
caddy: /app/caddy.sh
aria2c: /app/aria2c.sh

[[email protected] ~]# docker restart aria2-ui

bash-5.0# cd /app


Configure Aria2 to Move Completed File to /Gdrive

[[email protected] ~]docker exec -it aria2-ui /bin/bash

Inside the docker, create a shell script , vi /app/conf/rcloneupload.sh,with following code:


bash-4.3#vi /app/conf/rcloneupload.sh
#!/bin/bash

GID="$1";
FileNum="$2";
File="$3";
MinSize="5"  #限制最低上传大小,默认5k
MaxSize="157286400"  #限制最高文件大小(单位k),默认15G
RemoteDIR="/gdrive/";  #rclone挂载的本地文件夹,最后面保留/
LocalDIR="/data/";  #Aria2下载目录,最后面保留/

if [[ -z $(echo "$FileNum" |grep -o '[0-9]*' |head -n1) ]]; then FileNum='0'; fi
if [[ "$FileNum" -le '0' ]]; then exit 0; fi
if [[ "$#" != '3' ]]; then exit 0; fi

function LoadFile(){
  IFS_BAK=$IFS
  IFS=$'\n'
  if [[ ! -d "$LocalDIR" ]]; then return; fi
  if [[ -e "$File" ]]; then
    FileLoad="${File/#$LocalDIR}"
    while true
      do
        if [[ "$FileLoad" == '/' ]]; then return; fi
        echo "$FileLoad" |grep -q '/';
        if [[ "$?" == "0" ]]; then
          FileLoad=$(dirname "$FileLoad");
        else
          break;
        fi;
      done;
    if [[ "$FileLoad" == "$LocalDIR" ]]; then return; fi
    EXEC="$(command -v mv)"
    if [[ -z "$EXEC" ]]; then return; fi
    Option=" -f";
    cd "$LocalDIR";
    if [[ -e "$FileLoad" ]]; then
      ItemSize=$(du -s "$FileLoad" |cut -f1 |grep -o '[0-9]*' |head -n1)
      if [[ -z "$ItemSize" ]]; then return; fi
      if [[ "$ItemSize" -le "$MinSize" ]]; then
        echo -ne "\033[33m$FileLoad \033[0mtoo small to spik.\n";
        return;
      fi
      if [[ "$ItemSize" -ge "$MaxSize" ]]; then
        echo -ne "\033[33m$FileLoad \033[0mtoo large to spik.\n";
        return;
      fi
      eval "${EXEC}${Option}" \'"${FileLoad}"\' "${RemoteDIR}";
    fi
  fi
  IFS=$IFS_BAK
}
LoadFile;
make file become executable: chmod +x /app/conf/rcloneupload.sh
Edit Aria2 configuration file (vi /app/conf/aria2.conf) to add one following line at the file end: ($ in vi is to move to the end of line)
on-download-complete=/app/conf/rcloneupload.sh

Restart aria2-ui docker to take change into effect
[[email protected] ~]# docker restart aria2-ui
bash-4.3# cd /app/conf/
bash-4.3# ls
aria2.conf      aria2.session      aria2c.sh      key

bash-4.3# vi /app/conf/aria2.conf
# Bit Torrent: The amount of time and the upload-to-download ratio you wish to
# seed to. If either the time limit ( seconds ) or the seed ratio is reached,
# torrent seeding will stop. You can set seed-time to zero(0) to disable
# seeding completely.
seed-ratio=0.01
seed-time=1
on-download-complete=/app/conf/rcloneupload.sh


Configure Host Rclone to mount Google Drive

1   First to install epel source
yum -y install epel-release

Note:In Ubuntu, just use this command to install rclone: apt install rclone

2  Install some components
yum -y install wget unzip screen fuse fuse-devel


3  Install rclone
[[email protected] data]# curl https://rclone.org/install.sh | sudo bash

In Ubuntu, apt install rclone

4  configure rclone
rclone config

create a new remote configuration and name it as google-drive

Other steps for rclone configuration, you can check following posts: 


5  mount Google Drive using rclone
  • create a new folder at /root/gdrive
    • note: in our previous settings,  /root/gdrive has been created. 
mkdir -p /root/gdrive

  • mount system
rclone mount google-drive: /root/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes

Note: google-drive is the Rclone configuration name.


6   create rclone.service

To make rclone mount the google drive even after rebooted the vps, create /usr/lib/systemd/system/rclone.service with following information:
vi /usr/lib/systemd/system/rclone.service

[Unit]
Description=rclone

[Service]
User=root
ExecStart=/usr/bin/rclone mount google-drive: /root/gdrive --allow-other --allow-non-empty --vfs-cache-mode writes
Restart=on-abort


7  systemctl enable rclone.service

Strongly suggest to reboot system to confirm it is working after a reboot. 

8  Using browser to access system

FileBrowser: http://ip/files
aria2ng: http://ip

Default FileBrowser username/password will be admin/admin.

Aria will download files to docker's local /data folder . Once completed downloading, a script will be trigged to move the file to /gdrive folder.

Gdrive folder has been mounted with Google Drive. You can configure filebrowser to manage either temp download folder /data, or Google Drive folder /gdrive.


FileBrowse Page

http://download.51sec.eu.org/files/

It will take a while to show you all files since they are in your Rclone's connection to Google Drive. 
Edit users or add users into the Filebrowser. Default usernmae and password is admin/admin. 



Cloudflare Configuration


DNS A record should set to no proxy, else you will have a problem to get your AriaNG to connect to Aria2's rpc port 8000.

You will need to add a new configuration file into your Nginx. It can be copied from portainer.conf file. 

AriaNG Configuration

You will not need to make any change here as long as you disabled proxy status from Cloudflare. But if there is a problem with AriaNG connectingto Aria2 RPC port, you might need to make some changes here, such changing from domain name to ip address.






Remove Installed Docker

#Delete Docker ContainerID=`docker ps|grep wahyd4/aria2-ui|awk '{print $1}'` docker kill ${ContainerID} docker rm ${ContainerID} docker rmi `docker images|grep wahyd4/aria2-ui|awk '{print $3}'` #Remove Download Folder rm -rf ~/data
rm -rf /home/gdrive



There is another Github project which has Aria2 Pro docker available for creating a similar downloading site. 

Install Aria2 Pro docker



GitHub: https://github.com/P3TERX/docker-aria2-pro
Docker Hub: https://hub.docker.com/r/p3terx/aria2-proRelated Post: https://p3terx.com/archives/docker-aria2-pro.html


docker run -d \
    --name aria2-pro \
    --restart unless-stopped \
    --log-opt max-size=1m \
    --network host \
    -e PUID=$UID \
    -e PGID=$GID \
    -e RPC_SECRET=P3TERX \
    -e RPC_PORT=6800 \
    -e LISTEN_PORT=6888 \
    -v ~/aria2-config:/config \
    -v ~/rclone-downloads:/downloads \
    -e SPECIAL_MODE=rclone \
    p3terx/aria2-pro
  • --name aria2-pro - 容器名称,可自定义以示区分。
  • --restart unless-stopped - 设置容器重启策略,详情参见 Docker 官方文档
  • --log-driver json-file - 设置日志记录格式为 json 格式。这是 Docker 的默认值,某些特殊情况可能需要设置。
  • --log-opt max-size=1m - 日志大小限制为1MB,防止 Aria2 持续下载产生大量的日志占用磁盘空间。某些 GUI 可能没有相关选项。所以说有什么理由不用 CLI 一把梭?
  • --network host - 使用 host 网络模式。直接使用宿主机网络,免去端口映射导致的部分性能损失,且灵活性更高,可更方便的配置使用 IPv6 网络。host 网络模式仅适用于 Docker 17.06+ ,如果你的 Docker 版本低于此,请先升级。

  • 启动命令加入-e SPECIAL_MODE=rclone参数设定特殊模式环境变量后开启 RCLONE 自动上传功能,容器初次启动会安装 RCLONE ,且每次重启会自动更新 RCLONE


重启

Aria2 Pro 具有自我修复机制,遇到问题首先重启。如果你修改过配置文件和附加功能脚本,先删除后重启。

docker restart aria2-pro

重装

docker rm -f aria2-pro
docker rmi p3terx/aria2-pro
rm -rf ~/aria2-config
docker pull p3terx/aria2-pro
docker run <...>

查看日志

查看日志才能更好的找到问题的根本,即使你看不懂,也要学会如何查看。

  • 查看实时日志
docker logs -f --tail 30 aria2-pro
  • 导出日志
docker logs aria2-pro > ~/aria2-pro.log


YouTube Videos











References




2 comments:

  1. yes, done. Part 1: https://youtu.be/ewquhq_MjWc
    Part 2: https://youtu.be/E4qZff2iKDM

    ReplyDelete