[5 Mins Docker] Transfer Any Files Out from Command Line - Create Your Own Safe and Fast Temporary File Transfer Site - NETSEC

Latest

NETSEC

Learning, Sharing, Creating

Cybersecurity Memo

Sunday, October 8, 2023

[5 Mins Docker] Transfer Any Files Out from Command Line - Create Your Own Safe and Fast Temporary File Transfer Site

Have you met this issue? You are in a production environment with Internet connection, but you are not able to pull the logs or captured packets out from a hardened server for troubleshooting purpose. 

Here is a website which can help you in this situation. You do not have to open firewall to allow sftp connection to grab the files, or turn on a web service to give yourself a link to download them. As long as your servers can reach out to internet, you will be able safely to transfer your files out.

Transfer.sh is this kind of website which can give you a simple, easy and fast service for file sharing from the command-line. It is free and allows you to upload up to 10GB of data and files are stored for 14 days.



Github: https://github.com/dutchcoders/transfer.sh

Related blog post: https://blog.51sec.org/2024/04/5-mins-docker-your-paste-file-share.html


Features


  • Made for use with shell
  • Share files with a URL
  • Upload up to 10 GB
  • Files stored for 14 days
  • For free
  • Encrypt your files
  • Maximize amount of downloads
  • Self hosting using docker
  • Support popular cloud storages such as AWS S3, Google Drive, etc
  • More on Github site (https://github.com/dutchcoders/transfer.sh)
Notes:
[Arp 2024] It seems transfer.sh site is dwon now. Found another project, linx-server - https://put.icu/ - https://github.com/ZizzyDizzyMC/linx-server, which is similar. Currently, I am using put.icu site as alterative. 

Steps to Use Transfer.sh

There are quite a lot of use cases, which are listing at transfer.sh websites. You can upload multiple files and encrypt it, you can use wget to upload, and you also can use it from Windows PowerShell. 

In this section, I am gonna listing the steps for a simple use case, which is to just upload one file from your linux server with default settings. 

1 Log into Linux system


2 Upload a file

Upload a single file with default settings

curl -v --upload-file ./hello.txt https://transfer.sh/hello.txt



[node1] (local) [email protected] ~
$ curl -v --upload-file /docker.log https://transfer.sh/hello.txt
*   Trying 144.76.136.153:443...
* Connected to transfer.sh (144.76.136.153) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=transfer.sh
*  start date: Aug 14 19:42:51 2023 GMT
*  expire date: Nov 12 19:42:50 2023 GMT
*  subjectAltName: host "transfer.sh" matched cert's "transfer.sh"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* h2 [:method: PUT]
* h2 [:scheme: https]
* h2 [:authority: transfer.sh]
* h2 [:path: /hello.txt]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* h2 [content-length: 34559]
* Using Stream ID: 1 (easy handle 0x7fd872c60080)
> PUT /hello.txt HTTP/2
> Host: transfer.sh
> User-Agent: curl/8.1.2
> Accept: */*
> Content-Length: 34559
> 
* We are completely uploaded and fine
< HTTP/2 200 
< content-type: text/plain
< server: Transfer.sh HTTP Server
< x-made-with: <3 by DutchCoders
< x-served-by: Proudly served by DutchCoders
< x-url-delete: https://transfer.sh/xVaLXBfwcD/hello.txt/wx9PGrancWLzyeSV55EU
< content-length: 40
< date: Sun, 08 Oct 2023 15:55:01 GMT
< 
* Connection #0 to host transfer.sh left intact
https://transfer.sh/xVaLXBfwcD/hello.txt[node1] (local) [email protected] ~
$ 


3 Open the URL from any browser to fetch the file




Self Hosted Docker Installation Pre-requirements

Free resources you might need to complete this docker project:


Pre-installed services:

  • Docker, 
    • apt update
    • apt install docker.io
    • apt install docker-compose
    • apt upgrade docker.io
    • mkdir /root/data/docker_data/<docker_name>
  • Docker-Compose (Using Ubuntu OS for the commands)
    • Docker-compose down
    • Optional command : use following command to backup your Docker data. You might need to change your folder name based on your docker configuraiton
      • cp -r /root/data/docker_data/<docker_name> /root/data/docker_data_backup/<docker_name>
    • docker-compose pull
    • docker-compose up -d
    • docker image prune
  • Portainer (Optional)
    • docker volume create portainer_data
    • docker run -d -p 9000:9000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
  • Install some applications: apt install wget curl sudo vim git (Optional)
  • aapanel with Nginx (Optional)
  • Nginx Proxy Manager (Optional)
  • Install screen (Optional)
    • Install screen (Depends on the Linux Distribution if it came pre installed or not) : yum install screen
    • Initiate a Screen : screen or  screen -S <screen name> <command to execute>
    • Detach from the screen : "CTRL+A,D" not "CTRL+A+D"
    • List all the screen currently working : screen -ls
    • Reattach to a screen : screen  -r  <session number> or screen -r <screen name>
    • Kill specific screen: screen -X -S <screen name> quit
    • Kill all screens : pkill screen

Monitoring Usage

  • Docker stats

Remove Docker and Related folders

  • docker stop <Docker Name> # stop the docker but not remove anything. 
  • docker rm -f <Docker Name>  # remove speficic container, but will not delete mapped volumes
  • rm -rf /root/data/docker_data/<Docker Mapped Volumns>  # remove all mapped volumes

Restrick Journal Log File Size:

  • journalctl --vacuum-size=100M
  • Limit it to 25M:
nano /etc/systemd/journald.conf
SystemMaxUse=25M
systemctl restart systemd-journald.service


or 
sudo bash -c 'echo "SystemMaxUse=100M" >> /etc/systemd/journald.conf'
sudo systemctl restart systemd-journald



Enable IPv6 and Limit Log File Size (Ubuntu)

Special command to cleans all logs and you don't need to stop the containers.

  • sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log'
Add customized self defined IPv6 address segment to enable container's IPv6 fucntion. And limit log file's size and numbers in case log file to fill all hard drive's space. 

cat > /etc/docker/daemon.json << EOF
{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "20m",
        "max-file": "3"
    },
    "ipv6": true,
    "fixed-cidr-v6": "fd00:dead:beef:c0::/80",
    "eixperimental":true,
    "ip6tables":true
}
EOF

Restart Docker service:

systemctl restart docker

Limit number of log files:

cat /etc/logrotate.d/rsyslog

/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

You can change 4 to some other value, such as 1, so that only one file is stored.



Steps to Use Self Hosted Docker

For easy deployment, transfer.sh author has created an official Docker container. Here are steps we can follow to get docker running in 1 minute. 

1 Spin Up the docker

There are two variants, differing only by which user runs the process.

The default one will run as root:

docker run --publish 8080:8080 dutchcoders/transfer.sh:latest --provider local --basedir /tmp/

The one tagged with the suffix -noroot will use 5000 as both UID and GID:

docker run --publish 8080:8080 dutchcoders/transfer.sh:latest-noroot --provider local -

[node1] (local) [email protected] ~
$ pwd
/root
[node1] (local) [email protected] ~
$ docker run --publish 8080:8080 dutchcoders/transfer.sh:latest --provider local --basedir /tmp/
Unable to find image 'dutchcoders/transfer.sh:latest' locally
latest: Pulling from dutchcoders/transfer.sh
965ff63f17ca: Pull complete 
2003c3ea71dd: Pull complete 
3e870397f07a: Pull complete 
03e0e103d570: Pull complete 
83932710a96e: Pull complete 
Digest: sha256:912bae1a777722eaae79c305cabcc471b5bfce1f6cba323d086fb9b333170d99
Status: Downloaded newer image for dutchcoders/transfer.sh:latest
[transfer.sh]2023/10/08 15:19:45 Transfer.sh server started.
using temp folder: /tmp/
using storage provider: local
[transfer.sh]2023/10/08 15:19:45 starting to listen on: :8080
[transfer.sh]2023/10/08 15:19:45 ---------------------------
[transfer.sh]2023/10/08 15:19:48 172.18.0.1:61623 - - [2023-10-08T15:19:48Z] "GET / HTTP/1.1" 200  "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"



2 Transfer files


curl -v --upload-file ./hello.txt https://transfer.sh/hello.txt


[node1] (local) [email protected] ~
$ curl -v --upload-file /docker.log https://transfer.sh/hello.txt
*   Trying 144.76.136.153:443...
* Connected to transfer.sh (144.76.136.153) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=transfer.sh
*  start date: Aug 14 19:42:51 2023 GMT
*  expire date: Nov 12 19:42:50 2023 GMT
*  subjectAltName: host "transfer.sh" matched cert's "transfer.sh"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* h2 [:method: PUT]
* h2 [:scheme: https]
* h2 [:authority: transfer.sh]
* h2 [:path: /hello.txt]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* h2 [content-length: 34559]
* Using Stream ID: 1 (easy handle 0x7fd872c60080)
> PUT /hello.txt HTTP/2
> Host: transfer.sh
> User-Agent: curl/8.1.2
> Accept: */*
> Content-Length: 34559
> 
* We are completely uploaded and fine
< HTTP/2 200 
< content-type: text/plain
< server: Transfer.sh HTTP Server
< x-made-with: <3 by DutchCoders
< x-served-by: Proudly served by DutchCoders
< x-url-delete: https://transfer.sh/xVaLXBfwcD/hello.txt/wx9PGrancWLzyeSV55EU
< content-length: 40
< date: Sun, 08 Oct 2023 15:55:01 GMT
< 
* Connection #0 to host transfer.sh left intact
https://transfer.sh/xVaLXBfwcD/hello.txt[node1] (local) [email protected] ~
$ 




3 Fetch the file from any browser by access the url generated from step 2





Troublehooting


Error message: Wrong Version Number

It is caused by TLSv1.3 certificate . Transfer.sh service will require TLSv1.3, but the Play-with-docker provides only OpenSSL/3.1.1. To resolve this issue, we can remove https from the url.
[node2] (local) [email protected] /
$ curl -v --upload-file docker.log https://ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com/docker.log
*   Trying 40.76.55.146:443...
* Connected to ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com (40.76.55.146) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* OpenSSL/3.1.1: error:0A00010B:SSL routines::wrong version number
* Closing connection 0
curl: (35) OpenSSL/3.1.1: error:0A00010B:SSL routines::wrong version number
[node2] (local) [email protected] /

After removed https, we have http connection from command line to our Docker. All packets will be transfered in clear text format.


[node2] (local) [email protected] /
$ curl -v --upload-file docker.log ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com/docker.log
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 40.76.55.146:80...
* Connected to ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com (40.76.55.146) port 80 (#0)
> PUT /docker.log HTTP/1.1
> Host: ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com
> User-Agent: curl/8.1.2
> Accept: */*
> Content-Length: 34560
> 
} [34560 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Server: Transfer.sh HTTP Server
< X-Made-With: <3 by DutchCoders
< X-Served-By: Proudly served by DutchCoders
< X-Url-Delete: http://ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com/QwR2c1UDRs/docker.log/Q8DwSFS1yrr7GVoVQOOV
< Date: Sun, 08 Oct 2023 16:39:48 GMT
< Content-Length: 101
< 
{ [101 bytes data]
100 34661  100   101  100 34560   2543   849k --:--:-- --:--:-- --:--:--  890k
* Connection #0 to host ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com left intact
http://ip172-18-0-88-ckhdj44snmng00biqpe0-8080.direct.labs.play-with-docker.com/QwR2c1UDRs/docker.log[node2] (local) [email protected] /
$ 



Videos

 

Related video for Linx-server:




References








Read more

Subscribe via email

Author Image

About Netsec
Netsec - Learning, Sharing, Creating!

-
Tags:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)