How to Use Google Colab to Run Stable Diffusion Web GUI to Draw AI Images - NETSEC

Latest

Learning, Sharing, Creating

Cybersecurity Memo

Sunday, April 9, 2023

How to Use Google Colab to Run Stable Diffusion Web GUI to Draw AI Images

If you have a browser, you will be able to use Google Colab to run Stable Diffusion Web UI created by ATOMATIC1111, which has github project: https://github.com/AUTOMATIC1111/stable-diffusion-webui


In this post, I am going to show how to write your own notebook and customize the code to have more functions to meet your presonal customization requirements. 



Stable Diffusion Web UI Introduction

A browser interface based on Gradio library for Stable Diffusion.

Github: https://github.com/AUTOMATIC1111/stable-diffusion-webui


Steps to Run Stable Diffusion Web UI at Colab

The steps in this section is only for the first time. After you have completed the first time running, you wont need some steps since we have downloaded required models. 

Note: Following steps are mostly coming from this blog post: https://ivonblog.com/en-us/posts/google-colab-stable-diffusion-webui/
  1. Go to Google Colab, click "New notebook". Or from File menu's drop down list, select New notebook.

  2. In the new notebook, copy & paste following example code. 

  3. You will need to log into your Google Account. Your Google Drive will have at least 10GB free space for downloading reqired components. During running, it will take about 10GB space from your Google Drive.

  4. First we mount the Drive to /content/drive. And create a new folder (sd-webui-files) for storing files on your google drive's root folder.

1
2
3
from google.colab import drive
drive.mount('/content/drive')
!mkdir /content/drive/MyDrive/sd-webui-files
  1. Install dependencies
1
2
3
!pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 -U
!pip install -q xformers==0.0.16
!pip install -q triton==2.0.0
  1. Clone SD WebUI related files directly from ATOMATIC1111’s repo
1
!git clone --depth=1 https://github.com/AUTOMATIC1111/stable-diffusion-webui.git /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui
  1. Download checkpoint models from Hugging Face. I prefer to use Anything model for generating anime art. Or you can use original Stable Diffusion model for generating realistic arts.
1
!wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Stable-diffusion https://huggingface.co/andite/anything-v4.0/resolve/main/anything-v4.5-pruned.safetensors
  1. There are some issues of switching branches after launching SD WebUI. Thus, add these lines to fix them.
1
2
3
4
%cd /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/
!git reset --hard
!git pull
!sed -i -e 's/checkout {commithash}/checkout --force {commithash}/g' launch.py
  1. Finally, we will have python to launch the WebUI. We use --xformers to decrease the consumption of VRAM. By adding --enable-insecure-extension-access we can install extensions from URL in WebUI without getting AssertionError: extension access disabed.
1
!python launch.py --share --xformers --enable-insecure-extension-access --theme light
  1. Here is the full code
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# 掛載雲端硬碟
from google.colab import drive
drive.mount('/content/drive')

!mkdir /content/drive/MyDrive/sd-webui-files

# 安裝CUDA、xformers、Triton依賴
!pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 -U
!pip install -q xformers==0.0.16
!pip install -q triton==2.0.0

# 複製SD WebUI上游的儲存庫
!git clone --depth=1 https://github.com/AUTOMATIC1111/stable-diffusion-webui.git /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui

# 下載存檔點模型至models資料夾,網址為在模型網站按右鍵取得
!wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Stable-diffusion https://huggingface.co/andite/anything-v4.0/resolve/main/anything-v4.5-pruned.safetensors

# 下載LoRA模型
!wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Lora https://civitai.com/api/download/models/13739 -O Korean_Doll_Likenesss.safetensors

# 以git clone安裝擴充功能:中文化
!git clone --depth=1 https://github.com/benlisquare/stable-diffusion-webui-localization-zh_TW.git /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/extensions/stable-diffusion-webui-localization-zh_TW

# 以git clone安裝擴充功能:ControlNet,以及Scribbles模型
!git clone --depth=1 https://github.com/Mikubill/sd-webui-controlnet.git /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/extensions/sd-webui-controlnet
!wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/extensions/sd-webui-controlnet/models https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_scribble.pth

# 防止其他儲存庫造成錯誤
%cd /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/
!git reset --hard
!git pull
!sed -i -e 's/checkout {commithash}/checkout --force {commithash}/g' launch.py

# 啟動WebUI。直接從launch.py傳遞引數
!python launch.py --share --xformers --enable-insecure-extension-access --theme light
  1. Click Edit → Notebook Settings →Change Hardware accelerator type to use GPU as Hardware accelerator. Click Save.


  2. If you have not enabled GPU, the running result will show you an error message regardng torch can not use GPU. The whole process will be stopped from there. You will have to enable GPU and re-run the notebook codes again. Error message: "AssertionError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check"

  3. Click Run button beside the code to run the cell, wait for around 11-12 minutes (first time running usually take longer. Second time using shorter code will only take 5 minutes to complete). 

  4.  Click generated Gradio links (expired after 72 hours).

  5. Now we are ready to use Stable Diffusion WebUI.



The total process for first time running will take about 11-12 minutes. 

Here is the result using anything checkpoint to generate a nice girl image.




Run Simplified Code After First Time Run

Now we have installed SD WebUI on Google drive. Even we closed our instance. Don’t worry, your data and progress has been stored on your Google Drive. It has about 10.35GB data downloaded and stored on your Google Drive folders under folder "stable-diffusion-webui". 


Because we have downloaded all required files, no need to run downloading code again. So you can replace the code in the cell with only following seven lines:
1
2
3
4
5
6
7
from google.colab import drive
drive.mount('/content/drive')
!pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 -U
!pip install -q xformers==0.0.16
!pip install -q triton==2.0.0
%cd /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/
!python launch.py --share --xformers --enable-insecure-extension-access --theme light

This will install requirements while running SD WebUI and start SD WebUI from our Google Drive. The Gradio link should be ready in 7 minutes. You might get a pop up window to connect to your google drive and the following warning message about permitting notebook to access Google Drive Files. 




LoRA models

There are different model type for SD, such as Checkpoint, Textual Inversion, Hypernetwork, Aesthetic Gradient, LoRA, and LyCORIS. 



LoRA (Low-Rank Adaptation) models are small Stable Diffusion models that apply tiny changes to standard checkpoint models. They are usually 10 to 100 times smaller than checkpoint models. That makes them very attractive to people having an extensive collection of models. 

LoRA offers a good trade-off between file size and training power. Dreambooth is powerful but results in large model files (2-7 GBs). Textual inversions are tiny (about 100 KBs), but you can’t do as much.

LoRA sits in between: Their file sizes are manageable (2 – 200 MBs), and the training power is decent.

Like textual inversion, you cannot use a LoRA model alone. It must be used with a model checkpoint file. LoRA modifies styles by applying small changes to the accompanying model file.

In our this lab, we are going to use chilloutmix as our standard checkpoint model. We can find those files from https://civitai.com/.

Civitai hosts a large collection of LoRA models. Apply the LORA filter to see only LoRA models. You may find that they all tend to be similar: female portraits, anime, realistic illustration styles, etc.
Hugging Face is another source of LoRA libraries. You will find more varieties of LoRA models. But there are not as many LoRA models there. Their collection is a lot smaller.

Adding more checkpoints and models

Important folders:

There are some important folders under your Google Drive's stable-diffusion-webui folder:

  • extensions: As the name suggested, upload any extensions folder to this folder (Click Code button on Github repository of extesnions and download zip). Or you can install extensions by clicking Extensions → Install from URL in WebUI.
  • models/Stable-duffusion/: Upload new checkpoint models to this folder
  • models/Lora: Upload LoRa models to this folder

All changes would be applied after restarting SD WebUI.

For example, for this Lora model: https://civitai.com/models/11722/iu, we can find out the download link is https://civitai.com/api/download/models/18576. The download package is about 288.11MB for IU v3.5. It will be download to Lora folder under models.

Lora models:

  • https://civitai.com/models/17998/hongkongdolllikeness  - hongkongdoll - https://civitai.com/api/download/models/17998
  • https://civitai.com/models/11722/iu -iu v3.5 - https://civitai.com/api/download/models/18576
  • https://civitai.com/models/11619/korean-doll-likenesss - korean doll likenesss - https://civitai.com/api/download/models/13739
Command:
  • !wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Lora -O hongkongdolllikeness.safetensors https://civitai.com/api/download/models/18576

Chilloutmix Checkpoint file:

  • https://civitai.com/models/23302/graffiti-art-chilloutmix - 
Command:
  • !wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Stable-diffusion -O chilloutmix.safetensors https://civitai.com/api/download/models/27828


Note: After download, if there is missing extention for files, you will need to add the extension .safetensors for each file. 


You also can use aria2c command line download tool to do downloading. Here is an example:
  • !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/sd14/resolve/main/sd-v1-4.ckpt -d /content/stable-diffusion-webui/models/Stable-diffusion -o sd-v1-4.ckpt

Final codes

To sum up, the final full codes for first time run:

# 掛載雲端硬碟
from google.colab import drive
drive.mount('/content/drive')

!mkdir /content/drive/MyDrive/sd-webui-files

# 安裝CUDA、xformers、Triton依賴
!pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 -U
!pip install -q xformers==0.0.16
!pip install -q triton==2.0.0

# 複製SD WebUI上游的儲存庫
!git clone --depth=1 https://github.com/AUTOMATIC1111/stable-diffusion-webui.git /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui

# 下載存檔點模型至models資料夾,網址為在模型網站按右鍵取得
!wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Stable-diffusion https://huggingface.co/andite/anything-v4.0/resolve/main/anything-v4.5-pruned.safetensors

# 下載LoRA模型 - Korean Doll
!mkdir /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Lora
!wget -nc https://civitai.com/api/download/models/13739 -O /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Lora/Korean_Doll_Likenesss.safetensors

# 以git clone安裝擴充功能:中文化 - Optional
# !git clone --depth=1 https://github.com/benlisquare/stable-diffusion-webui-localization-zh_TW.git /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/extensions/stable-diffusion-webui-localization-zh_TW

# 以git clone安裝擴充功能:ControlNet,以及Scribbles模型
!git clone --depth=1 https://github.com/Mikubill/sd-webui-controlnet.git /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/extensions/sd-webui-controlnet
!wget -nc -P /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/extensions/sd-webui-controlnet/models https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_scribble.pth

# Add new checkpoint and LoRA model (Hongkong Doll)
!wget -nc -O /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Lora/hongkongdolllikeness.safetensors https://civitai.com/api/download/models/17998
!wget -nc -O /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/models/Stable-diffusion/chilloutmix.safetensors https://civitai.com/api/download/models/27828

# Fix change branch issues
%cd /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/
!git reset --hard
!git pull
!sed -i -e 's/checkout {commithash}/checkout --force {commithash}/g' launch.py

# Launch WebUI
!python launch.py --share --xformers --enable-insecure-extension-access --theme light

After first time run, you can run following codes instead:

from google.colab import drive
drive.mount('/content/drive')
!pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 -U
!pip install -q xformers==0.0.16
!pip install -q triton==2.0.0
%cd /content/drive/MyDrive/sd-webui-files/stable-diffusion-webui/
!python launch.py --share --xformers --enable-insecure-extension-access --theme light

Note about Google Colab:

There are no official references for 'Idle' and 'Maximum Lifetime' durations, but testing done by Korakot Chaovavanich shows that:

  • The 'maximum lifetime' of a running notebook is 12 hours (browser open)
  • An 'Idle' notebook instance cuts-off after 90 minutes
  • You can have a maximum of 2 notebooks running concurrently
  • If you close the notebook window and open it while the instance is still running, the cell outputs and variables will still persist. However if the notebook instance has been recycled, your cell outputs and variables will no longer be available.

A sneaky workaround you can try is to have the Colabs instance open in your mobile browser in order to prevent the instance from being considered "Idle".



No comments:

Post a Comment