The stage features a rectangular drape with rounded edges at the bottom, with projection mapping applied. For the visual design, we aimed to create dynamic, "living" paintings that resonate with the themes of the work and allow real-time control. Particles were selected for their unique aesthetic and the Zen energy integral to the project. To achieve this, we decided to use AI-generated Chinese Shanshui paintings, transform them into particle systems in Unreal Engine, and develop an interactive panel for live control.

​

My role involved researching and summarizing applications of Deep Neural Networks using PyTorch, training the AI model with StyleGAN3, and collaborating on the development of the interactive cue system in TouchDesigner and Unreal Engine.

​

Here is my workflow:

​

1) Training Chinese Shanshui Painting AI Model (StyleGAN3)

​

To train the AI model for generating Chinese Shanshui paintings, I started by exploring online resources, including online courses on Deep Neural Networks and GitHub repositories such as StyleGAN2, StyleGAN2-ADA, and StyleGAN3. StyleGAN3 was chosen for its smooth and continuous image generation.

 

I was provided with 550 Chinese Shanshui painting images, which were insufficient for a complete dataset. To address this, I used an existing pre-trained landscape model (source: Landscape 256) as the foundation for further fine-tuning.

​​

[1] Environment Setup

 

• Install necessary tools: 

Python, Anaconda, PyTorch, Visual Studio, and CUDA Toolkit.

​​​

• Set up the coding environment using Anaconda:​

 

conda create -n stylegan3 python=3.8
conda activate stylegan3
 

• Missing dependencies can be installed with:

 

pip install <package-name>​​​

​

[2] Dataset Preprocessing

​​​​​

The provided Shanshui images were preprocessed to meet training requirements:

​

• Code for Resizing and Cropping Images to Squares:

 

from PIL import Image
import os

input_folder = r'path_to_input_images'
output_folder = r'path_to_output_images'

os.makedirs(output_folder, exist_ok=True)

for filename in os.listdir(input_folder):
    if filename.endswith('.jpg') or filename.endswith('.png'):
        image_path = os.path.join(input_folder, filename)
        image = Image.open(image_path)

        size = min(image.width, image.height)
        left = (image.width - size) // 2
        top = (image.height - size) // 2
        right = left + size
        bottom = top + size
        square_image = image.crop((left, top, right, bottom))

        output_path = os.path.join(output_folder, filename)
        square_image.save(output_path)

 

• Run the Preprocessing Script:

 

python preprocess.py

​

• Convert to PyTorch-Compatible Format:

 

python dataset_tool.py --source="dataset path" --dest="destination path" --resolution=256x256

​

​​[3] Training the Model

​

Resuming the pre-trained landscape model with the processed dataset (trained for 4 days):

​

python train.py --outdir=~/training-runs --cfg=stylegan3-t --data=~/datasets/shanshui-256x256.zip --gpus=1 --batch=32 --gamma=32 --batch-gpu=4 --snap=5 --kimg=5000 --metrics=none --resume=pretrain model/lhg-256-stylegan3-t-25Mimg.pkl

​

[4] Generating Images and Videos

​

• Generate Images:

​​

python gen_images.py --outdir=out --trunc=1 --seeds=2 --network="pickle/model path"
 

• Generate Videos:

 

python gen_video.py --output=lerp.mp4 --trunc=1 --seeds=0-31 --grid=4x2 --network="pickle/model path"

​

• Using StylanGAN's interactive model visualization tool to generate animated videos:

 

python visualizer.py