Advanced Cluster Deployment: High Availability and Multi-Master Setup

## Architecture Overview

LarkXR adopts a distributed microservices architecture consisting of the following service layers:

- **Access Layer:** Nginx (reverse proxy and load balancing)
- **Management Node Cluster:** Multiple Admin instances sharing MySQL 8.0, Redis, and object storage (Local/OSS/OBS/S3/MinIO)
- **Rendering Node Cluster:** Multiple Agent+Render instances connected via OkHttp

This architecture supports both single-machine and cluster deployment. Services can be split and combined based on production requirements to optimize for concurrency and high availability.

**Single-machine deployment:** All services run on one GPU server.

**Simple cluster deployment:** Rendering and proxy services run on GPU servers; management and storage services run on non-GPU servers. Adding more GPU nodes increases concurrent processing capacity.

**Advanced cluster deployment:** Management node services are further split across independent servers with redundant deployment for high availability.

## Server Requirements

| Component | Version | Specs | OS | Count | Notes |
|---|---|---|---|---|---|
| Redis | 6.0.8 | 8-core 16GB | Ubuntu 20.04 | 1 | - |
| Nginx | 1.21.0 | 8-core 16GB | Ubuntu 20.04 | 1 | Download simple cluster package from official website |
| MySQL | 8.0 | 8-core 16GB | Ubuntu 20.04 | 1 | Manually create `larkxr_center` database with UTF-8 encoding |
| LarkXR Management Service | 3.2.8.2+ | 8-core 16GB | Ubuntu 20.04 or Windows | 1 | Requires JDK 17 |
| LarkXR Rendering Service | 3.2.8.2+ | GPU server | Windows Server 2019 | n | - |

## Installation Guide

Download the simple cluster version package from [https://paraverse.cc/larkxr-resource/](https://paraverse.cc/larkxr-resource/).

### Windows Management Node (`larkxr-center-windows`)

**Step 1: Set the Local IP Address**

In `application.properties`, set `server.node-ip` to the local IP address. Disable all service components except `admin`.

**Step 2: Configure Redis and Database**

Open `admin/application.yaml` and update the Redis and MySQL connection addresses.

**Step 3: Configure Nginx**

Copy the configuration files from the downloaded simple cluster Nginx folder (`conf/conf.d`, `admin-back`, `admin-front`) into the Nginx deployment directory.

Ensure `nginx.conf` includes:

```nginx
include conf.d/*.conf;
```

Set the reverse proxy address to the management node's IP address.

## Multi-Master Node Configuration

For high availability, deploy multiple admin service instances behind Nginx.

**Step 1: Configure an Additional Admin Service**

Repeat the single-node setup steps for each additional admin server and start the service.

**Step 2: Update Nginx Configuration**

Add an upstream server block and update the reverse proxy to load balance across all admin instances.

**Step 3: Switch Application Storage to Object Storage**

In the backend management portal under System Settings, change the storage method to object storage. See [How to Change to Object Storage?](https://docs.paraverse.cc/doc/315/) for details.

**Step 4: Configure Worker IDs to Prevent Key Conflicts**

In each admin node's `application.properties`, set unique worker IDs:

```properties
# Set to 0, 1, 2, ... for each respective admin node
pxy.id-generator.workerId=0
```

## Cloud Deployment: Batch Starting Rendering Services

For public cloud deployments using cloud images and batch-starting rendering programs, refer to the [batch startup guide](https://www.pingxingyun.com/contribute/detail?topicId=10103&categoryId=3).