How to Combine a Visual Novel with an LLM and a Desktop Pet in Rust

Most interfaces for working with neural networks look equally dull. They're either ChatGPT clones with a thread list on the left and an input field on the right, or command-line utilities for the terminal. But if you've ever played Japanese visual novels (Galgame), you know how important visual contact, character emotion changes, and the context of what's happening are.

Recently I stumbled upon an interesting project called LingChat, whose author tried to combine dialogue mechanics from visual novels, a virtual desktop pet system, and modern language models. The best part for us: under the hood it's honest Rust, not another resource-hungry Electron app.

What's Inside the Project

The LingChat repository offers a rethinking of the familiar format for communicating with AI. Instead of a dry chat window, we're presented with an animated character that reacts to dialogue, keeps track of your schedule, and can participate in interactive scenarios.

The project tackles several tasks at once:

• Emotional sprite responses. The model doesn't just generate text but also marks its own emotional state. Depending on the tone of the response, the character changes poses, facial expressions, and animations.
• Desktop Pet mode. The character lives right on top of your system's open windows, reacting to clicks and user actions in the background.
• Scheduler and task tracker. The dialogue engine has built-in calendar and reminder functionality, so the assistant will promptly remind you about a call or deadline.
• Branching story modules. The author added support for scenario branches, turning a regular chat into an interactive game with condition checks and choices.

Why the Choice of Rust Matters

Usually, such interactive applications are built with Electron + React or Unity. Both approaches have an unpleasant downside: resource hunger. The desktop pet needs to run in the background all workday without eating half your RAM and spinning up your laptop fans.

LingChat is written in Rust. This provides low resource consumption during constant sprite rendering and mouse event processing. The core application logic runs fast, leaving most system resources for local model inference or parallel work tasks.

The code implements a parsing system for processing emotion tags from LLM responses. When the model returns a response, the engine intercepts special mood markers before displaying text on screen and synchronously switches the character's animation frames.

Where This Can Be Applied

Besides using it directly for fun and nostalgia for old virtual pets like Shimeji, the project has interesting architectural ideas for developers.

First, it's a great reference for creating NPCs in indie games. If you want to add generative AI-based characters to your project, you can look here for inspiration on how to link a model's text output with animation graph state changes.

Second, the project nicely demonstrates the mechanics of creating personalized assistants with personality. Instead of a faceless bot, you get a companion whose behavior parameters and character are configured through system prompts and configuration files.

Where to Start

The source code is open under the AGPL-3.0 license. Documentation and setup guides are collected on the project's official website https://lingchat.wiki/.

To run it, you'll need a configured Rust toolchain and an API key for a compatible LLM provider (or a locally running server like Ollama / vLLM with OpenAI-compatible API support).

If you've been wanting to build your own custom desktop companion or experiment with dynamic sprite switching via LLM, digging into the SlimeBoyOwO/LingChat repository is definitely worth it.