DeepSeek ‘Fat Fish’ Transformed into Interactive Live2D AI Companion
A groundbreaking DIY project merges AI, Live2D animation, and real-time multimodal interaction — all in a desktop-ready virtual roommate.
🌟 Viral Sensation Meets Virtual Reality
DeepSeek’s beloved “Fat Fish” mascot has taken the internet by storm — sparking memes, fan art, and now, an innovative technical implementation: a fully interactive, voice-enabled, screen-aware Live2D AI companion.
This isn’t just another chatbot. It’s a responsive, animated persona that:
– ✅ Talks back using custom TTS with character-consistent voice cloning
– ✅ Reacts to mouse touches (ears, head, hands) with unique animations
– ✅ Watches your screen in real time and delivers witty, context-aware commentary
– ✅ Switches seamlessly between text and voice input/output
– ✅ Runs locally or via lightweight web deployment

🛠️ Technical Architecture Overview
The project builds upon the open-source AIRI framework — an extensible AI virtual partner platform — and enhances it with three core layers:
| Layer | Technology | Key Capability |
|---|---|---|
| Presentation | Live2D Cubism + Seethrough auto-decomposer | PSD-based rigging, physics-driven motion, real-time lip-sync |
| Cognition | DeepSeek API (via Qoder CN integration) | Role-aware LLM responses, dynamic personality tuning via system prompts |
| Perception & Interaction | Whisper ASR + GPT-SoVITS TTS + Screen capture + Vision model hooks | Full duplex voice chat, screen-aware commentary, gesture-triggered reactions |

🧩 Step-by-Step Implementation Highlights
1. Character Creation Pipeline
- Generated Q-style whale maid concept art using Kimi K3
- Automated layer decomposition with Seethrough, followed by AI-assisted repair of critical parts (eyes, mouth, ears)
- Mesh binding & motion physics configured for expressive idle behaviors

2. AI Integration Stack
- Integrated DeepSeek API via secure key injection in AIRI’s backend config
- Fine-tuned system prompt for “Fat Fish” persona: playful, observant, gently sarcastic, privacy-conscious
- Enabled automatic screen analysis — without reading sensitive content — to trigger contextual quips (e.g., “Back at DeepSeek’s house again?”)

3. Voice & Presence Enhancement
- Trained voice clone using GPT-SoVITS, sourced from Bilibili fan videos (5–10 sec clean samples)
- Synchronized mouth animation to audio amplitude in real time
- Configured Whisper ASR with auto-submit mode → enabling true “video-call-like” conversation flow

📦 Open Source & Accessibility
- ✅ Full source code published on GitHub (link in original article comments)
- ✅ Zero-code setup supported via Qoder CN — a Chinese-language AI coding assistant
- ✅ Lightweight local deployment: runs smoothly on mid-tier laptops (no GPU required)
- ✅ Modular design: easily adaptable to GPT, Claude, Qwen, or other LLM backends
💡 Pro Tip: The entire pipeline was built using ~1000 Qoder CN credits — optimized for cost and speed during their Double 11 promotion.

🚀 What’s Next?
This project exemplifies the democratization of embodied AI — where creativity, open tools, and accessible frameworks converge. Future extensions could include:
– Multi-character switching (e.g., “GPT Maid”, “Claude Scholar”)
– AR overlay via WebXR
– Real-time emotion inference from webcam feed
– Plugin ecosystem for third-party actions (calendar, music, notifications)
Whether you’re a developer, designer, or AI enthusiast — this is your invitation to build the next generation of living interfaces.

Project origin: JackCui | Source: AITNT AI News Platform