X Platform Fully Opensources Its Recommendation Algorithm — Powered by Transformer
In a bold move toward transparency, 𝕏 (formerly Twitter) has officially open-sourced its core recommendation algorithm, revealing that it is driven by a Transformer architecture similar to the one used in xAI’s Grok model. The announcement, fulfilled by Elon Musk, marks a significant shift in social media platform openness.
A Transparent Approach to Content Ranking
The new system powers the “For You” feed and ranks posts based on predicted user interactions such as likes, replies, reposts, and more. Unlike traditional platforms, X emphasizes real-time learning and transparency:
“We know the algorithm is clunky and needs improvement, but at least you can now see us working to make it better — live and transparently. No other social media company does this.“
— Elon Musk
This release comes amid regulatory pressures, including a €120 million EU fine in December for violating Digital Services Act transparency rules, and an earlier investigation by Paris prosecutors into alleged algorithmic bias.
Full Code Release on GitHub
The complete codebase is now available on GitHub:
🔗 https://github.com/xai-org/x-algorithm
It includes the full pipeline behind the “For You” feed, combining both in-network (from followed accounts) and out-of-network (discovered content) posts using machine learning.
System Architecture Overview
Figure: X’s recommendation system architecture
The entire process is built around two key components: Thunder and Phoenix, eliminating most hand-written rules in favor of end-to-end neural ranking.
🔹 Thunder: Real-Time In-Network Content Engine
Thunder acts as a high-speed, memory-based storage system that tracks user-generated content in real time. Key features include:
- Consumes post creation/deletion events from Kafka
- Maintains separate storage for original posts, replies, reposts, and videos per user
- Delivers sub-millisecond query responses without external database access
- Automatically purges outdated content
Its primary role is to instantly retrieve what users you follow have recently posted.
🔹 Phoenix: Machine Learning Core with Transformer Model
Phoenix is the AI heart of the system, consisting of two main modules:
1. Recall (Two-Tower Retrieval Model)
- User Tower: Encodes user profile and historical interactions into embeddings
- Candidate Tower: Encodes all potential posts into embeddings
- Uses dot-product similarity to retrieve top-K relevant out-of-network content
2. Ranking (Transformer with Isolation Mechanism)
- Takes user context and candidate post as input
- Applies attention masking so candidates cannot “see” each other
- Predicts probabilities for multiple engagement types:
Predictions:
├── P(favorite)
├── P(reply)
├── P(repost)
├── P(quote)
├── P(click)
├── P(profile_click)
├── P(video_view)
├── P(photo_expand)
├── P(share)
├── P(dwell)
├── P(follow_author)
├── P(not_interested)
├── P(block_author)
├── P(mute_author)
└── P(report)
A final weighted score determines the ultimate ranking:
FinalScore = Σ (weight_i × P(action_i))
Negative signals like mute, block, or report are assigned negative weights, actively suppressing low-quality or unwanted content.
New Rules of Engagement: Quality Over Virality
With this update, X’s algorithm shifts focus from short-term virality to long-term relationship quality. Key implications:
- ✅ Deep interactions matter more than likes: shares, quote tweets, profile clicks, and dwell time are prioritized
- ❌ Negative feedback is penalized heavily: ‘Not interested’, mute, block, and report suppress visibility
- ⚠️ Clickbait & sensationalism lose value: Short-lived spikes are offset by later negative signals
- 📉 Frequent posting is devalued: Multiple posts from the same author are downweighted to prevent spam
- 🔗 Follow relationships gain strength: In-network content retains full weight; out-of-network is discounted
The Bigger Picture
By opening the black box, X invites researchers, developers, and users to understand, audit, and improve the system. While challenges remain, this unprecedented transparency could set a new standard for ethical AI in social media.
As the platform commits to updating the code every four weeks, the era of open algorithmic governance may finally be beginning.