Lila Scavenger Hunt
Let's study software architecture by learning about the how the open-source Lichess website is made, which hosts over 100,000,000 rated chess games every month.
To win the scavenger hunt, complete the missing items. Please use any code
search tools, language servers, or documentation you'd like for this
scavenger hunt. Though I originally answered all of these questions just
using GitHub search (the
keys) and online editor (the
Venture out and explore! We do scavenger hunts not intrinsically to find the answers, but because we're curious to see, experience, and learn about the scenery along the way.
Begin at the “Starting from scratch” blog post and source code. The biggest challenge in Lichess's backend is that it requires serving a lot of real-time features that scale to millions of users. It's primarily written in the programming language and uses , a library for asynchronous streams and actors.
Notice that the entire core Lila application, which handles all chess games from every single user on lichess.org, is hosted on just a single machine, named . This machine has 96 logical cores and 192 GiB of RAM. By remaining a monolith and only using a single big machine, Lichess is able to reduce hosting and infrastructure management costs greatly, while simplifying development. The downside is that it's limited to vertical scaling, but computers are fast! Lila is written in a very performant way, to handle hundreds of thousands of concurrent active users just fine.
UI rendering in Lila is done in a hybrid manner, some on the server and some
in the browser. On the server side, you can check the
folder to see that it uses the library for static templating
of HTML fragments. On the frontend side, helps with dynamic
rendering from TypeScript. The site is styled using , which
compiles to CSS.
Lichess has a library called that renders all chessboards
on its website, in 2D and 3D styles, on both web and mobile. If you inspect one
of the boards on the website, you'll see that it's wrapped in a custom
<piece> elements inside it.
Internationalization (i18n) is the process of offering software in multiple languages. Within the lila repository, the folder has variants in every language for all written text on the website.
Now let's get curious and examine a feature. On the lobby page of
lichess.org, you notice that the bottom-right has a couple pieces of text
that say “X players / Y games in play,” and you're curious what file this
belongs to and how it works. After some searching and inspection, you find
that this UI element, as well as the entire right column with dark mode
toggle and three “Create a game” buttons is defined in the
file (full path).
Real-time web communication
Let's dive into the "X players" indicator. It's updated in real time on the
frontend, meaning that there should be a WebSocket connection sending
messages. Use your Chrome DevTools Network tab to look for open WebSockets
on lichess.org. There should be a bunch of real-time updates streaming in,
"t" field. The messages with the numbers of
players and games in play have this field set to
These messages are handled on the client side by several TypeScript files,
but the one relevant for this particular statistics message is
ui/lobby/src/boot.ts. This file contains the class, which Lichess uses to manage a socket connection and register event
On the server-side, WebSocket connections are handled by a separate Scala service in a different repository called .
Let's move to the WebSocket service repository for a while. The
folder contains shared logic for sending and receiving messages. It connects
inputs and outputs from the browser WebSocket, ClientIn/ClientOut, with data
updates from the server, LilaIn/LilaOut. The class is
responsible for the "number of users and games" message we saw earlier. This
message is sent from the singleton, which responds
to client pings.
We already established that ClientIn/ClientOut messages are sent over WebSocket. LilaIn/LilaOut messages are actually transported over an internal database. They get sent over streams in real time.
We can examine some examples of how this communication works. Look at the
src/main/scala/Tv.scala. This defines handlers for Lichess
TV. A connection to Lichess TV returns all recent “bullet”-speed games with
activity in the past , while other games are stored
Now let's try to look at how in-game chat is implemented. In ClientOut, there is a type of message called “talk” that is used by the browser to send a chat. This maps to the case class of ClientOut, as an object in Scala. When it is forward to the Lila side, this case class maps to the command.
Go back to the main Lila repository. Lila uses a specific MongoDB driver called that intends to avoid blocking operations, running fully asynchronously to handle large numbers of clients.
On top of this driver, Lila defines its own domain-specific language (DSL)
for making queries to its database from Scala. This is defined in the
package. Search the codebase for imports of this package; notice that it's used
in many files to make complex queries to the database. When stored in MongoDB,
the field in the Game collection contains all user IDs that are
playing in a given game.
Another place the database is used is in choosing the daily chess puzzle. The query for this automatically filters for puzzles that have been played at least some number of times. A puzzle must have at least plays to be selected as the daily puzzle.
Let's move to the scalachess repository now, which implements the rules of
chess. Looking around, there is a chess opening database hardcoded into this
package, in parts. We can also see in the
file that there are 10 chess variants
supported by Lichess, each with their own custom rule definitions.
Now let's look at the compression repository. It uses a coding algorithm to efficiently encode indices in the legal move list. This reduces the amount of space required to store and re-simulate chess games, as they can be represented very compactly.
For many chess algorithms it's useful to get an even more compact representation of the board state and history, which can then be used as the key to a hash table or otherwise. For example, this is common in other games like Go as well. Lichess uses the hash algorithm.
Curious about how to get more Internet points, you snoop around and find out that Lichess uses the rating system. The default deviation parameter is set to , and games are treated as having 1 points for wins, 0.5 points for draws, and 0 points for losses.
That module defined the mathematical formula for rating calculations, but you want to understand where this fits into the system as a whole. You find the class in the same rating module, which specifies how ratings are read and stored in the MongoDB database. It also has support for rating refunds!
How are players matched with each other using this rating field? It turns out that the matchmaking function is actually another simple mathematical formula. It's specified in the function, which takes into account rating ranges, how close their ratings are, and how many “waves” of matchmaking a player has missed. The maximum score difference to pair two users is depends on rating, but for ratings above , it's set to the numerical rating value divided by
Waves of matchmaking are specified for every pool of games, which are split by clock time control. A wave happens either after some number of sections or a threshold number of players is waiting for a game, whichever comes first. The pools are defined in the object, while the class is constructed once for each pool and is responsible for actually scheduling waves.
Lichess offers free access to the Stockfish chess engine to analyze games. This is deployed on their distributed “AI cluster” on donated hardware called since Stockfish is compute-intensive. This cluster is separated from Lila. Actually, anyone can run their own client and help analyze Lichess games if they have spare CPU cycles. The client is written in Rust.
Lichess also offers a free API for developers to access game data. The API is implemented in the module. It's a REST API that uses JSON for data serialization. The API is documented on the Lichess website.
Other services, monitoring, and metrics
Although the core application logic is in Lila, which acts as a monolith, there are many other services that handle smaller tasks, mainly for scalability or isolation reasons. For example, is written in Rust and handles trillions of unique positions for all variants of chess. It compacts opening data in , an embedded database tuned for performance.
Another side service is , which grew out of need
because some Lichess tournaments have tens of thousands of connected clients
at once, and Lila was overloaded with HTTP traffic. It's also written in
Rust, using the
axum library, and the only route it handles is
Search happens in the package, which uses the open-source search engine to index games, forums, teams, and studies.
All cloud monitoring happens on the machine, which runs Prometheus, InfluxDB, and Grafana.
Backpressure and fault tolerance
We saw a lot of infrastructure here in this scavenger hunt, but that's just a very high-level picture of how the application works. There's a lot more to dig into. Here's some food for thought.
Based on what we've seen so far, imagine a scenario where Lichess's load quickly increases to an unusually high level. What system would break first? Would it crash, or would it just temporarily stop responding to requests? How long would it take to restart if it did crash? Would your answer change if one of the machines lost power suddenly, or there was somehow a segmentation fault in Lila?
If one of the services is overloaded, would it return errors to requests, or would it just let them time out? Also, would this cause a chain reaction of upstream services? Pick a service, where in the codebase would you look to predict the effects of that going down?
If you were running Lichess and the website stopped working, where would you look first to diagnose the problem, and how could you make it easier to resolve the problem quickly?
to reset the scavenger hunt.