How a Private ‘Anime’ Torrent Tracker Became an Essential Tool For Facebook

[Ernesto]

Large scale web-services need tens of thousands of servers to keep things running smoothly for their millions of users.

Keeping all of these servers updated with the latest code can be time and resource intensive and it was no different at Facebook during its early years.

However, most problems disappeared when the social networking company discovered BitTorrent. With BitTorrent all servers in the network could help to distribute code updates and as a result deployment took minutes rather than hours or days.

After discovering these benefits Facebook changed its BitTorrent implementation quite a bit. Among other tools, the company is now using the open source tracker software Chihaya, named after a school girl starring in the manga series Chihayafuru.

While this might seem like a peculiar name for a piece of BitTorrent tracking code, all becomes clear when you look at the history of the software and its links to a private anime torrent tracker.

In 2012 a developer named Kotoko started working on a new tracker backend written in the then-new programming language Go. Named Chihaya, the project (originally developed for a private anime community) aimed to become a replacement for the Ocelot tracker used by many Gazelle-based torrent sites.

Around the same time the people behind the Waffles community were working on a full replacement for Gazelle named Batter, and the Chihaya developer eventually jumped on this bandwagon. The project also drew the attention of other programmers, including Jimmy Zelinskie and Justin Li, both college students at the time.

“I was interested in helping out with Chihaya back then because I wanted to work on a project to cement my skills in the Go programming language,” Zelinskie tells TF.

After a while priorities changed. Chihaya was never connected to a tracker frontend, but Zelinskie and Li kept improving it bit by bit.

“The Batter project fizzled out, but Chihaya development continued,” Zelinskie says. “We restructured Chihaya a few times, trying to decide how to make it scalable and ultimately landed on what we have today.”

Chihaya

Over the past several years Chihaya has evolved into one of the most advanced pieces of tracker software around, with support for multi-cored processors and peers announcing on IPv4, IPv6 or both.

“The architecture of the project is entirely modular and in doing so, we’ve made the tracker so it could potentially support any transport protocol like HTTP or UDP and any backend BitTorrent indexing software like Gazelle,” Zelinskie tells TF.

This didn’t go unnoticed by others, including the engineering team at Facebook who also started to use the code for their server deployment.

“Facebook started using the project because of our proper IPv6 support,” Zelinskie says, adding that they optimized the tracker even more for a local setup.

“We soon after added the ability to prefer peers based on a subnet of their IP address; for example, if your IP address is 192.168.1.1, you can configure the tracker to deliver you all the ‘closest’ peers in the 192.168.1.X range before any others,” he notes.

Zelinskie currently works at CoreOS, a company that specializes in the deployment of software. He believes that BitTorrent-supported distribution is the future for companies, large and small. Chihaya certainly fits into this picture.

This leads to the remarkable conclusion that an open source private tracker, originally programmed to serve anime torrents, is now powering one of the largest technology companies in the world.

For Zelinskie, this transition not only shows the true power of open source, but also of BitTorrent.

“This is the reason why I write open source software and my company releases so much of what we do open source. Having as many people as possible working towards a common goal, in this case a solid BitTorrent tracker implementation, is beneficial to all of society, not just one set of individuals,” he says.

“BitTorrent is far too often associated with copyright infringement. When in reality, BitTorrent is simply the best file transfer protocol. Whether it’s being used by you, me, or even Facebook.”

—

Facebook didn’t respond to our request for comment but their use of Chihaya was confirmed by an insider.

source