Live Commenting system of Facebook, rolled out a couple of weeks ago. Live Comments – this new Facebook engineering concept does more than just bring an IM-like conversation to the comments that appear on the wall under the posts on Facebook pages. A “write locally, read globally” approach is utilized to manage the traffic of people who are just reading content vs. traffic from those who are posting comments to that content, that eventually lightens the loads on servers of Facebook.
The company’s Ken Deeter posted a post on Facebook’s Engineering blog, which explains how Facebook is saving server space and decreasing latency by directing traffic. FB engineers were facing challenges while deploying this system, so to understand them Deeter actively noted some details on why it made the change.Every minute, Facebook users post 100 million pieces of content to the site that could possibly receive comments, and in that same minute, users submit 650,000 comments that need to be routed to the right Facebook user.
“To make this feature work, we needed to invent new systems to handle load patterns that we had never dealt with before,” Deeter wrote.
Facebook initially considered a poll-based system that would periodically check back and see if there had been comments. Even though this occured in less than five seconds, Facebook found that that wasn’t fast enough.
“Because humans are so sensitive to latency in real-time communications, creating a truly serendipitous commenting experience requires comments to arrive as quickly as humanly and electronically possible,” Deeter wrote.
As a result, Facebook switched to a push-based approach, but while “storing these one-to-one, viewer-to-content associations in a database is relatively easy, keeping up with 16 million new associations per second is not.”
“Up until this point, Facebook engineering had built up infrastructure optimized for many more reads than writes. But now we had flipped the game,” he continued. “Every page load now requires multiple writes (one for each piece of content being displayed). Each write of a comment requires a read (to figure out the recipients of an update). We realized that we were building something that was fundamentally backwards from most of our other systems.”
As a result, Facebook deployed distributed storage tiers that handles writes locally. “For example, when a user loads his News Feed through a request to our data center in Virginia, the system writes to a storage tier in the same data center, recording the fact that the user is now viewing certain pieces of content so that we can push them new comments,” Deeter wrote. “When someone enters a comment, we fetch the viewership information from all of our data centers across the country, combine the information, then push the updates out.”
While this means that Facebook has to do multiple cross-country reads for every comment on the site, Deeter said it works because there are more comments than views. “Reading globally saves us from having to replicate a high volume of writes across data centers, saving expensive, long-distance bandwidth,” he concluded.
Now we know that the Facebook engineers had worked hard & intelligently to make this LIVE COMMENTING system work. They have literally flipped the entire process on its head to make commenting better. So the next time you are on facebook & suddenly a new comment magically appears on a post while you’re either commenting or scrolling along just catching up on your friends’ latest adventures, you know the new engineering done on it.
