Twitter System Design Part-1

Twitter is an American microblogging and social networking service on which users post and interact with messages known as “tweets”. Registered users can post, like, and retweet tweets, but unregistered users can only read those that are publicly available.

Functional Requirements:

1. Tweets Obviously
2. User Timeline
3. Home Timeline
4. Trending Tweets (HashTags(#), Topics)
5. Search Tweet

Non Functional Requirements:

1. Highly Available
2. Eventual Consistency (It’s ok if a tweet is shown to followers after some time).
3. Low Latency (Real Time)

Estimations:

Characters Supported per tweet: 280 Characters

Write Requests: 1000 tweets/sec

Read Requests: 1000,000 tweets/sec

So It’s a read-heavy system.

Storage Estimations:

Tweets/Sec: 1000*280 = 280000 Bytes/sec = 280 KB/sec ~ 0.3 MB/sec

Tweets/Day: 0.3*3600*24 = 26 GB/Day

Jamboard Link:

Twitter System Design - Google Jamboard

Database:

Database Design

Let’s see how we use cache to store tweets in order to reduce latency.

For this example, I am taking Redis as my cache.

We can use a stored set to store tweets and followers.

Example:

1. <user_id1>_tweets: {1,2,3..}
2. <user_id2>_tweets: {4,5,6..}
3. <user_id1>_follows: {1,2,3…}
4. <user_id2>_follows: {4,5,6…}

For fast loading, we can store 5 years of data in Redis:

Total Space: (26 GB/Day) * (365*5 Day) = 65700 GB ~ 66 TB

User Timeline:

For the user timeline, we need to store all the tweets the user tweeted.

Let’s discuss how we store and read tweets from Redis/DB.

We can use a sorted set as discussed above to store tweets based on some scores say timestamp. Say we can fetch the top 20 tweets and once we read the top 10 tweets we can call from the backend to load 10 more we can use the timestamp of the last tweet (i.e the 20th tweet in a window) as the offset here.

Here’s the example:

Let’s see how we can store tweets in Redis. For this, we can use set commands.

Example:

tweet_id_1: “Hello There!”

tweet_id_2: “Check out my new article”

We can reduce key characters from tweet_id to T_ID to save some space.

HomeTime:

For the home timeline let’s discuss the steps:

1. Get followers
2. Get latest tweets
3. Merge and Display Tweets

For each user, we can maintain a home timeline basically key-value just like we use to store tweets in the user timeline.

Example:

H_U_ID1: {1,2,3…}

H_U_ID2: {3,4,5…}

So we can use a cache populator service to populate home timeline data in Redis. For online users, we can populate the tweet from a user to its followers directly. But we can’t do the same thing with celebrity tweets right as they have millions of followers.

Now for Celebrity tweets what we can do is wait and let the cache populator service(CRON) update tweets in individual user hometime caches.

So In short:

1. For Normal Users: Tweets push to followers' home timelines.
2. For Celebrity Users: Tweets pull to followers’ home timelines.

Example:

Normal Users Followers Home Timeline

Celebrity Users Followers Home Timeline

Services:

1. User App/ Web App: Used to write and read tweets.
2. Tweet Write Service: Handle all the write request
3. Tweet Write Consumer Service: To save data in the database and Redis cache based on the user and home timeline.
4. Tweet Read Service: Handle all the read requests.
5. API Gateway: That forwards traffic to different Load balancers (Path-Based API Gateway).
6. Cache Populator Service: Explained Above
7. Notification Service. (Real-Time)
8. Search Service: (Will be discussed In Part 2)
9. Trend Service: (Missing Here Will be discussed In Part 2)

High-Level Architecture:

HLD Twitter

ThankYou. Please do give suggestions.