Intro #

Performance is one of the hottest topics in the software development industry. In this post, I would like to share my experience when optimizing a CRM system which serves ~1k requests/second.

Problem #

Imagine, a CRM system has ~1000 user preferences (e.g. time zone, date format, currency,…), those user preferences are stored in a SQL database, and it will be loaded every single click to reflect the preference of the user. If your web app serves ~1K requests/second, it means there are ~1M requests to the DB per second (in the worst case). At this point, you can see we can improve the performance of the app by reducing the number of requests to the database and it’s time for the caching layer to shine!.

Breaking down the problem #

We break down into smaller problems:

1. P1: In the current system, we have a helper class, let’s say UserPreference. It exposed 2 methods: string Get(userId, key) and void Set(userId, key, value). These methods will go to the database to get or set data. How to reduce code rewriting to reduce risks? What is the backup plan if the changes cause problem in production?
2. P2: The UserPreference class was shared to many projects (Asp.net web form, Asp.net Web API, Window service, Load balancer nodes…). So, data consistency in distributed environment is a problem.
3. P3: In the future if we need to cache more things (Big query result, File content,…). The cache system should be easy to extend.

Solution #

To solve the problem, I create CacheBucket pattern. See the class diagram below:

Imagine, a bucket can contain values (water, sand, oil,…) and it can contain another Bucket. This is the core concept of the CacheBucket pattern.

The CacheBucket pattern includes:

• ICacheStorage: An interface which take responsibility to get or set cache data from a storage. (Memory, Redis, file…)
• Client$: A place which uses the pattern.
• CacheBucket: Manage inner buckets and cache values, it depends on ICacheStorage.
• UserPreferenceCacheBucket and BigQueryCacheBucket: Inherit the CacheBucket class for specific ICacheStorage. In detail, the UserPreferenceCacheBucket will use InMemoryCacheStorage. On another hand, the BigQueryCacheBucket will use the RedisCacheStorage.
• InMemoryCacheStorage and RedisCacheStorage: Implementations of the ICacheBucket

Ok! Now we will see how the CacheBucket pattern can solve the problem.

First, the CacheBucket class handled all logic about caching management so we just update small code change in the UserPreference class for adding caching into the system. For the backup solution we just add a flag (configurable value) which helps to detect if the caching system is enabled or not. In case there is a problem in production we can switch back to the old code by changing the value of the flag. So, P1 is solved.

Second, to solve the concurrency problem, we should use a centralized cache server such as Redis. Fortunately, with the CacheBucket pattern we can easily change the cache storage by implementing the ICacheStorage interface. So, P2 is solved.

Therefore, the CacheBucket pattern is very easy to extend. For example, if you want to cache the reporting result, all you need is to extend the CacheBucket class.

Benchmark #

To do the benchmark, I have created an example project. We can switch between enabling or disabling the cache. So, we can do benchmark the latency of the web application while enable/disabled the caching layer.

For the tool, I used the WRK (a HTTP Benchmarking tool) to simulate 200 connections (users) with this command:

wrk -t2 -c200 -d60s --timeout 3s http://cachebucket.com:5000/\?user_id\=1

The result #

When the caching layer is enabled, the latency is 465.46 ms and the web app can response 481.27 requests/sec. On another hand, the latency is 658.68 ms and 305.73 requests/sec when the caching layer is disabled. You can see more details below:

How to use CacheBucket pattern: #

Install package #

To use CacheBucket pattern you need to install packages from NuGet.

There are 3 packages:

1. CacheBucket.Core contains the core of cache bucket. In theory, you can use cache bucket pattern with this package only. However, for more convenience, you can consider other 2 packages below.
2. CacheBucket.InMemory contains an InMemoryCacheStorage which is an implementation of ICacheStorage to store cache data in memory.
3. CacheBucket.Factory contains some helper classes and extension methods that help to create a CacheBucket inline code. e.g: CacheBucketFactory.Create("UserPreference:1")

This is an example command to install CacheBucket.Core in “Package Management Tools”

Install-Package CacheBucket.Core #replace CacheBucket.Core by another package name to install another package.

or if you prefer the dotnet command:

dotnet add package CacheBucket.Core

Sample code #

Because of the CacheBucket class is open for extending you can create a derived class as the sample code below:

using CB.Core;
using CB.InMemory;

namespace WebApplication.Helpers {
    public class UserPreferenceCacheBucket : CacheBucket {
        public const string NAME = "UserPreference";

        public UserPreferenceCacheBucket (InMemoryCacheStorage cacheStorage) : base (NAME, cacheStorage) { }
    }
}

And then you can inject the UserPreferenceCacheBucket class into a client:

using CB.Core;
using WebApplication.Data;
using WebApplication.Data.Models;

namespace WebApplication.Helpers {
    public class UserPreferenceHelper {
        private readonly UserPreferenceCacheBucket _cacheBucket;
        private readonly ApplicationDbContext _dbContext;

        // The UserPrefrenceCacheBucket can be inject by IoC container.
        public UserPreferenceHelper (UserPreferenceCacheBucket cacheBucket, ApplicationDbContext dbContext) {
            _cacheBucket = cacheBucket;
            _dbContext = dbContext;
        }

        public string Get (int userId, string key) {
            CacheBucket userBucket = null;
            if (MvcApplication.EnableCacheBucket) {
                userBucket = _cacheBucket.In (userId.ToString ());

                // get cache value.
                var cacheValue = userBucket.GetValue (key);

                if (string.IsNullOrEmpty (cacheValue) == false) {
                    return cacheValue;
                }
            }

            // get db value.
            UserPreference userPreference = GetUserPreference (userId, key);

            if (userPreference == null) {
                return null;
            }

            // set into cache.
            userBucket?.SetValue (key, userPreference.Value);

            return userPreference.Value;
        }

        /// ...
    }
}

Conclusion #

Overall, the caching layer is simple and easy to apply but effective. You don’t need to use CacheBucket to apply caching layer, you can apply the caching pattern in your own way as long as it is the most effective way for your situation.

This is my personal experience. If you have any better way, please share it on the comments, and we can discuss later😜