What is Req Shield

What is Req-Shield

The Req-Shield is a lib that regulates the cache-based requests an application receives in terms of request-collapsing.
(Notes : LY Corporation Tech Blog)

How does it work?

Req-Shield works on two main principles.

Using 'lock' when creating a cache

A typical application will utilize the cache in the following way.

In the image above, N requests are sent to a specific infra, including the DB, when there is no cache.
The problem here is the Thundering Herd Problem, which occurs when many processes or threads try to access the same resource at the same time.
There will be N requests to create cache as well as DB.

What if the same request happens multiple times and we combine them into a single request (using a lock)?

We can reduce the number of requests to the DB to one and the number of requests to create cache to one.

Asynchronous update based on cache time

If a cache exists but is about to expire, it would be beneficial to have a single request asynchronously update the cache at that time, which would eliminate the Thundering Herd problem as well as the dead time in the cache.

AS-IS
TO-BE

Feature Definitions

The features are summarized below.

Modules

core

The module that contains the actual core unit logic
Default implementation with blocking method
core-reactor

Modules that support reactor based on actual core unit logic
core-kotlin-coroutine

Modules that support kotlin-coroutine based on actual core unit logic
core-spring

A module with added logic based on the actual core logic and easy to use in Spring MVC environment (Support for annotations)
core-spring-webflux

A module with added logic based on the actual core logic and easy to use in Spring WebFlux environment (Support for annotations)
core-spring-webflux-kotlin-coroutine

A module with added logic based on the actual core logic and easy to use in Spring WebFlux with Kotlin Coroutine environment (Support for annotations)
support

Provide common code and utils used by each module

How to use

Common considerations

ReqShieldData

The cache data stored by Req-Shield is wrapped in a class called ReqShieldData. ReqShieldData is organized as follows. The reason for using a wrapping class called ReqShieldData is for cache updates, because we need the creation time and lifetime to calculate how much time is left based on the lifetime of the currently retrieved cache. You should always keep in mind that ReqShieldData is used as the unit of data handled by Req-Shield and stored in cache platforms like Redis.

data class ReqShieldData<T>(
    var value: T?,
    var status: Status,
    val createdAt: Long,
    val timeToLiveMillis: Long,
)

value : The actual data that the application needs is entered.
status : Represents the state of the ReqShieldData. As of the current version (v1.2.0), this is a fixed value.
createdAt : The time at which this cache was created is entered as Long(epoch time).
timeToLiveMillis : Displays the lifetime of this cache as Long (ms).

ReqShieldConfiguration

The ReqShieldConfiguration, which is a required element when creating an instance of the ReqShield class, is shown below.

data class ReqShieldConfiguration<T>(
    val setCacheFunction: (String, ReqShieldData<T>, Long) -> Boolean,
    val getCacheFunction: (String) -> ReqShieldData<T>?,
    val globalLockFunction: ((String, Long) -> Boolean)? = null,
    val globalUnLockFunction: ((String) -> Boolean)? = null,
    val isLocalLock: Boolean = true,
    val lockTimeoutMillis: Long = DEFAULT_LOCK_TIMEOUT_MILLIS, // for local and global lock
    val executor: ScheduledExecutorService =
        Executors.newScheduledThreadPool(
            Runtime.getRuntime().availableProcessors() * 10,
        ),
    val decisionForUpdate: Int = DEFAULT_DECISION_FOR_UPDATE, // %
    val keyLock: KeyLock =
        if (isLocalLock) {
            KeyLocalLock(lockTimeoutMillis)
        } else {
            KeyGlobalLock(globalLockFunction!!, globalUnLockFunction!!, lockTimeoutMillis)
        },
    val maxAttemptGetCache: Int = MAX_ATTEMPT_GET_CACHE,
)

setCacheFunction : Put a function(Callable) that sets to the cache platform you use. As mentioned above, ReqShieldData goes in as the value
getCacheFunction : Put a function(Callable) to get the ReqShieldData from the cache platform you use.
globalLockFunction : Include a function (Callable) that acquires the global lock. If isLocalLock is true, you don't need to set it.
globalUnLockFunction : Include a function (Callable) that releases the global lock. If isLocalLock is true, you don't need to set it.
isLocalLock : Whether the lock to use when creating and modifying the cache should be a local lock. If false, it will use a global lock.
lockTimeoutMillis : timeout when using local lock and global lock
executor : The executor used for asynchronous operations within Req-Shield (depending on the reactor, kotlin-coroutine, this may be different or non-existent)
decisionForUpdate : A number that determines what percentage of the cache lifetime to start updating from when updating the cache.
keyLock : This is the part that is automatically determined by isLocalLock.
maxAttemptGetCache : This is the number of times waiting requests that failed to acquire a lock attempt to acquire the cache. The default is 50. (Waits 50ms for each request)

Using the core module

Choose from the following based on your platform

implementation("com.linecorp.cse.reqshield:core:{version}")
implementation("com.linecorp.cse.reqshield:core-reactor:{version}")
implementation("com.linecorp.cse.reqshield:core-kotlin-coroutine:{version}")

//If use local lock

val reqShield = ReqShield<Product>(
    ReqShieldConfiguration(
        setCacheFunction = { key, data, timeToLiveMillis ->
            cacheSpec.put(key, data, timeToLiveMillis)
            true
        },
        getCacheFunction = { key ->
            cacheSpec.get(key)
        },
        isLocalLock = true,
        decisionForUpdate = 70
    )
)

fun getProductCore(productId: String): Product {

    return reqShield.getAndSetReqShieldData(
        key = productId,
        callable = { productRepository.findById(id) },
        timeToLive = 3000).value as Product
    }

//If use global Lock

val reqShield = ReqShield<Product>(
    ReqShieldConfiguration(
        setCacheFunction = { key, data, timeToLiveMillis ->
            cacheSpec.put(key, data, timeToLiveMillis)
            true
        },
        getCacheFunction = { key ->
            cacheSpec.get(key)
        },
        globalLockFunction = { key, timeToLiveMillis ->
            reqShieldCache.globalLock(key, timeToLiveMillis)
        },
        globalUnLockFunction = { key ->
            reqShieldCache.globalUnLock(key)
        },
        isLocalLock = false,
        decisionForUpdate = 70
    )
)

fun getProductCore(productId: String): Product {

    return reqShield.getAndSetReqShieldData(
        key = productId,
        callable = { productRepository.findById(id) },
        timeToLive = 3000).value as Product
    }

As soon as you get an instance of ReqShield, you can use it right away by calling the getAndSetReqShieldData method.
core-reactor, core-kotlin-coroutine have the same structure.
You can also create it as a bean, as shown below.

//If use local lock
@Bean
fun reqShield(): ReqShield<T> =
    ReqShield(
        ReqShieldConfiguration(
        setCacheFunction = {
            key,
            value,
            timeToLiveMillis,
            ->
            redisTemplate.opsForValue().setIfAbsent(key, value, Duration.ofMillis(timeToLiveMillis)) ?: false
        },
        getCacheFunction = { key -> redisTemplate.opsForValue()[key] },
    ),
)

//If use global lock
@Bean
fun reqShield(): ReqShield<T> =
    ReqShield(
        ReqShieldConfiguration(
            setCacheFunction = { key, data, timeToLiveMillis ->
                cacheSpec.put(key, data, timeToLiveMillis)
            },
            getCacheFunction = { key ->
                cacheSpec.get(key)
            },
            globalLockFunction = { key, timeToLiveMillis ->
                reqShieldCache.globalLock(key, timeToLiveMillis)
            },
            globalUnLockFunction = { key ->
                reqShieldCache.globalUnLock(key)
            },
            isLocalLock = false,
            decisionForUpdate = 70
        )
    )

Using the core-spring module

Choose from the following based on your platform

implementation("com.linecorp.cse.reqshield:core-spring:{version}")
implementation("com.linecorp.cse.reqshield:core-spring-webflux:{version}")
implementation("com.linecorp.cse.reqshield:core-spring-webflux-kotlin-coroutine:{version}")

There are tasks that must be done before using the core-spring module.
You need to implement the following interface included in the lib and register it as a bean.

This is because we need a cache handling method to use in AOP.

interface ReqShieldCache<T> {
    fun get(key: String): ReqShieldData<T>?
    fun put(key: String, value: ReqShieldData<T>, timeToLiveMillis: Long)
    fun evict(key: String): Boolean?
    fun globalLock(key: String, timeToLiveMillis: Long): Boolean = true
    fun globalUnLock(key: String): Boolean = true
}

@Service
class ReqShieldCacheImpl<T>(
    private val redisTemplate: RedisTemplate<String, ReqShieldData<T>>,
    private val redisTemplateForGlobalLock: RedisTemplate<String, String>
) : ReqShieldCache<T> {
    override fun get(key: String): ReqShieldData<T>? {
        return redisTemplate.opsForValue()[key]
    }

override fun put(key: String, value: ReqShieldData<T>, timeToLiveMillis: Long) {
    return redisTemplate.opsForValue().set(key, value, Duration.ofMillis(timeToLiveMillis))
}

override fun evict(key: String): Boolean? {
    return redisTemplate.delete(key)
}

//If you use a local lock, you don't need to implement it.
override fun globalLock(key: String, timeToLiveMillis: Long): Boolean {
    return redisTemplateForGlobalLock.opsForValue().setIfAbsent(key, key, Duration.ofMillis(timeToLiveMillis)) ?: false
}

//If you use a local lock, you don't need to implement it.
override fun globalUnLock(key: String): Boolean {
    return redisTemplateForGlobalLock.delete(key)
}

You can then add @ReqShieldCacheable to any method that you want to use ReqShield for.
This annotation requires a spring-aop dependency because it utilizes AOP.
The main params are cacheName, decisionForUpdate, and timeToLive. All of them can be found in the ReqShieldConfiguration part.

@ReqShieldCacheable(cacheName = "product", decisionForUpdate = 70, timeToLive = 6000)
fun getProduct(productId: String): Product {
    log.info("get product (Simulate db request) / productId : $productId")

    return Product(productId, "product_$productId")
}

core-spring-webflux, core-spring-webflux-kotlin-coroutine have the same structure.
The specification for ReqShieldCacheable is shown below.

@Target(AnnotationTarget.FUNCTION)
@Retention(AnnotationRetention.RUNTIME)
@Inherited
annotation class ReqShieldCacheable(
    val cacheName: String,
    val key: String = "",
    val isLocalLock: Boolean = true,
    val lockTimeoutMillis: Long = 3000,
    val decisionForUpdate: Int = 90,
    val condition: String = "",
    val unless: String = "",
    val maxAttemptGetCache: Int = MAX_ATTEMPT_GET_CACHE,
    val timeToLive: Long = 10 * 60 * 1000
)

cacheName : The name used to generate the cache key, along with param and returnType.
key : The value that will be used as the cache key first, if present. (It takes precedence over cacheName.)
isLocalLock : Same as isLocalLock for ReqShieldData.
lockTimeoutMillis : Same as lockTimeoutMillis for ReqShieldData.
decisionForUpdate : Same as decisionForUpdate for ReqShieldData.
condition : TBU
unless : TBU
maxAttemptGetCache : This is the number of times a request that fails to obtain a lock acquires the cache. The default is 50, which means there is a 20ms wait per request.
timeToLive : Same as timeToLive for ReqShieldData.

Performance Test

Test Env

<Application>
1. Spring Boot 2.7.17
2. Redis (docker in local environment) : 6.2.7-alpine
3. Simulate back-end query: sleep 3 second
4. Redis Cache TTL : 20 second

<Req Shield Configuration (req-Shield test only)>
1. decisionForUpdate: 70%
2. timeToLive : 20 second (same as redis cache ttl)

<NGrinder configuration>
1. vUser: 100 (process count 4 * thread count 25)
2. Ramp-up: No
The redis key is set by randomly extracting 10 numbers and setting them to

@Cacheable
@Cacheable(sync=true)
Req-Shield

ReqShield performs 55.31% better than @Cacheable and 102.88% better than @Cacheable(sync=true)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly