🥇The tale of how Realm's cascading deletion won the long run

All users take fast launch and responsive UI in mobile applications for granted. If the application starts for a long time, the user starts to feel sad and angry. It is easy to spoil the client experience or completely lose the user even before he has started using the application.

Once we found that the Dodo Pizza application starts up on average 3 seconds, and for some "lucky ones" it takes 15-20 seconds.

Under the cut, a story with a happy ending: about the growth of the Realm database, a memory leak, how we accumulated nested objects, and then pulled ourselves together and fixed everything.

Author of the article: Maxim Kachinkin — Android developer at Dodo Pizza.

Three seconds from clicking on the application icon to onResume() of the first activity is infinity. And for some users, the startup time reached 15-20 seconds. How is that even possible?

A very brief summary for those who have no time to read
Our Realm database has grown endlessly. Some nested objects were not deleted, but constantly accumulated. Application startup time gradually increased. Then we fixed it, and the launch time came to the target - it became less than 1 second and no longer grows. In the article, an analysis of the situation and two solutions - in a quick way and in a normal way.

Search and analysis of the problem

Today, any mobile app needs to launch quickly and be responsive. But it's not just about the mobile app. The user experience of interaction with the service and the company is a complex thing. For example, in our case, delivery speed is one of the key indicators for a pizza service. If the delivery is fast, then the pizza will be hot, and the customer who wants to eat now will not have to wait long. For the application, in turn, it is important to create a feeling of a fast service, because if the application only starts up for 20 seconds, then how long will you have to wait for pizza?

At first, we ourselves were faced with the fact that sometimes the application starts up for a couple of seconds, and then other colleagues began to reach us with complaints that it was “long”. But we were not able to repeat this situation stably.

How long is how long? According to Google documentation, if the cold start of the application takes less than 5 seconds, then it is considered "as if normal". Dodo Pizza Android app launched (according to Firebase metric _app_start) at cold start an average of 3 seconds - "Not great, not terrible", as they say.

But then complaints began to appear that the application starts for a very, very, very long time! To begin with, we decided to measure what a "very, very, very long" is. And we used Firebase trace for this App start trace.

This standard trace measures the time between the moment the user opens the app and the moment the onResume() of the first activity is executed. In the Firebase Console, this metric is called _app_start. It turned out that:

Startup times for users above the 95th percentile are nearly 20 seconds (some more), despite a median cold start time of less than 5 seconds.
Launch time is not a constant value, but grows with time. But sometimes there are falls. We found this pattern when we increased the analysis scale to 90 days.

Two thoughts came to mind:

Something is leaking.
This “something” is discarded after the release and then leaks again.

“Probably something with the database,” we thought, and we were right. Firstly, we use the database as a cache, we clear it during migration. Secondly, the database is loaded when the application starts. Everything converges.

What's wrong with the Realm database

We began to check how the content of the database changes over the lifetime of the application, from the first installation and further in the process of active use. You can view the contents of the Realm database through stetho or in more detail and clearly by opening the file via Realm Studio. To view the contents of the database via ADB, copy the Realm database file:

adb exec-out run-as ${PACKAGE_NAME} cat files/${DB_NAME}

Looking at the contents of the database at different times, we found out that the number of objects of a certain type is constantly increasing.

The picture shows a fragment of Realm Studio for two files: on the left - the application base some time after installation, on the right - after active use. It can be seen that the number of objects ImageEntity и MoneyType increased greatly (the screenshot shows the number of objects of each type).

Linking Database Growth to Startup Time

Uncontrolled database growth is very bad. But how does this affect application startup time? To measure this is quite simple through the ActivityManager. Starting with Android 4.4, logcat displays a log with the string Displayed and the time. This time is equal to the interval from the moment the application starts to the end of the drawing of the activity. During this time, the following events take place:

Starting a process.
Object initialization.
Creation and initialization of the activity.
Creating a layout.
Application drawing.

Suits us. If you run ADB with the -S and -W flags, you can get an expanded output with the start time:

adb shell am start -S -W ru.dodopizza.app/.MainActivity -c android.intent.category.LAUNCHER -a android.intent.action.MAIN

If you rake from there grep -i WaitTime time, you can automate the collection of this metric and look at the results visually. The graph below shows the dependence of the application launch time on the number of cold launches of the application.

At the same time, the dependence of the size and growth of the database was the same, which grew from 4 MB to 15 MB. In total, it turns out that over time (with the growth of cold launches), both the application launch time and the size of the database grew. We have a hypothesis. Now it was necessary to confirm the dependence. Therefore, we decided to remove the “leaks” and see if this speeds up the launch.

Causes of Infinite Database Growth

Before removing the "leaks", it is worthwhile to figure out why they appeared at all. To do this, let's remember what Realm is.

Realm is a non-relational database. It allows you to describe relationships between objects in a similar way that many ORM relational databases on Android describe. At the same time, Realm stores objects directly in memory with the least number of transformations and mappings. This allows you to read data from the disk very quickly, which is Realm's strong point, for which it is loved.

(For the purposes of this article, this description will be enough for us. You can read more about Realm in the cool documentation or in their Academy).

Many developers are used to working more with relational databases (eg ORM databases with SQL under the hood). And things like cascading data deletion often seem like a no-brainer. But not in Realm.

By the way, the feature of cascading deletion has been asked to be done for a long time. This finalization и anotherassociated with it, was actively discussed. It felt like it would be done soon. But then everything turned into the introduction of strong and weak links, which would also automatically solve this problem. For this task, he was quite lively and active. pull request, which has been paused due to internal difficulties.

Data leak without cascading deletion

How exactly is data leaked, if we hope for a non-existent cascading deletion? If you have nested Realm objects, then they must be deleted.
Consider a (nearly) real-life example. We have an object CartItemEntity:

@RealmClass
class CartItemEntity(
 @PrimaryKey
 override var id: String? = null,
 ...
 var name: String = "",
 var description: String = "",
 var image: ImageEntity? = null,
 var category: String = MENU_CATEGORY_UNKNOWN_ID,
 var customizationEntity: CustomizationEntity? = null,
 var cartComboProducts: RealmList<CartProductEntity> = RealmList(),
 ...
) : RealmObject()

The product in the cart has different fields, including a picture ImageEntity, customized ingredients CustomizationEntity. Also, the product in the basket can be a combo with its own set of products RealmList (CartProductEntity). All listed fields are Realm-objects. If we insert a new object (copyToRealm() / copyToRealmOrUpdate()) with the same id, this object will be completely overwritten. But all internal objects (image, customizationEntity and cartComboProducts) will lose their connection with the parent and remain in the database.

Since the connection with them is lost, we no longer read them and do not delete them (unless we explicitly access them or clear the entire “table”). We called it "memory leaks".

When we work with Realm, we must explicitly go through all the elements and explicitly delete everything before such operations. This can be done, for example, like this:

val entity = realm.where(CartItemEntity::class.java).equalTo("id", id).findFirst()
if (first != null) {
 deleteFromRealm(first.image)
 deleteFromRealm(first.customizationEntity)
 for(cartProductEntity in first.cartComboProducts) {
   deleteFromRealm(cartProductEntity)
 }
 first.deleteFromRealm()
}
// и потом уже сохраняем

If you do this, then everything will work as it should. In this example, we assume that there are no other nested Realm objects inside image, customizationEntity, and cartComboProducts, so there are no other nested loops and deletes.

The "quick fix" solution

First of all, we decided to clean up the fastest growing objects and check the results - whether this solves our original problem. First, the most simple and intuitive solution was made, namely: each object should be responsible for deleting its children. To do this, we introduced such an interface that returned a list of its nested Realm objects:

interface NestedEntityAware {
 fun getNestedEntities(): Collection<RealmObject?>
}

And implemented it in our Realm objects:

@RealmClass
class DataPizzeriaEntity(
 @PrimaryKey
 var id: String? = null,
 var name: String? = null,
 var coordinates: CoordinatesEntity? = null,
 var deliverySchedule: ScheduleEntity? = null,
 var restaurantSchedule: ScheduleEntity? = null,
 ...
) : RealmObject(), NestedEntityAware {

 override fun getNestedEntities(): Collection<RealmObject?> {
   return listOf(
       coordinates,
       deliverySchedule,
       restaurantSchedule
   )
 }
}

В getNestedEntities we return all children in a flat list. And each child object can also implement the NestedEntityAware interface, indicating that it has internal Realm objects to delete, for example ScheduleEntity:

@RealmClass
class ScheduleEntity(
 var monday: DayOfWeekEntity? = null,
 var tuesday: DayOfWeekEntity? = null,
 var wednesday: DayOfWeekEntity? = null,
 var thursday: DayOfWeekEntity? = null,
 var friday: DayOfWeekEntity? = null,
 var saturday: DayOfWeekEntity? = null,
 var sunday: DayOfWeekEntity? = null
) : RealmObject(), NestedEntityAware {

 override fun getNestedEntities(): Collection<RealmObject?> {
   return listOf(
       monday, tuesday, wednesday, thursday, friday, saturday, sunday
   )
 }
}

And so on nesting of objects can be repeated.

Then we write a method that recursively removes all nested objects. Method (made as an extension) deleteAllNestedEntities gets all top-level objects and method deleteNestedRecursively recursively deletes everything nested using the NestedEntityAware interface:

fun <T> Realm.deleteAllNestedEntities(entities: Collection<T>,
 entityClass: Class<out RealmObject>,
 idMapper: (T) -> String,
 idFieldName : String = "id"
 ) {

 val existedObjects = where(entityClass)
     .`in`(idFieldName, entities.map(idMapper).toTypedArray())
     .findAll()

 deleteNestedRecursively(existedObjects)
}

private fun Realm.deleteNestedRecursively(entities: Collection<RealmObject?>) {
 for(entity in entities) {
   entity?.let { realmObject ->
     if (realmObject is NestedEntityAware) {
       deleteNestedRecursively((realmObject as NestedEntityAware).getNestedEntities())
     }
     realmObject.deleteFromRealm()
   }
 }
}

We did this with the fastest growing objects and checked what happened.

As a result, those objects that we covered with this solution stopped growing. And the overall growth of the base slowed down, but did not stop.

The "normal" solution

The base, although it began to grow more slowly, but still grew. So we started looking further. In our project, data caching in Realm is very actively used. Therefore, writing all the nested objects for each object is laborious, plus the risk of error increases, because you can forget to specify the objects when changing the code.

I wanted to make sure that I did not use interfaces, but that everything worked by itself.

When we want something to work on its own, we have to use reflection. To do this, we can loop through each field of the class and check if it is a Realm object or a list of objects:

RealmModel::class.java.isAssignableFrom(field.type)

RealmList::class.java.isAssignableFrom(field.type)

If the field is a RealmModel or a RealmList, then we add the object of this field into a list of nested objects. Everything is exactly the same as we did above, only here it will be done by itself. The cascade deletion method itself is very simple and looks like this:

fun <T : Any> Realm.cascadeDelete(entities: Collection<T?>) {
 if(entities.isEmpty()) {
   return
 }

 entities.filterNotNull().let { notNullEntities ->
   notNullEntities
       .filterRealmObject()
       .flatMap { realmObject -> getNestedRealmObjects(realmObject) }
       .also { realmObjects -> cascadeDelete(realmObjects) }

   notNullEntities
       .forEach { entity ->
         if((entity is RealmObject) && entity.isValid) {
           entity.deleteFromRealm()
         }
       }
 }
}

Extension filterRealmObject filters and passes only Realm objects. Method getNestedRealmObjects finds all nested Realm objects through reflection and adds them to a linear list. Then we do the same recursively. When deleting, you need to check the object for validity isValid, because it can happen that different parent objects can have nested identical ones. It is better not to allow this and just use id auto-generation when creating new objects.

Complete implementation of the getNestedRealmObjects method

private fun getNestedRealmObjects(realmObject: RealmObject) : List<RealmObject> {
 val nestedObjects = mutableListOf<RealmObject>()
 val fields = realmObject.javaClass.superclass.declaredFields

// Проверяем каждое поле, не является ли оно RealmModel или списком RealmList
 fields.forEach { field ->
   when {
     RealmModel::class.java.isAssignableFrom(field.type) -> {
       try {
         val child = getChildObjectByField(realmObject, field)
         child?.let {
           if (isInstanceOfRealmObject(it)) {
             nestedObjects.add(child as RealmObject)
           }
         }
       } catch (e: Exception) { ... }
     }

     RealmList::class.java.isAssignableFrom(field.type) -> {
       try {
         val childList = getChildObjectByField(realmObject, field)
         childList?.let { list ->
           (list as RealmList<*>).forEach {
             if (isInstanceOfRealmObject(it)) {
               nestedObjects.add(it as RealmObject)
             }
           }
         }
       } catch (e: Exception) { ... }
     }
   }
 }

 return nestedObjects
}

private fun getChildObjectByField(realmObject: RealmObject, field: Field): Any? {
 val methodName = "get${field.name.capitalize()}"
 val method = realmObject.javaClass.getMethod(methodName)
 return method.invoke(realmObject)
}

As a result, in our client code, we use "cascading deletion" for each data change operation. For example, for an insert operation, it looks like this:

override fun <T : Entity> insert(
 entityInformation: EntityInformation,
 entities: Collection<T>): Collection<T> = entities.apply {
 realmInstance.cascadeDelete(getManagedEntities(entityInformation, this))
 realmInstance.copyFromRealm(
     realmInstance
         .copyToRealmOrUpdate(this.map { entity -> entity as RealmModel }
 ))
}

Method First getManagedEntities gets all the added objects, and then the method cascadeDelete recursively removes all collected objects before writing new ones. We end up using this approach throughout the application. Memory leaks in Realm are completely gone. Having carried out the same measurement of the dependence of the launch time on the number of cold launches of the application, we see the result.

The green line shows the dependence of the application startup time on the number of cold starts during automatic cascade deletion of nested objects.

Results and Conclusions

The ever-growing Realm database was causing the application to launch very slowly. We've released an update with our own "cascading deletion" of nested objects. And now we track and evaluate how our decision affected the application launch time through the _app_start metric.

For analysis, we take a time period of 90 days and see: the application launch time, both the median and the one that falls on the 95th percentile of users, began to decrease and no longer rises.

If you look at the seven-day chart, the _app_start metric is completely adequate and is less than 1 second.

Separately, it is worth adding that, by default, Firebase sends notifications if the median value of _app_start exceeds 5 seconds. However, as we can see, you should not rely on this, but it is better to go in and check it explicitly.

The Realm database is special in that it is a non-relational database. Despite its simple use, similarity to working with ORM solutions and object linking, it does not have a cascading deletion.

If this is not taken into account, then nested objects will accumulate, "leak". The database will grow constantly, which in turn will affect the slowdown or application startup.

I shared our experience on how to quickly do a cascading deletion of objects in Realm, which is not yet out of the box, but about which for a long time they say и they say. In our case, this greatly accelerated the launch time of the application.

Despite the discussion of the imminent appearance of this feature, the lack of cascading deletion in Realm is by design. If you are designing a new application, then keep this in mind. And if you are already using Realm, check if you have such problems.

Source: habr.com

The tale of how cascading deletion in Realm's long launch won