Core Data issues with memory allocation

I’m using Core Data in an application that I am developing that uses the Address Book.   I created some test data and started writing some code that imported this information into Core Data.    Curious about the memory usage of this import, I used the Allocations Instrument for analysis and thus began a long process where I believe I may have uncovered several bugs related to Core Data memory management.

What prompted the analysis?

I have seen references indicating that your app risks being shut down for memory usage when you use more than 20 MB of RAM.   A couple interesting articles include:

Memory limit and iOS memory allocation in iphone SDK?

ios app maximum memory budget

Apple’s recommendations

The documentation does discuss how to efficiently use memory with Core Data for imports in the sections Reducing Peak Memory Footprint and Reducing Memory Overhead.   The first reference talks about strong reference cycles caused by relationships and the second about pruning the object graph with faulting and context reset.

CDSnapshot

Using Apple’s recommendations, I first attempted to fault managed objects after a save with refreshObject: mergeChanges:NO

@property (nonatomic, strong) UIManagedDocument *contactDatabase;

...
for (NSManagedObject *mo in [self.contactDatabase.managedObjectContext registeredObjects]) {
    [self.contactDatabase.managedObjectContext refreshObject:mo mergeChanges:NO];
}

I noticed that memory wasn’t being freed as expected and there were allocations with the name _CDSnapshot_<Entity> as shown below.

CDSnapshot_1

I searched and found that I was not alone in seeing these allocations:

Setter for NSManagedObject creates _CDSnapshot_Provence_

Multiple instances of _CDSnapshot_[entityName]_

I searched the documentation and found this section:

Snapshot Management

“An application that fetches hundreds of rows of data can build up a large cache of snapshots. Theoretically, if enough fetches are performed, a Core Data-based application can contain all the contents of a store in memory. Clearly, snapshots must be managed in order to prevent this situation.

Responsibility for cleaning up snapshots rests with a mechanism called snapshot reference counting. This mechanism keeps track of the managed objects that are associated with a particular snapshot—that is, managed objects that contain data from a particular snapshot. When there are no remaining managed object instances associated with a particular snapshot (which Core Data determines by maintaining a list of these references), the strong reference to the snapshot is broken.”

Clearly something wasn’t working in my case.

GitHub Demonstration project for Core Data Issues

I decided to create a demonstration project to provide test cases for all the issues that I was encountering, as a means of documenting them for reproducibility when I filed a bug using the Apple Bug Reporter.

The project is scottcarter/CoreDataMemoryBug

Take a look at the README and BugReport files for detailed information on using the project.

What are are the main issues?

My goal was to release all memory held by Core Data after I did a save.  I had no need to keep any of the managed objects in memory after an import for my particular use case.   The main issues I found were:

  • Faulting all objects with refreshObject did not release all memory.
  • Performing a context reset did not release all memory.
  • setUndoManager:nil (this should be the default) was having adverse effects.
  • Allocations for NSTemporaryObjectID_default were created and not released.
  • I could not turn off all warnings about missing inverse relationships (when I tried this approach) using MOMC_NO_INVERSE_RELATIONSHIP_WARNINGS

I tried several experiments to try and workaround the memory allocation issues.

refreshObject followed by context reset

This worked to release most of the Core Data held memory if I didn’t have a reference loop caused by an inverse relationship between my entities.  In the case of an inverse relationship this had no effect.

Without an inverse relationship the combination was effective, but only if the reset that followed the refreshObject calls was not in the same event loop!

What does that mean?

This did not work to release memory:

[self.contactDatabase saveToURL:self.contactDatabase.fileURL
                   forSaveOperation:UIDocumentSaveForOverwriting
                  completionHandler:^(BOOL success) {
                      if(success){
                          for (NSManagedObject *mo in [self.contactDatabase.managedObjectContext registeredObjects]) {
                              [self.contactDatabase.managedObjectContext refreshObject:mo mergeChanges:NO];
                          }

                         [self.contactDatabase.managedObjectContext reset];

                      }

                  }];

This worked:


- (void)resetContext
{
    [self.contactDatabase.managedObjectContext reset];
}

...

[self.contactDatabase saveToURL:self.contactDatabase.fileURL
                   forSaveOperation:UIDocumentSaveForOverwriting
                  completionHandler:^(BOOL success) {
                      if(success){
                         for (NSManagedObject *mo in [self.contactDatabase.managedObjectContext registeredObjects]) {
                              [self.contactDatabase.managedObjectContext refreshObject:mo mergeChanges:NO];
                          }

                          [NSTimer scheduledTimerWithTimeInterval:0.0
                                                           target:self
                                                         selector:@selector(resetContext)
                                                         userInfo:nil
                                                          repeats:NO];

                      }

                  }];

Inverse Relationships

I was discouraged that the combination of faulting and context reset was not effective in the presence of an inverse relationship between entities, so I dug a little deeper into this topic.

The first reference that I came across that discussed retain loops with Core Data was from an older posting by Ben Trumbull in the thread Core Data huge memory usage – is this right?

I thought about not using an inverse relationship in my application since it wasn’t needed, but got an Xcode warning when I tried it.   The warning can supposedly be disabled by setting MOMC_NO_INVERSE_RELATIONSHIP_WARNINGS to YES in the build settings which I ran across in the thread How to disable no inverse relationship warning for CoreData in Xcode 4.2?

Unfortunately doing this (for both PROJECT and TARGETS build settings) only got rid of one of a pair of identical warnings for me.

While it is legal not to have an inverse relationship, there are definitely reasons that you do want to have one as mentioned in the thread Why does an entity need an inverse?   The Apple docs also discuss this in the section Unidirectional Relationships where they state

“Not modeling a relationship in both directions, however, imposes on you a great number of responsibilities, to ensure the consistency of the object graph, for change tracking, and for undo management. For this reason, the practice is strongly discouraged.”

setUndoManager:nil

In the documentation for NSManagedObjectContext:

“You can set the undo manager to nil to disable undo support. This provides a performance benefit if you do not want to support undo for a particular context, for example in a large import process.”

With the note:  “on iOS, the undo manager is nil by default.”

Thus, calling setUndoManager:nil should have no effect.   I found problems though when issuing this call:

  • The combination of refreshObject/context reset is no longer effective (even with no inverse relationships)
  • A hang is caused in a call to removePersistentStore of the persistentStoreCoordinator of a
    managed object context (see next section below).  This only happens when an inverse relationship is used between entities.

The Sledge Hammer – recreating the persistent store

sledge-hammer

I was looking for some way to more thoroughly clean up allocations held by Core Data, when I ran across these threads:

How do I delete all objects from my persistent store in Core Data?

Delete/Reset all entries in Core Data?

I didn’t actually want to remove the underlying file, so I ended up with code that looked like:

// Remove the persistent store for our context.
//
- (void)recreatePersistentStore
{
NSError *error = nil;
NSManagedObjectContext *managedObjectContext = self.contactDatabase.managedObjectContext;

// Retrieve the store URL
NSURL * storeURL = [[managedObjectContext persistentStoreCoordinator] URLForPersistentStore:[[[managedObjectContext persistentStoreCoordinator] persistentStores] lastObject]];

[managedObjectContext lock];  // Lock the current context

// Remove the store from the current managedObjectContext
if ([[managedObjectContext persistentStoreCoordinator] removePersistentStore:[[[managedObjectContext persistentStoreCoordinator] persistentStores] lastObject] error:&error])
{

// Recreate the persistent store
if(![[managedObjectContext persistentStoreCoordinator] addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:nil error:&error]){
NSLog(@"Could not add persistent store");
}
}

else {
NSLog(@"Could not remove persistent store");
}

[managedObjectContext reset];
[managedObjectContext unlock];

NSLog(@"Completed recreatePersistentStore");
}

Calling my method recreatePersistentStore from the save completion block (instead of refreshObject and/or reset) worked nicely to free all Core Data held memory with or without an inverse relationship.

NSTemporaryObjectID_default

In addition to memory that I might be able to free using refreshObject/reset or recreatePersistentStore, I noticed that there were lots of allocations still present called NSTemporaryObjectID_default.

NSTemporaryObjectID_default

Collectively they used a lot of living references and occupied a sizable amount of memory.

I found a solution that was inspired by the thread When to call obtainPermanentIDsForObjects:?

Before I do a save, I obtain permanent ids for all the managed objects:

// Convert all temporary ids to permanent ids.

NSArray *objArr = [[self.contactDatabase.managedObjectContext registeredObjects] allObjects];
BOOL result = [self.contactDatabase.managedObjectContext obtainPermanentIDsForObjects:objArr error:nil];
if(result == NO){
    NSLog(@"WARNING: Was not able to obtain permanent ids for all objects.");
}

This solved the problem and released all occurrences of NSTemporaryObjectID_default

Conclusion

I’ve submitted a bug report to Apple and pointed them to my test cases at:

scottcarter/CoreDataMemoryBug

I prefer to use inverse relationships (as recommended by Apple), so for now my solution to reclaim memory held by Core Data is:

  • Obtain permanent ids for all managed objects before a save
  • Avoid a call to setUndoManager:nil on the managed object context
  • Recreate the persistent store after a save

Admittedly the last item isn’t a solution that I like (I’d prefer to fault the objects), but it allows me to move on until I get information back from my bug report.

Advertisements

13 responses to “Core Data issues with memory allocation

  1. Found another article discussing obtainPermanentIDsForObjects:

    Core Data could not fullfil fault for object after obtainPermanantIDs
    http://stackoverflow.com/questions/11321717/core-data-could-not-fullfil-fault-for-object-after-obtainpermanantids/

    Like

  2. Scott- Nice Article. I believe I have run into a very similar problem. I would really appreciate it if you would post any results you receive from Apple.
    I will probably use your strategy of recreating the persistent store after a save which seems to solve all of my memory problems that relate to core data and more specifically relationships and memory not being deallocated like I think it should. It feels “wrong” of course but like you said it is a way to move on to the rest of the project until I can gather more information. Thanks Again.

    Like

  3. Pingback: The development of Contacts2Web | Finalize.com: My journey with iOS and other code adventures

  4. Hi! Really great article, thanks for your analysis. I am having the same issue with remaining memory and iOS not able to free it. It is quite frustrating because I have tried Apples guidelines, and many other suggestions like yours but I still keep having this large memory not able to be drained. Did you ever had response on your Apples bug report?

    Thanks!

    Like

  5. Hi Tamara,

    I have not yet had any response to my bug report. I will post any results when I hear something more.

    Like

  6. Pingback: Core Data Import - Not releasing memory | BlogoSfera

  7. Thanks for writing the article. Getting rid of inverse relationships is a complete hack. Even if it worked, I wouldn’t do it. We used your “sledgehammer” technique.

    Any update to the bug report?

    Like

  8. Hi Damon,

    I initially filed one bug report that covered all the issues that I was seeing with Core Data. The Apple Engineering team asked me to split the report into separate bug reports covering more distinct issues, which I did and filed on January 5, 2013. There has been no further activity on these reports since then. I just added a comment on one of them to ping them.

    I’ve heard that Apple might pay more attention to issues that are widely reported. I would suggest that you and others reading this blog post file your own bug reports as well. You may wish to reference this blog post in your bug report.

    Thanks.

    Like

  9. Same issues here still. I’m guessing these bugs havent been resolved. We are using UIManagedDocument to manage our ManagedObjectContexts and saving and it likes like you were doing the same Scott. Do you know if these issues still exist if you’re not using UIManagedDocument?

    Like

  10. Hi Matt,

    I have not investigated whether the issues exist without using UIManagedDocument. No feedback from Apple on these bugs I filed. Please file a bug report yourself to help draw attention to the issues and reference this blog post.

    Thanks.

    Like

  11. Update 11/8/14: Just tried my sample project again (CoreDataMemoryBug) and noted that problems still exist when profiling with Xcode 6 and the iOS 8 simulator.

    Like

  12. Hi Scott,

    I am having this issue in mid-2015, and what I found was after I had turned my core data objects back into faults (they were objects that stored full-screen images), if I called [managedObjectContext registeredObjects], it purged the objects from memory, thus reducing the memory profile of the app! (Assuming of course no application object was retaining the images).

    Something about [managedObjectContext registeredObjects] forced CoreData to acknowledge faults. Hope this helps!

    Like

  13. Also, I should mention that my core data objects (the full screen images), have parent objects, and I need to call refreshObject:mergeChanges on the PARENT object for the memory to be purged.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s