Why store all data in memory?
When it comes to storing website or backend data, the first choice for most rational people is to opt for an SQL database.
However, sometimes the thought occurs that the data model is not suitable for SQL: for example, when building a search engine or social graph, there is a need to conduct searches based on complex relationships between objects.
The worst situation is when you are working in a team, and a colleague does not know how to construct fast queries. How much time have you spent solving N+1 problems and building additional indexes just to ensure that the SELECT on the main page executes in a reasonable timeframe?
Another popular approach is NoSQL. A few years ago, there was a lot of hype surrounding this topic — for any convenient case, MongoDB was deployed, and everyone was happy with the JSON document responses. (By the way, how many hacks did you have to implement due to cyclic references in the documents?).
I suggest trying another alternative approach — why not try storing all data in the application's memory, periodically saving it to an arbitrary storage (file, remote database)?
Memory has become cheap, and any possible data for most small and medium projects will fit into 1 GB of memory. (For example, my favorite home project — , which maintains daily statistics and history of my expenses, balances, and transactions for a year and a half, consumes only 45 MB of memory.)
Pros:
- Accessing data becomes easier — there’s no need to worry about queries, lazy loading, or ORM peculiarities; work is done with regular C# objects;
- No problems related to access from different threads;
- Very fast — there are no network requests, code does not need to be translated into query language, and no (de)serialization of objects is needed;
- It is acceptable to store data in any form — whether in XML on disk, SQL Server, or Azure Table Storage.
Cons:
- Horizontal scaling is lost, and consequently, zero downtime deployment cannot be achieved;
- If the application crashes, some data may be partially lost. (But our application never crashes, right?)
Initially, a check is performed: does the client device support power via PoE? A voltage of 2.8 to 10 volts is supplied, and the input resistance is determined. If the results obtained are satisfactory for powering via PoE, the power device proceeds to the next stage.
The algorithm is as follows:
- At startup, a connection is established with the data storage, and data is loaded;
- An object model is built, along with primary indexes and relationship indexes (1:1, 1:Many);
- A subscription is created to changes in object properties (INotifyPropertyChanged) and to addition or removal of items in a collection (INotifyCollectionChanged);
- When the subscription is triggered, the changed object is added to the queue for writing to the data storage;
- Periodically (on a timer), changes are saved to storage in the background thread;
- When exiting the application, changes are also saved to storage.
Code Example
Add the necessary dependencies
// Основная библиотека
Install-Package OutCode.EscapeTeams.ObjectRepository
// Хранилище данных, в котором будут сохраняться изменения
// Используйте то, которым будете пользоваться.
Install-Package OutCode.EscapeTeams.ObjectRepository.File
Install-Package OutCode.EscapeTeams.ObjectRepository.LiteDb
Install-Package OutCode.EscapeTeams.ObjectRepository.AzureTableStorage
// Опционально - если нужно хранить модель данных для Hangfire
// Install-Package OutCode.EscapeTeams.ObjectRepository.HangfireDescribe the data model that will be stored in the storage
public class ParentEntity : BaseEntity
{
public ParentEntity(Guid id) => Id = id;
}
public class ChildEntity : BaseEntity
{
public ChildEntity(Guid id) => Id = id;
public Guid ParentId { get; set; }
public string Value { get; set; }
}Then the object model:
public class ParentModel : ModelBase
{
public ParentModel(ParentEntity entity)
{
Entity = entity;
}
public ParentModel()
{
Entity = new ParentEntity(Guid.NewGuid());
}
public Guid? NullableId => null;
// Example of a 1:Many relation
public IEnumerable Children => Multiple(x => x.ParentId);
protected override BaseEntity Entity { get; }
}
public class ChildModel : ModelBase
{
private ChildEntity _childEntity;
public ChildModel(ChildEntity entity)
{
_childEntity = entity;
}
public ChildModel()
{
_childEntity = new ChildEntity(Guid.NewGuid());
}
public Guid ParentId
{
get => _childEntity.ParentId;
set => UpdateProperty(() => _childEntity.ParentId, value);
}
public string Value
{
get => _childEntity.Value;
set => UpdateProperty(() => _childEntity.Value, value);
}
// Access by index search
public ParentModel Parent => Single(ParentId);
protected override BaseEntity Entity => _childEntity;
}And finally, the repository class for data access:
public class MyObjectRepository : ObjectRepositoryBase
{
public MyObjectRepository(IStorage storage) : base(storage, NullLogger.Instance)
{
IsReadOnly = true; // For tests, allows not to save changes to the database
AddType((ParentEntity x) => new ParentModel(x));
AddType((ChildEntity x) => new ChildModel(x));
// If Hangfire is used and there is a need to store the data model for Hangfire in ObjectRepository
// this.RegisterHangfireScheme();
Initialize();
}
}Create an instance of ObjectRepository:
var memory = new MemoryStream();
var db = new LiteDatabase(memory);
var dbStorage = new LiteDbStorage(db);
var repository = new MyObjectRepository(dbStorage);
await repository.WaitForInitialize();If HangFire will be used in the project
public void ConfigureServices(IServiceCollection services, ObjectRepository objectRepository)
{
services.AddHangfire(s => s.UseHangfireStorage(objectRepository));
}Inserting a new object:
var newParent = new ParentModel()
repository.Add(newParent);In this call, the object ParentModel It is added both to the local cache and to the write queue in the database. Therefore, this operation takes O(1), and you can work with this object immediately.
For example, to find this object in the repository and ensure that the returned object is the same instance:
var parents = repository.Set();
var myParent = parents.Find(newParent.Id);
Assert.IsTrue(ReferenceEquals(myParent, newParent));What happens in this case? Set() brings back TableDictionary, which contains ConcurrentDictionary and provides additional functionality for primary and secondary indexes. This allows methods for searching by Id (or other arbitrary user indexes) without fully iterating through all objects.
When adding objects to ObjectRepository , a subscription to changes in their properties is added, so any change in properties also leads to adding this object to the write queue.
Updating properties from the outside looks the same as working with a POCO object:
myParent.Children.First().Property = "Updated value";You can remove an object in the following ways:
repository.Remove(myParent);
repository.RemoveRange(otherParents);
repository.Remove(x => !x.Children.Any());In this case, the object is also added to the delete queue.
How does saving work?
ObjectRepository When tracked objects are modified (such as adding, deleting, or changing properties), it raises an event ModelChanged, which is subscribed to by IStorage. Implementations IStorage when the event occurs ModelChanged enqueue changes into 3 queues — for addition, for update, and for deletion.
Also, implementations IStorage create a timer during initialization that calls for saving changes every 5 seconds.
Additionally, there is an API for forcing a save: ObjectRepository.Save().
Before each save, meaningless operations (e.g., duplicate events — when an object was changed twice or quick addition/deletion of objects) are first removed from the queues, and only then the actual save occurs.
In all cases, the current object is saved in its entirety, which means that there may be situations where objects are saved in a different order than they were changed, including more recent versions of objects being saved than at the time they were added to the queue.
What else is there?
- All libraries are based on .NET Standard 2.0. They can be used in any modern .NET project.
- The API is thread-safe. Internal collections are based on ConcurrentDictionary, event handlers either have locks or do not require them.
The only thing to remember is to call ObjectRepository.Save(); - Arbitrary indexes (require uniqueness):
repository.Set().AddIndex(x => x.Value);
repository.Set().Find(x => x.Value, "myValue");Who uses this?
Personally, I started using this approach in all my hobby projects because it's convenient and doesn't require significant costs for writing a data access layer or deploying heavy infrastructure. For me, storing data in LiteDB or a file is typically sufficient.
But in the past, when my team worked on the now-defunct startup EscapeTeams (I thought there they are, the money — but no, it was just another experience) — we used Azure Table Storage for data storage.
Plans for the future
I want to address one of the main downsides of this approach — horizontal scaling. This requires either distributed transactions (sic!), or a firm decision that the same data from different instances should not change, or let them change according to the principle of "last one wins."
From a technical standpoint, I see the following possible scheme:
- Store instead of the object model: EventLog and Snapshot
- Find other instances (add endpoints for all instances in settings? UDP discovery? master/slave?)
- Replicate EventLog between instances using any consensus algorithm, for example RAFT.
There's also another problem that concerns me — cascading deletes or detecting cases of object deletion that are referenced from other objects.
Source Code
If you made it this far — then you only have to read the code, which can be found on GitHub:
Source: habr.com
