Stop wasting memory
When was the last time your machine ran out of RAM turning your work into slow motion nightmare?
Recall the dark day you shut down Spotify, cats memes with an illusive hope of having sufficient RAM for mainstream work tasks...
What if one-third of your app memory was wasted? Meaning you did not have to sacrifice that much and still could have worked with dignity.
Sounds too good to be the truth, right?
You should not trust everything that is written on the web without proof (clickable):
Memory is consumed by objects => objects carry data => the more data == the more memory?
If only we could locate repeatable data patterns to reuse immutable object instances instead of producing new ones...
.NET already provides a concept of reusing immutable data, aka interning.
Two immutable types are mentioned as RAM consumers on the picture - string and Sitecore.Data.ID.
The second picture with reduced memory pressure has only half of string type and does not mention IDs at all.
There is a possibility of interning inside platform since 8.1 Update 3 !
Sitecore.Data.Interning namespace defines foundation for re-using immutable reference types instead of keeping duplicated values inside process memory.
The OOB implementation is plugged into PrefetchData reducing duplicated values for item fields that are known to have limited set of field values (like workflow, workflow state, created by).
WITH DuplicatedFieldValues AS (
SELECT
v.FieldId,
FieldDefinitionRow.[Name],
CONVERT(NVARCHAR(250),v.[Value]) AS [Field Value],
COUNT(1) AS [Hits]
FROM
[VersionedFields] v
JOIN
[Items] FieldDefinitionRow ON FieldDefinitionRow.ID = v.fieldID
WHERE
v.FieldId NOT IN
(
/* Fields already interned OOB by Sitecore.Interning.config */
'BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A' /* updated by */,
'5DD74568-4D4B-44C1-B513-0AF5F4CDA34F' /* created by */,
'52807595-0F8F-4B20-8D2A-CB71D28C6103' /* owner */,
'3E431DE1-525E-47A3-B6B0-1CCBEC3A8C98' /* workflow state */
)
GROUP BY
FieldId, [Name], CONVERT(NVARCHAR(250),[Value])
HAVING
COUNT(*) > 500 /* How many same field values must be met to be shown */)
SELECT * FROM DuplicatedFieldValues
ORDER BY [Hits] DESC
SELECT
REPLACE(CONCAT('<', [Name], '>'), ' ','_') AS [BeginTag],
CONCAT('{',FieldId,'}') AS [FieldId],
REPLACE(CONCAT('</', [Name], '>'), ' ','_') AS [EndTag],
SUM(Hits) AS [Duplicates],
COUNT(1) AS [Unique Values]
FROM
DuplicatedFieldValues
GROUP BY
[FieldId], [Name]
ORDER BY
[Duplicates] DESC
This query will return duplicated values met in VersionedFields more than X times.
You can review & add bubbled field ids to the Sitecore.Interning.config, fieldIdsToIntern section.
You could play with SQL Server table statistics ( db-> versionedFields -> statistics) to locate repeatable values and where are they coming from.
My test plan was to load all solution content via following snipped with disabled cache size limits:
var db = Sitecore.Configuration.Factory.GetDatabase("master");
using (new SecurityDisabler())
{
var root = db.GetItem(new ID("{110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9}"));
var descendants = root.Axes.GetDescendants();
MainUtil.Nop(descendants);
}
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, true, true);
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, true, true);
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, true, true);
Once page done its job, memory snapshot was collected via Task Manager.
The steps were repeated with ON/OFF interning to get the difference. Resulting snapshots were dropped to dotMemory.
A) Interning functionality runs on object pool design pattern that will pre-allocate large arrays to avoid expensive/blocking in-memory growth.
B) .NET processes do not like giving back memory with no reason. The process size may look huge, while internal GC heaps will be almost empty. Process will count on conquered memory for further allocations requests.
Immutable data types allow re-using already obtained values => reducing count of objects in application.
The provided example highlights interning truncates ~3.25 million objects from memory (~30 % of total heap count).