Misconfigured examine settings lead to OutOfMemoryExceptions on Azure webapp

[bowserm]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

13.5.2

Bug summary

We have an Umbraco 13.5.2 site hosted in Azure webapps with a single backoffice instance and a single public instance both connected to the same database. The public instance has not been scaled out. Both the backoffice and public instances were configured with "Umbraco__CMS__Examine__LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory" which is a problem. About once a month, the public instance would crash with OutOfMemoryExceptions. It would crash repeatedly. The only way to get the site back up was to route all traffic away from the public site until it could stabilize. We have since fixed the configuration, so the public instance has "Umbraco__CMS__Examine__LuceneDirectoryFactory": "TempFileSystemDirectoryFactory"

The mystery is the OutOfMemoryExceptions. Why was this misconfiguration resulting in OutOfMemoryExceptions? Is this a common symptom of our misconfiguration or is there something else deeper going on?

There is an Umbraco forum post for the same issue here:
https://forum.umbraco.com/t/misconfigured-examine-settings-lead-to-outofmemoryexceptions-on-azure-webapp/409

Specifics

The site is Umbraco 13.5.2
The version of Umbraco.Cms.Examine.Lucene is 13.6.0

When we look at the Umbraco logs of the public site after we get it stable again, we see cycles of:

1. The site would restart mid day. We aren’t sure what triggered this yet
2. The site would go through its regular boot process
3. When the site would boot back up, it would replicate the indexes from the %Home% directory back to its %TEMP% directory with no apparent errors. There are info level logs indicating this.
4. Around 1min later, the site would throw an OutOfMemoryException and restart
5. The cycle would repeat

The Umbraco log entries would see the Umbraco site booting up followed by the following over and over:

{"@t":"2025-02-27T21:22:30.4225110Z","@mt":"Checked main index {IndexName} and it is clean.","@tr":"641cad4a27414be3553211e27ec644f7","@sp":"6465ee2f812db791","IndexName":"ExternalIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ActionId":"6ab4ac69-f568-4114-9da2-c0f2406daa04","ActionName":"Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)","RequestId":"40004806-0000-7d00-b63f-84710c7967bb","RequestPath":"/search-results/","ProcessId":6720,"ProcessName":"w3wp","ThreadId":55,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO ","HttpRequestId":"4b9759ec-7948-4c84-9874-28c380b5fe4f","HttpRequestNumber":2,"HttpSessionId":"764367b2-df6a-857a-97e9-e46e05e2e38a"}
{"@t":"2025-02-27T21:22:30.4902234Z","@mt":"Replicating index from {SourceIndex} to {DestinationIndex}","@tr":"641cad4a27414be3553211e27ec644f7","@sp":"6465ee2f812db791","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\ExternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\ExternalIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\ExternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\ExternalIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ActionId":"6ab4ac69-f568-4114-9da2-c0f2406daa04","ActionName":"Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)","RequestId":"40004806-0000-7d00-b63f-84710c7967bb","RequestPath":"/search-results/","ProcessId":6720,"ProcessName":"w3wp","ThreadId":55,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO ","HttpRequestId":"4b9759ec-7948-4c84-9874-28c380b5fe4f","HttpRequestNumber":2,"HttpSessionId":"764367b2-df6a-857a-97e9-e46e05e2e38a"}
{"@t":"2025-02-27T21:22:30.6710440Z","@mt":"Replication from index {SourceIndex} to {DestinationIndex} complete.","@tr":"641cad4a27414be3553211e27ec644f7","@sp":"6465ee2f812db791","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\ExternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\ExternalIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\ExternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\ExternalIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ActionId":"6ab4ac69-f568-4114-9da2-c0f2406daa04","ActionName":"Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)","RequestId":"40004806-0000-7d00-b63f-84710c7967bb","RequestPath":"/search-results/","ProcessId":6720,"ProcessName":"w3wp","ThreadId":55,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO ","HttpRequestId":"4b9759ec-7948-4c84-9874-28c380b5fe4f","HttpRequestNumber":2,"HttpSessionId":"764367b2-df6a-857a-97e9-e46e05e2e38a"}
{"@t":"2025-02-27T21:22:52.9499328Z","@mt":"Checked main index {IndexName} and it is clean.","IndexName":"MembersIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:52.9578952Z","@mt":"Replicating index from {SourceIndex} to {DestinationIndex}","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\MembersIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\MembersIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\MembersIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\MembersIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:52.9657925Z","@mt":"Replication from index {SourceIndex} to {DestinationIndex} complete.","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\MembersIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\MembersIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\MembersIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\MembersIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:53.4554326Z","@mt":"Checked main index {IndexName} and it is clean.","IndexName":"UmbracoFormsRecordsIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:53.4674661Z","@mt":"Replicating index from {SourceIndex} to {DestinationIndex}","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\UmbracoFormsRecordsIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\UmbracoFormsRecordsIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\UmbracoFormsRecordsIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\UmbracoFormsRecordsIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:53.5292112Z","@mt":"Replication from index {SourceIndex} to {DestinationIndex} complete.","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\UmbracoFormsRecordsIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\UmbracoFormsRecordsIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\UmbracoFormsRecordsIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\UmbracoFormsRecordsIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:54.4668320Z","@mt":"Checked main index {IndexName} and it is clean.","IndexName":"InternalIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:54.4826063Z","@mt":"Replicating index from {SourceIndex} to {DestinationIndex}","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\InternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\InternalIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\InternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\InternalIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:54.6406734Z","@mt":"Replication from index {SourceIndex} to {DestinationIndex} complete.","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\InternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\InternalIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\InternalIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\InternalIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:54.7436755Z","@mt":"Checked main index {IndexName} and it is clean.","IndexName":"DeliveryApiContentIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:54.7506367Z","@mt":"Replicating index from {SourceIndex} to {DestinationIndex}","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\DeliveryApiContentIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\DeliveryApiContentIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\DeliveryApiContentIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\DeliveryApiContentIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:54.7587255Z","@mt":"Replication from index {SourceIndex} to {DestinationIndex} complete.","SourceIndex":"SimpleFSDirectory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\DeliveryApiContentIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\DeliveryApiContentIndex","DestinationIndex":"SimpleFSDirectory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\DeliveryApiContentIndex lockFactory=NoPrefixSimpleFsLockFactory@C:\\local\\Temp\\ExamineIndexes\\4621a58203b7266528d6eeb8fc3db8b1\\DeliveryApiContentIndex","SourceContext":"Examine.Lucene.ExamineReplicator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:22:54.8678094Z","@mt":"The Delivery API is not enabled, no indexing will performed for the Delivery API content index.","SourceContext":"Umbraco.Cms.Infrastructure.Examine.DeliveryApiContentIndexPopulator","ProcessId":6720,"ProcessName":"w3wp","ThreadId":8,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"INFO "}
{"@t":"2025-02-27T21:23:54.8090549Z","@mt":"Unhandled exception in AppDomain (terminating).","@l":"Error","@x":"System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.\r\n   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, Char* pThreadName)\r\n   at System.Threading.Thread.StartCore()\r\n   at System.Threading.PortableThreadPool.WorkerThread.CreateWorkerThread()\r\n   at System.Threading.PortableThreadPool.WorkerThread.MaybeAddWorkingWorker(PortableThreadPool threadPoolInstance)\r\n   at System.Threading.PortableThreadPool.GateThread.GateThreadStart()\r\n   at System.Threading.Thread.StartCallback()","SourceContext":"Umbraco.Cms.Infrastructure.Runtime.CoreRuntime","ProcessId":6720,"ProcessName":"w3wp","ThreadId":7,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1sdwk00077A","Log4NetLevel":"ERROR"}

Steps to reproduce

This issue is difficult to reproduce at will. It is something we are only seeing in Azure. It appears to be some sort of race condition.

Expected result / actual result

If the site does need to crash, it should not crash endlessly. The site should come back up.

Really, our goal is to understand the connection (if there is one) between the misconfiguration of Examine for load balancing in Azure and the OutOfMemoryExceptions and make sure that doesn't happen anymore.

[github-actions]

Hi there @bowserm!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂