Sitecore: Performance issue on page load, Analytics?

 

Background of Infra:

·         The App service is hosted inside Isolated App Service Environment.

·         Sitecore XP Scaled topology used.

·         Sitecore Version: 9.1.1

 Issue & Troubleshooting

The website started experiencing slowness in Page load, sometimes it was taking more than a minute. On checking the Profiler trace it was found that the Sitecore analytics process is getting blocked while creating a tracker object.

 

StackTrace    ROOT   Sitecore.Mvc!Sitecore.Mvc.Routing.RouteHttpHandler.BeginProcessRequest    Sitecore.Mvc!Sitecore.Mvc.Pipelines.PipelineService.RunPipeline    Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run    Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run    Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run    Sitecore.Mvc.Analytics!dynamicClass.    Sitecore.Mvc.Analytics!Sitecore.Mvc.Analytics.Pipelines.MvcEvents.RequestBegin.StartTracking.Process    Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions.Monitor    Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions+<>c__DisplayClass1...    Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run    Sitecore.Analytics!dynamicClass.    Sitecore.Analytics!Sitecore.Analytics.Tracker.Initialize    Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run    Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run    Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run    Sitecore.Analytics!dynamicClass.    Sitecore.Analytics!Sitecore.Analytics.Pipelines.CreateTracker.GetTracker.Process    Sitecore.Analytics!Sitecore.Analytics.DefaultTracker.EnsureSessionContext    Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions.Monitor    Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions+<>c__DisplayClass1...    Sitecore.Analytics!Sitecore.Analytics.Pipelines.EnsureSessionContext.EnsureSessionContextPipeline+<>c__DisplayClass4_0.<Run>b__0    Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run    Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run    Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run    Sitecore.Analytics!dynamicClass.    Sitecore.Analytics!Sitecore.Analytics.Pipelines.EnsureSessionContext.LoadContact.Process    Sitecore.Analytics!Sitecore.Analytics.Tracking.ContactManager.LoadContact    Sitecore.Analytics!Sitecore.Analytics.Data.ContactRepository.LoadContact    Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions.Monitor    Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.DataAccess.XConnectDataAdapterProvider+<>c__DisplayClass15_0.<LoadContact>b__0    Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.DataAccess.XConnectDataAdapterProvider.ExecuteWithExceptionHandling    Sitecore.XConnect.Client.Configuration!Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient    Sitecore.Kernel!Sitecore.Configuration.DefaultFactory.CreateObject    Sitecore.Kernel!Sitecore.Configuration.DefaultFactory.CreateObject    Sitecore.Kernel!Sitecore.Configuration.DefaultFactory.CreateObject    Sitecore.XConnect.Client.Configuration!Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize    Sitecore.XConnect.Client!Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock    Sitecore.Xdb.Common.Web!Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock    Sitecore.Xdb.Common.Web!Sitecore.Xdb.Common.Web.Synchronous.CurrentThreadTaskScheduler.ProcessTaskQueue    OTHER <<System!System.Collections.Concurrent.BlockingCollection`1+<GetConsumingEnumerable>d__68[System.__Canon].MoveNext    BLOCKED_TIME OTHER <<clr!JITutil_MonTryEnter>>    BLOCKED_TIME LAST_BLOCK (Last blocking operation in trace)    BLOCKED_TIME

On observing the Sitecore error log:

The first error in log file says there is connectivity issue while starting tracker "because connected host has failed to respond 172.XX.XX.XXX:443"

ERROR Cannot start analytics Tracker Exception: System.Net.Http.HttpRequestException Message: An error occurred while sending the request. Source: Sitecore.Xdb.Common.Web at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory) ... at Sitecore.Analytics.Pipelines.StartTracking.StartTrackingPipeline.Run(StartTrackingArgs args) at Sitecore.Analytics.DefaultTracker.StartTracking() Nested Exception ... Nested Exception Exception: System.Net.Sockets.SocketException Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 172.XX.XX.XXX:443 Source: System at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult) at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)

Then, in order to validate the connectivity between the app services the following checks were carried out in the KUDU console

•  Name resolution from domain name to IP address of the xconnect service. The name was successfully resolving to the correct IP address .
•  TCPping and CURL to the IP address was showing that they are unable to make a connection to the xconnect service , hence a connection time out error

So, from this analysis it is evident that CD server is not able to reach the Xconnect service.

As a temporary solution we have disabled the tracker & continued the troubleshooting

On further analysis

Checked the ASE  > Network tab which showed 1 test failed on the Route table.  

On investigation, It was seen that a new Route was added to the Route Table which was having address range 172.XX.XX.XXX/32 is affecting intra subnet communications. 

The error said, "Network traffic destined for the subnet is being sent out of the subnet".

This above said 172.XX.XX.XXX IP was the Virtual IP of the app services.

Since the App services are inside the ASE (same subnet) it may NOT need explicit VNet integration to connect to each other.

 

 Resolution:

Finally found the root cause of the connectivity issue between app services.

As part Azure DevOps migration to new server a new Route to the Route table was added which was causing the traffic from inside the ASE to be routed out causing communication outage.

This route was deleted  and so the connectivity was restored & then Sitecore Analytics  was enabled back which was temporarily disabled earlier. 

Incases where the Sitecore app services are not inside the App Service environment(ASE), below diagram can be used to ensure the connectivity between xconnect and xp roles probably through vnet integration.