Event Log Warning Event ID 8088 - Taxonomy

I experienced a lone Warning event in the Application Event log with the ID 8088 after server restart. Text of the event was 'The Managed Metadata Service 'Managed Metadata Service Application' is inaccessible.'.

I dug deeper in ULS and found this log entry:

Failed to get term store for proxy 'Managed Metadata Service Application'. Exception: System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:09.9375192. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://sp2:32843/ee1f85842acc4eb1b543bb2b1395980c/MetadataWebService.svc' has exceeded the allotted timeout of 00:00:09.9990000. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out   
 at System.Net.HttpWebRequest.GetResponse()   
 at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     -
 -- End of inner exception stack trace ---   
 at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)   
 at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)   
 at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     -
 -- End of inner exception stack trace ---    Server stack trace:   
 at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)   
 at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)   
 at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)   
 at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)   
 at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)   
 at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown
 at [0]:   
 at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)   
 at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)   
 at Microsoft.SharePoint.Taxonomy.Internal.IDataAccessReadOnly.GetSessionData(Guid& termStoreId, Guid rawPartitionId, Int32 lcid, String systemGroupName, String systemGroupDescription, String keywordsTermsetName, String keywordsTermsetDescription, String orphanedTermsTermsetName, String orphaneedTermsTermsetDescription, String hashtagsTermsetName, String hashtagsTermsetDescription)   
 at Microsoft.SharePoint.Taxonomy.Internal.TaxonomyProxyAccess.<>c__DisplayClass4.<GetSessionData>b__3(IMetadataWebServiceApplication serviceApplication)   
 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.<>c__DisplayClass38.<RunOnChannel>b__36()   
 at Microsoft.Office.Server.Security.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)   
 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.RunOnChannel(CodeToRun codeToRun, Double operationTimeoutFactor)   
 at Microsoft.SharePoint.Taxonomy.Internal.TaxonomyProxyAccess.GetSessionData(Guid& termStoreId, Guid rawPartitionId, Int32 lcid, String systemGroupName, String systemGroupDescription, String keywordsTermsetName, String keywordsTermsetDescription, String orphanedTermsTermsetName, String orphanedTermsTermsetDescription, String hashtagsTermsetName, String hashtagsTermsetDescription)   
 at Microsoft.SharePoint.Taxonomy.Internal.DataAccessManager.GetSessionData(Guid& termStoreId)    

Source of the log entry was Timer Job Query Classification Dictionary Update for Search Application.

So a web service call to Managed Metadata web service timed out after roughly 10 seconds. After a bit of digging I found out that the timer job uses C:\Program Files\Microsoft Office Servers\15.0\WebClients\Metadata\client.config for the client endpoint configuration. There I saw that both http and https bindings were set up to have timeouts set to 30 seconds. This puzzled me so I reflected the code in question. Turns out that timer job is using operationTimeoutFactor which was set to 0.333333 in my case. It multiplies the factor with timeout setting in the configuration file to get the real timeout. That's why web service call timed out after 30 * 0.333333 = 9.999 seconds.

I just increased all of the timeouts to 1:30 minutes and restarted to SharePoint Timer service. Warnings in the Event Log didn't appear any more.

C:\Program Files\Microsoft Office Servers\15.0\WebClients\Metadata\client.config after modification:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>
    <client>
      <endpoint
        name="http"
        contract="Microsoft.SharePoint.Taxonomy.IMetadataWebServiceApplication"
        binding="customBinding"
        bindingConfiguration="MetadataWebServiceHttpBinding" />
      <endpoint
        name="https"
        contract="Microsoft.SharePoint.Taxonomy.IMetadataWebServiceApplication"
        binding="customBinding"
        bindingConfiguration="MetadataWebServiceHttpsBinding" />
    </client>
    <bindings>
      <customBinding>
        <binding
          name="MetadataWebServiceHttpBinding"
                 receiveTimeout="00:01:30"
                 sendTimeout="00:01:30"
                 openTimeout="00:01:30"
                 closeTimeout="00:01:30">
          <security
            authenticationMode="IssuedTokenOverTransport"
            allowInsecureTransport="true" />
          <binaryMessageEncoding>
            <readerQuotas
               maxStringContentLength="2147483647"
               maxArrayLength="2147483647"
               maxBytesPerRead="2147483647" />
          </binaryMessageEncoding>
          <httpTransport
            transferMode="StreamedResponse"
            maxReceivedMessageSize="2147483647"
            authenticationScheme="Anonymous"
            useDefaultWebProxy="false" />
        </binding>
        <binding
          name="MetadataWebServiceHttpsBinding"
                 receiveTimeout="00:01:30"
                 sendTimeout="00:01:30"
                 openTimeout="00:01:30"
                 closeTimeout="00:01:30">
          <security
            authenticationMode="IssuedTokenOverTransport" />
          <binaryMessageEncoding>
            <readerQuotas
              maxStringContentLength="2147483647"
              maxArrayLength="2147483647"
              maxBytesPerRead="2147483647" />
          </binaryMessageEncoding>
          <httpsTransport
            transferMode="StreamedResponse"
            maxReceivedMessageSize="2147483647"
            authenticationScheme="Anonymous"
            useDefaultWebProxy="false" />
        </binding>
      </customBinding>
    </bindings>
  </system.serviceModel>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="10000" />
    </connectionManagement>
  </system.net>
</configuration>