Failure retrieving Diagnostics Logs location

I've been getting these Unexpected ULS entries from Timer Job hourly-all-sptimerservice-health-analysis-job:

Failure retrieving Diagnostics Logs location:
System.ArgumentException: Object must be a root directory ("C:\") or a drive letter ("C").
at System.IO.DriveInfo..ctor(String driveName)
at Microsoft.SharePoint.Administration.Health.AppServerDrivesAreNearlyFull. GetDrivesForValidation()

Turns out I had to change Central Administration > Monitoring > Configure diagnostic logging > Path from %CommonProgramFiles%\Microsoft Shared\Web Server Extensions\15\LOGS to C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\LOGS.

Error is caused by Health Analysis Job (Hourly, Microsoft SharePoint Foundation Timer, All Servers) timer job. Specifically DriveInfo constructor cannot resolve environment variables, so log location has to be a real path.

Alternative Powershell script solution:
$diagSvc = [Microsoft.SharePoint.Administration.SPDiagnosticsService]::Local
$commonProgramFiles = Get-ChildItem -Path Env:\ | where Name -EQ "CommonProgramFiles"
$newLogLocation = $diagSvc.LogLocation.Replace("%CommonProgramFiles%", $commonProgramFiles[0].Value)
$diagSvc.LogLocation = $newLogLocation

$diagSvc.Update()

Event Log Warning Event ID 8088 - Taxonomy

I experienced a lone Warning event in the Application Event log with the ID 8088 after server restart. Text of the event was 'The Managed Metadata Service 'Managed Metadata Service Application' is inaccessible.'.

I dug deeper in ULS and found this log entry:

Failed to get term store for proxy 'Managed Metadata Service Application'. Exception: System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:09.9375192. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://sp2:32843/ee1f85842acc4eb1b543bb2b1395980c/MetadataWebService.svc' has exceeded the allotted timeout of 00:00:09.9990000. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out   
 at System.Net.HttpWebRequest.GetResponse()   
 at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     -
 -- End of inner exception stack trace ---   
 at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)   
 at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)   
 at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     -
 -- End of inner exception stack trace ---    Server stack trace:   
 at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)   
 at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)   
 at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)   
 at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)   
 at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)   
 at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown
 at [0]:   
 at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)   
 at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)   
 at Microsoft.SharePoint.Taxonomy.Internal.IDataAccessReadOnly.GetSessionData(Guid& termStoreId, Guid rawPartitionId, Int32 lcid, String systemGroupName, String systemGroupDescription, String keywordsTermsetName, String keywordsTermsetDescription, String orphanedTermsTermsetName, String orphaneedTermsTermsetDescription, String hashtagsTermsetName, String hashtagsTermsetDescription)   
 at Microsoft.SharePoint.Taxonomy.Internal.TaxonomyProxyAccess.<>c__DisplayClass4.<GetSessionData>b__3(IMetadataWebServiceApplication serviceApplication)   
 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.<>c__DisplayClass38.<RunOnChannel>b__36()   
 at Microsoft.Office.Server.Security.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)   
 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.RunOnChannel(CodeToRun codeToRun, Double operationTimeoutFactor)   
 at Microsoft.SharePoint.Taxonomy.Internal.TaxonomyProxyAccess.GetSessionData(Guid& termStoreId, Guid rawPartitionId, Int32 lcid, String systemGroupName, String systemGroupDescription, String keywordsTermsetName, String keywordsTermsetDescription, String orphanedTermsTermsetName, String orphanedTermsTermsetDescription, String hashtagsTermsetName, String hashtagsTermsetDescription)   
 at Microsoft.SharePoint.Taxonomy.Internal.DataAccessManager.GetSessionData(Guid& termStoreId)    

Source of the log entry was Timer Job Query Classification Dictionary Update for Search Application.

So a web service call to Managed Metadata web service timed out after roughly 10 seconds. After a bit of digging I found out that the timer job uses C:\Program Files\Microsoft Office Servers\15.0\WebClients\Metadata\client.config for the client endpoint configuration. There I saw that both http and https bindings were set up to have timeouts set to 30 seconds. This puzzled me so I reflected the code in question. Turns out that timer job is using operationTimeoutFactor which was set to 0.333333 in my case. It multiplies the factor with timeout setting in the configuration file to get the real timeout. That's why web service call timed out after 30 * 0.333333 = 9.999 seconds.

I just increased all of the timeouts to 1:30 minutes and restarted to SharePoint Timer service. Warnings in the Event Log didn't appear any more.

C:\Program Files\Microsoft Office Servers\15.0\WebClients\Metadata\client.config after modification:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>
    <client>
      <endpoint
        name="http"
        contract="Microsoft.SharePoint.Taxonomy.IMetadataWebServiceApplication"
        binding="customBinding"
        bindingConfiguration="MetadataWebServiceHttpBinding" />
      <endpoint
        name="https"
        contract="Microsoft.SharePoint.Taxonomy.IMetadataWebServiceApplication"
        binding="customBinding"
        bindingConfiguration="MetadataWebServiceHttpsBinding" />
    </client>
    <bindings>
      <customBinding>
        <binding
          name="MetadataWebServiceHttpBinding"
                 receiveTimeout="00:01:30"
                 sendTimeout="00:01:30"
                 openTimeout="00:01:30"
                 closeTimeout="00:01:30">
          <security
            authenticationMode="IssuedTokenOverTransport"
            allowInsecureTransport="true" />
          <binaryMessageEncoding>
            <readerQuotas
               maxStringContentLength="2147483647"
               maxArrayLength="2147483647"
               maxBytesPerRead="2147483647" />
          </binaryMessageEncoding>
          <httpTransport
            transferMode="StreamedResponse"
            maxReceivedMessageSize="2147483647"
            authenticationScheme="Anonymous"
            useDefaultWebProxy="false" />
        </binding>
        <binding
          name="MetadataWebServiceHttpsBinding"
                 receiveTimeout="00:01:30"
                 sendTimeout="00:01:30"
                 openTimeout="00:01:30"
                 closeTimeout="00:01:30">
          <security
            authenticationMode="IssuedTokenOverTransport" />
          <binaryMessageEncoding>
            <readerQuotas
              maxStringContentLength="2147483647"
              maxArrayLength="2147483647"
              maxBytesPerRead="2147483647" />
          </binaryMessageEncoding>
          <httpsTransport
            transferMode="StreamedResponse"
            maxReceivedMessageSize="2147483647"
            authenticationScheme="Anonymous"
            useDefaultWebProxy="false" />
        </binding>
      </customBinding>
    </bindings>
  </system.serviceModel>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="10000" />
    </connectionManagement>
  </system.net>
</configuration>

Failed to load receiver assembly

This is a very common issue and it manifests in Visual Studio during deploy. For me, remedy was this:

1. Delete obj and bin folders of the solution.
2. Restart Visual Studio.
3. Set Always Force Install to true on all features.
4. Deploy.
5. Optionally, after successful deployment revert Always Force Install to false on all features.

More on the subject: https://sharepoint.stackexchange.com/questions/49612/failed-to-load-receiver-assembly-when-activate-feature

Getting Rid of http://server/Analytics_{GUID}/ Requests

When I'm developing software I often have Fiddler turned on to capture client HTTP traffic. I've noticed weird http://server/Analytics_{GUID}/ Requests appearing in a batch of 4 requests every minute:

Investigation led me to Application Server Administration Service Timer Job which in turn called Search Service Instance's Synchronize method. This method calls code which uses rather odd way of determining whether file share exists:
    private static bool ShareExists(string path)
    {
      try
      {
        Directory.GetDirectories(path);
        return true;
      }
      catch (UnauthorizedAccessException ex)
      {
        return true;
      }
      catch (Exception ex)
      {
        return false;
      }
    }

Line Directory.GetDirectories(path); causes those 4 requests. This execution path happens when you don't have Search Service Application provisioned but you do have SharePoint Server Search service started.

Anyway, I stopped the service using cmdlet:
Get-SPEnterpriseSearchServiceInstance -Local | Stop-SPEnterpriseSearchServiceInstance

And the problem was solved, no more pesky Analytics requests.

Setting Related Item on a SharePoint 2013 Worfklow's SingleTask

Sometimes when we design workflows it is necessary to set related item in SingleTask to something different than the default listitem. Default list item is the item on which workflow was started. So, lets say we start workflow on doc1.docx. Inside the workflow we define SingleTask. This SingleTask, when opened in SharePoint UI, will have, by default, a link to doc1.docx. Sometimes we would like this SingleTask to have related item which is different than that particular doc1.docx.

Microsoft has intended for this purpose SingleTask argument named RelatedContentLinkListItemIntegerId. Great, you just set the list ID and item ID and you have your problem solved. Wrong!! Setting this property will have no impact on related item. At least, not if you use default generated SingleTask activity.

The solution:
Setting RelatedContentLinkListItemIntegerId will only work if you delete a particular child element in workflow markup (XAML). So, first open your workflow in Code mode (Visual Studio View Code - F7). Then locate the SingleTask activity which you would like to configure. This should be p:SingleTask XML element. Delete p:SingleTask.RelatedContentLinkListItemId child elemet. Save your workflow and voila, it will change related item to whatever value you set it.

SharePoint 2016 Application Pools and Other Processes Stop Immediately After Starting

At one point of developing SharePoint 2016 Farm solution on my dev machine SharePoint application pools, namely web application pool and CA application pool, started to crash on start. I restarted the server and then all of a sudden no SharePoint services (SharePoint Timer Service, SharePoint Tracing Service) could be started.

All google results pointed to faulty application pool account. I double checked that account was not locked and had appropriate perms. No luck there.

System Event log was full of events 5002, 5009 and 5011.

After inspecting a situation with Process Monitor we pin pointed an error to missing HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\16.0\Location value in the registry. After adding C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ data to Location value all services went back online after server restart.

Only change in server configuration that day was installment of Microsoft Office Professional Plus 2016. I don't have time to investigate what caused deletion of aforementioned value in registry but I am eager to find out.

Hopefully, I will edit this post if I find out more details.

OpenQuery Failed with status ID: 0x800007d0

ULS on my clients' farm was polluted by messages "OpenQuery Failed with status ID: 0x800007d0" every few seconds.

Turns out I had to add user which is running mssearch.exe process (SP_Search in my case) to local group called Performance Monitor Users. After adding the user I rebooted the machine and the problem went away.