index

Windows Search Index monitoring

Windows Search Index monitoring

While rolling out Windows Search Indexing I bumped in to quite a few issues and things I wanted to monitor. One of the main information I actually was after is the amount of files in the Windows Search Index. I could see this information in the Indexing Options of the system and it was constantly updating, but I was not able to find the same information per WMI, PowerShell cmdlets, Performance Counters etc.

This caused me headache cause it was clear that the index a) had to grow till it reached a fully indexed system and b) if the index-size dropped below a certain amount (once finished) I surely had an issue – I saw the database rebuilding out of no where – due to pagefile-issues or space on the partition where the index-database resided.

All of this made it clear that monitoring was inherent – but I did not want to play around with EventIDs – clearly the amount of files in the index was a way better indicator.

This caused me to write the following PowerShell script – it will invoke a command to a target system and count the files currently in the index. This has to be invoked, though the OLEDB provider of the Search Index allows remote-requests, it does not give you accurate numbers for the overall scope of the index on the system using remote requests. Getting to a point to even get this information was quite a challenge, I am certain who ever finds this here will know already, assuming you did some research. Hope it helps, though.

An update to this or what I learned

Of course you wouldn’t just monitor the amount of indexed files with the script above. You likely have a dedicated drive / partition where the index resides. You definitive want to monitor the used or free drive space there as well. What you will discover especially in the beginning but as well later if many files are moved or copied to the server, is that the Windows Search Index database will grow after the indexing of huge amounts of files is done and shrink again. As far as I understand this, there is some maintenance and deduplication going on.

On a server with about 14 million files that took about 1 week to index, I ended up with a 250 GB index database, the second it finished it took another 12 hours while the database grew another 100 GB and then shrunk back to actually 200 GB.

Don’t let those numbers scare you, we talk about 10 TB (terra byte) of data in those 14 million files. This is quite a bit. Most other file servers won’t have such huge databases and you won’t see such huge increases while the index database is doing some kind of clean up, nor will it take a week for the initial index to finish.

What I wanted to show with this is simply that you really want to monitor all those information and keep a close eye on it. I saw the database make huge jumps in size in very very short periods of time. If the drive then is full your index possibly would get corrupted and Windows will start from zero again. You want to avoid this. Once the index is finished and no huge incoming file operations take place you won’t see to many jumps anymore. It will calm down. But still, always make sure you have enough space on the partition where the index database resides and proper monitoring on it to be able to react quickly (going as far as automating a service stop for the Windows Search index while the space goes down to avoid database corruption).

Setting up Windows Search Index

Setting up Windows Search Index

The Windows Search indexing is a solution from Microsoft that will index your file servers and their files full text and allow your end users to get results quickly while actually engaging the fulltext search database seamlessly.

This is accomplish by just using the search box in the upper right of the Windows Explorer while residing on a network share or mapped network drive.

Many people say that the Windows Search does not work right, but my experience is that quite the opposite is true and it only depends on the right set up. This I will explain here further.

Before we go in to details – there is a challenge that goes along with DFS namespaces. I found a way to bypass this, look here to understand how it works.

  • add an additional drive to your file servers
    • I roughly recommend about 10% of the total use size of your file shares that you want to index
    • you might be able to go with less or more – but expect 10% to be more on the safe side
  • give the new drive a drive-letter (e.g. I: for Index) and name it INDEX (to make clear that this is only to be used for the INDEX database)
  • add the following path to this drive
    • I:\ProgramData\Microsoft
    • The path above mirrors the default path on the C: drive where the index will reside by default, I recommend to mirror this path on the new target drive to keep things simple and clear
  • add the feature Windows Search to your server
  • download the Microsoft Filter Pack 2.0 and install it on your server
    • yes – this is a Office 2010 SP2 filter pack and yes that’s okay
    • just download the x64 bit version of it and install it
    • this will add a iFilter pack to Windows that allows Windows to understand DOCX and XLSX files (and others) and full text index them
  • enable the new service Windows Search
    • set it to start type automatic
    • start the service
  • from your start menu (or Control Panel) open the Indexing Options
    • click on ADVANCED
      • select the earlier created path I:\ProgramData\Microsoft as new path for the index
      • after you click on OK Windows will stop the service and move the existing index
        • you might need to manually start the service again
    • click now on MODIFY and select the server shares respectively local paths you want to index
    • if you leave the Indexing Options open you will see that Windows updates the count constantly
      • Windows will slow down indexing if you work on the console or via RDP on the server Desktop
      • You will see the message INDEXING COMPLETED once all files have been index
        • this can take many days – don’t be surprised and be patient
  • think about monitoring the index and the partition the index resides on

One challenge resides – if your indexing drive becomes full, Windows Search indexing will crash the index database and likely determine it as corrupt and start from scratch. This can happen within minutes, even before any of your monitoring solutions might warn you about the full drive. There are eventlog entries about and you would easily see that the number of indexed files dropped to a very low number again. The drive becomes suddenly pretty empty as well again. This is sure one of the downsides of how Microsoft implemented this, but if you provide plenty of space in the first place, you likely won’t experience this issue.

iFilters for PDF files seem not to be necessary on current Windows versions. In the past I downloaded a iFilter for PDF files from Adobe but I experienced many issues with temporary files on the C:\ of my file servers. This is a known bug with the Adobe iFilter – as for Windows 2016 those files are full text index, but the same restrictions as in the past apply – PDF file can hold editable text that can be indexed or they hold pictures – what is often due to scans of documents – as long your scanner didn’t due OCR and translate the image to text, those files can’t be full text indexed.

Monitoring

It is almost essential to have proper monitoring in place for this. Actually, you want to monitor everything within your network, but that’s another story. I faced the same challenges and even was able to find out more about how the index database grows and why many people aren’t happy with it. Eventually it comes down to improper configurations and bad or nor monitoring at all. I highly recommend you have a look at this additional blog article about Windows Index Monitoring.