We still have a group of about 30 networks that are not reporting airings since around 4AM on 9/29.  We are being told by our ACR provider that airings data during this time is not lost but the system is still playing catch up.  Apparently there are a lot of thumbnail images and airing level videos to process on the impacted servers.  When we brought them back online, some are having to run file disk check operations to check for, and resolve any possible issues of, data corruption.   While this type of recovery mode is typical, because DRMetrix has millions of airing level thumbnail image files on various servers, a very large number of files are having to be checked which is delaying completion of recovery mode on some servers.  When these servers finish recovery mode, they will begin processing the back airings data and things will return to normal.  We expected this process would have been completed on all servers by now but some servers that are impacting these 30 networks are still in recovery mode and/or currently back logged processing data and trying to catch back up to processing live airings data.

We are in conversations with our co-location vendor about exactly what happened at their facility that caused these issues.  Apparently many companies they service have been negatively impacted.  While we are constantly working to improve our systems, and to achieve a high level of uptime service, when something out of our direct control like this happens, it is extremely discouraging.  We thank all of our customers for your continued patience, support, and understanding.  We deeply regret and apologize for this situation and will do our very best to work with our co-location vendor to find solutions to keep this type of situation from occurring again.

Update 9:30PM ET – Progress!  More of the servers came out of recovery mode today and more data is being reported as the system continues to play catch up.  Some networks that had no data for today or yesterday are now caught up through yesterday and are processing today’s airings.  Some are a bit further behind but quickly catching up.   We’re down to about 10 networks that are still in recovery mode which is a big improvement from yesterday.  More details to follow tomorrow.  We appreciate everyone being so understanding and patient.  Thank you!!