DRMetrix had a network capture server fail from 12/22/20 at 18:00 ET to 12/23/20 at 18:00 ET impacting 7 networks. Airing detections on the following networks were impacted: Viceland, History Channel, Science, USA Network, VH-1, WGN Superstation, and Women’s Entertainment. Systems that are in place to monitor and alert our team to such failures did not work as expected in this instance. We have investigated the cause and are making changes to these systems to help mitigate future occurrences. We sincerely apologize for any inconvenience caused.

On 12/10 at 8am EST there was an outage impacting some Viacom networks caused by a satellite receiver issue. The outage was resolved by 9:30am EST. DRMetrix’s break type (DPI) signaling was also impacted and not fully restored until 9AM EST today. This will impact DPI break type flagging for this period of time on the following networks: BET, MTV2, Nickelodeon, TVLand, and Nick-At-Night. We apologize for any inconvenience caused.

We updated the media rates for Q3 from Standard Media Index and the system fell behind processing all of the rate updates. Airings are delayed getting into AdSphere and AVS by about 9 hours as of 3pm ET today. We expect the system to catch up by the end of today. We apologize for any inconvenience.

On Saturday (10/31/20) at approximately 9AM ET, our signal for Family Entertainment Television (FETV) went offline due to a technical issue. The signal was restored at 9AM ET today. We apologize for any inconvenience.

Over the past week, a couple of issues were brought to DRMetrix’s attention by a couple of our clients, which we greatly appreciate! Based on their feedback, we were able to isolate and correct an error that was made back in July that caused DRMetrix to misreport some airings in national ad breaks as running in DPI ad breaks. These errors going back to Q3 have been corrected in the AdSphere™ database. We were also able to correct some Q3 rate anomalies on ABC, Golf, and ESPN Deportes where the valuation of certain dayparts was too high. We have put some additional safeguards in place to prevent this from happening in the future. We apologize for any inconvenience and we appreciate the helpful feedback from those clients who brought these issues to our attention.

Our detections for Qubo, ION, and ION life were offline today between 10a to 5p EST due to more broadcast tower work being performed by one of our monitored broadcast affiliates.  We hope they have finished with their repairs and that these over-the-air broadcast digital networks will be back online and stable for the foreseeable future.  We apologize for the inconvenience.

Qubo, ION, and ION life, are offline today as one of our monitored broadcast affiliates is working on their broadcast tower today.  We apologize for any inconvenience.

5:16PM PDT Update – The above networks are back online. The broadcast affiliate in question had their tower down from 10A – 6P EDT today.

What a happy Saturday it is!  Checked the remaining 9 networks that were still behind on data processing and all of them seem to be caught up this morning.  Airing data pulls through 10/2 for media week 40 should be complete at this time.  We wish to thank you all again for your patience and understanding over the past few days.

Our Cloudflare vendor was having some odd problems which began yesterday evening.  These issues seemed largely resolved by Friday morning but they appear to have impacted our AVS API for a period of time today.  We are happy to report that the issues specific to the API have been resolved with Cloudflare.

We estimate that systems are within 24-36 hours from being caught up on processing of new detections within normal parameters.  Most networks are already back to normal.  Next week AdSphere and AVS reporting should be back to normal.  We appreciate everyone’s patience this week and wish everyone a relaxing weekend!

We still have a group of about 30 networks that are not reporting airings since around 4AM on 9/29.  We are being told by our ACR provider that airings data during this time is not lost but the system is still playing catch up.  Apparently there are a lot of thumbnail images and airing level videos to process on the impacted servers.  When we brought them back online, some are having to run file disk check operations to check for, and resolve any possible issues of, data corruption.   While this type of recovery mode is typical, because DRMetrix has millions of airing level thumbnail image files on various servers, a very large number of files are having to be checked which is delaying completion of recovery mode on some servers.  When these servers finish recovery mode, they will begin processing the back airings data and things will return to normal.  We expected this process would have been completed on all servers by now but some servers that are impacting these 30 networks are still in recovery mode and/or currently back logged processing data and trying to catch back up to processing live airings data.

We are in conversations with our co-location vendor about exactly what happened at their facility that caused these issues.  Apparently many companies they service have been negatively impacted.  While we are constantly working to improve our systems, and to achieve a high level of uptime service, when something out of our direct control like this happens, it is extremely discouraging.  We thank all of our customers for your continued patience, support, and understanding.  We deeply regret and apologize for this situation and will do our very best to work with our co-location vendor to find solutions to keep this type of situation from occurring again.

Update 9:30PM ET – Progress!  More of the servers came out of recovery mode today and more data is being reported as the system continues to play catch up.  Some networks that had no data for today or yesterday are now caught up through yesterday and are processing today’s airings.  Some are a bit further behind but quickly catching up.   We’re down to about 10 networks that are still in recovery mode which is a big improvement from yesterday.  More details to follow tomorrow.  We appreciate everyone being so understanding and patient.  Thank you!!