System Update – 7/17/19

There is an issue affecting AdSphere this morning and we understand users are receiving timeout error messages.  We are currently investigating the issue and hope to have it resolved soon.  Updates will follow.  We apologize for the inconvenience.

Update – Some runaway queries were discovered and terminated.  We are still investigating the root cause but service has been restored.

System Update – 6/25/19

This TV and Grit are back online as of this past Sunday 6/23.

A recent programming bug caused airings in local DPi breaks not to be properly flagged.  These airings are incorrectly showing as national in Adsphere.  We anticipate that this issue will be fully remedied by tomorrow, 6/26, when all impacted airings will properly reflect whether they are national or local.  We apologize for the inconvenience!  This unfortunately impacted our weekly spend index rankings which will also be updated by tomorrow

 

System Update – 6/22/19

One of our over-the-air feeds for “This TV” has been offline since yesterday and we’re also experiencing issues with one of the over-the-air signals for “Grit ” which appears to be going offline on occasion.   The broadcast digital networks can be impacted by weather or technical issues more so than the commercial grade feeds we monitor via commercial grade satellite for all of the Cable Networks.   We’ll advise as soon as we have an ETA on full signal restoration for both This TV and Grit.

System Update – 6/3/19

We apologize for the recent issues with some of the broadcast digital networks.  As of today, all of the broadcast digital networks are back online.  Weather issues had some broadcast towers impacted so in some cases we had to switch to different market network affiliates to regain service.   We thank you for your patience and apologize for the inconvenience.

System Update – 5/23/19

Airings have been delayed today due to a bad query which has slowed down the ETL.  It’s an unfortunate case of human error and we apologize for it.  We’re looking into a solution that may help to prevent future similar occurrences.  The ETL should be caught up in the next few hours and then all airings for 5/22 & 5/23 will be up to date in the system.

CW Network came back online on 5/20.  Now we’re having an issue with Comet and Charge.  Monitoring the broadcast diginets over the air comes with its fair share of complications.  In this case, a broadcast tower was damaged due to weather which took one of our two dual broadcast feeds offline.

We will update when Comet and Charge are back online and apologize for the inconvenience.

 

System Update – 5/16/19

Our ETL (Extract, transform, Load) routine got stuck yesterday at some odd hour and wasn’t discovered until this morning.  We’re researching why the system didn’t send a warning earlier but one of our nice clients pointed out that no airings for 5/16 were being shown in AdSphere.  Rest assured, no airings have been lost and the ETL is just catching up over the next few hours and everything should return to normal.

There was an issue with the Network Log View that has been fixed.

We’ve lost one of our two CW network broadcast feeds so this network is currently offline.  We’re working to acquire another feed so we can put the CW back online.

We apologize for these issues and are working to resolve.

System Update – 4/26/19

Some Fridays start out better than others….

Around 8:40AM PDT we experienced a power failure at our co-location facility that took DRMetrix offline.  A team of people have been working on this issue ever since.

10:00AM PDT – Power has been restored.   The underlying cause is still being researched.  We have redundancy on power so in theory this should not have happened.  We are waiting to understand if the power issue was internal or external to our systems.

10:50AM PDT – All of our servers are being brought back online one by one.   We put Adsphere into maintenance mode because the database servers are still in recovery mode.  As soon as the database servers are happy again, we’ll bring AdSphere out of maintenance mode.

11:23AM PDT – We are still struggling to get the database servers back to normal.  Queries are running way too slow for production forcing us to leave AdSphere in Maintenance mode for the time being.

12:10PM PDT – AdSphere has been taken out of maintenance mode and the database server is performing better.

We are still dealing with numerous issues.  Network capture servers are still having issues and it will take some time to get them all back online and working normally.  Unfortunately, this means we’ll be dealing with some data loss (inability to capture airings data while systems are down).  Even when our systems are fully restored, we will be unable to report airings data during the period of time when our systems were down.  Also, break type detection will be impacted.  Our hope is that we can get all of our system back online over the weekend, if not sooner.  Please know that we will be doing additional system maintenance over the weekend and Adsphere will be offline at times.

Since we began operation in 2014, DRMetrix has worked with an exceptional group of technical professionals as valued partners and vendors.  It’s heartbreaking for our team when something like this happens and we feel terrible for inconveniencing our valued clients.   DRMetrix and its technology partners will endeavor to determine the root cause of what occurred and what measures can be taken to prevent future occurrences.  We appreciate everyone’s patience and sincerely apologize for this situation.

Updates:

DRMetrix operations were largely back online by Friday afternoon.  We are still experiencing some issues with our registration system.  Registration of new ads since Friday AM will be delayed as our team catches up over the next few days.

===================

It’s been a little over two weeks.  Our co-location provider determined that there was a brief power fluctuation that took place across their electrical infrastructure on 4/26 that impacted multiple DRMetrix circuits.  The source of this was external to DRMetrix’s hardware and systems.  Much work has been done in an effort to trace down numerous issues caused by the power dip and to return all of our systems to normal operation.  Our equipment has been moved to a different set of electrical circuits and additional redundant power equipment has been installed.  Many hours have been spent troubleshooting and resolving issues caused by the power dip.  One remaining challenge which has dragged on for the past two weeks was a mysterious I/O issue that was causing random issues with video capture and playback quality for brief periods across random networks.   The issue would come and go at random times and affect different networks.  We estimate that airing detection accuracy levels have been reduced by a few percentage points during this period of  time.  We believe we recently identified the culprit and, since taking corrective action, no further I/O issues have been detected.  We hope that all of the issues resulting from the power failure of 4/26/19 have now been resolved.

Thank you to our customers for all of your support and understanding.

System Update – 11/30/18

12:08PM EST

We have been alerted to issues occuring in AdSphere where reports are not running properly and error messages are being thrown.  We have alerted our technical teams and they are in the process of troubleshooting.  It may be an issue with the database server and hopefully it can be resolved quickly.

More details to follow.  We apologize for the inconvenience.

1:00PM EST

System are back to normal after reboot of SQL and production servers.  Underlying causes for initial problem are still being investigated.