System Update – 5/23/19

Airings have been delayed today due to a bad query which has slowed down the ETL.  It’s an unfortunate case of human error and we apologize for it.  We’re looking into a solution that may help to prevent future similar occurrences.  The ETL should be caught up in the next few hours and then all airings for 5/22 & 5/23 will be up to date in the system.

CW Network came back online on 5/20.  Now we’re having an issue with Comet and Charge.  Monitoring the broadcast diginets over the air comes with its fair share of complications.  In this case, a broadcast tower was damaged due to weather which took one of our two dual broadcast feeds offline.

We will update when Comet and Charge are back online and apologize for the inconvenience.

 

System Update – 5/16/19

Our ETL (Extract, transform, Load) routine got stuck yesterday at some odd hour and wasn’t discovered until this morning.  We’re researching why the system didn’t send a warning earlier but one of our nice clients pointed out that no airings for 5/16 were being shown in AdSphere.  Rest assured, no airings have been lost and the ETL is just catching up over the next few hours and everything should return to normal.

There was an issue with the Network Log View that has been fixed.

We’ve lost one of our two CW network broadcast feeds so this network is currently offline.  We’re working to acquire another feed so we can put the CW back online.

We apologize for these issues and are working to resolve.

System Update – 4/26/19

Some Fridays start out better than others….

Around 8:40AM PDT we experienced a power failure at our co-location facility that took DRMetrix offline.  A team of people have been working on this issue ever since.

10:00AM PDT – Power has been restored.   The underlying cause is still being researched.  We have redundancy on power so in theory this should not have happened.  We are waiting to understand if the power issue was internal or external to our systems.

10:50AM PDT – All of our servers are being brought back online one by one.   We put Adsphere into maintenance mode because the database servers are still in recovery mode.  As soon as the database servers are happy again, we’ll bring AdSphere out of maintenance mode.

11:23AM PDT – We are still struggling to get the database servers back to normal.  Queries are running way too slow for production forcing us to leave AdSphere in Maintenance mode for the time being.

12:10PM PDT – AdSphere has been taken out of maintenance mode and the database server is performing better.

We are still dealing with numerous issues.  Network capture servers are still having issues and it will take some time to get them all back online and working normally.  Unfortunately, this means we’ll be dealing with some data loss (inability to capture airings data while systems are down).  Even when our systems are fully restored, we will be unable to report airings data during the period of time when our systems were down.  Also, break type detection will be impacted.  Our hope is that we can get all of our system back online over the weekend, if not sooner.  Please know that we will be doing additional system maintenance over the weekend and Adsphere will be offline at times.

Since we began operation in 2014, DRMetrix has worked with an exceptional group of technical professionals as valued partners and vendors.  It’s heartbreaking for our team when something like this happens and we feel terrible for inconveniencing our valued clients.   DRMetrix and its technology partners will endeavor to determine the root cause of what occurred and what measures can be taken to prevent future occurrences.  We appreciate everyone’s patience and sincerely apologize for this situation.

Updates:

DRMetrix operations were largely back online by Friday afternoon.  We are still experiencing some issues with our registration system.  Registration of new ads since Friday AM will be delayed as our team catches up over the next few days.

===================

It’s been a little over two weeks.  Our co-location provider determined that there was a brief power fluctuation that took place across their electrical infrastructure on 4/26 that impacted multiple DRMetrix circuits.  The source of this was external to DRMetrix’s hardware and systems.  Much work has been done in an effort to trace down numerous issues caused by the power dip and to return all of our systems to normal operation.  Our equipment has been moved to a different set of electrical circuits and additional redundant power equipment has been installed.  Many hours have been spent troubleshooting and resolving issues caused by the power dip.  One remaining challenge which has dragged on for the past two weeks was a mysterious I/O issue that was causing random issues with video capture and playback quality for brief periods across random networks.   The issue would come and go at random times and affect different networks.  We estimate that airing detection accuracy levels have been reduced by a few percentage points during this period of  time.  We believe we recently identified the culprit and, since taking corrective action, no further I/O issues have been detected.  We hope that all of the issues resulting from the power failure of 4/26/19 have now been resolved.

Thank you to our customers for all of your support and understanding.

System Update – 11/30/18

12:08PM EST

We have been alerted to issues occuring in AdSphere where reports are not running properly and error messages are being thrown.  We have alerted our technical teams and they are in the process of troubleshooting.  It may be an issue with the database server and hopefully it can be resolved quickly.

More details to follow.  We apologize for the inconvenience.

1:00PM EST

System are back to normal after reboot of SQL and production servers.  Underlying causes for initial problem are still being investigated.

Get System Status Updates via SMS!

Be the first to know!

Sign up to receive system status updates via SMS on the new AdSphere™ User Group on Facebook!

You can request to join at:  https://www.facebook.com/groups/adsphere   Once you’re a member, you can read our recent notification regarding our new SMS notification service.  Be sure a leave a comment letting us know you’d like to be added to our SMS distribution list.

 

 

Friday – August 24th Update

We are experiencing a signal outage that began on 8/24/18 at 4:41pm EDT.  It appears to be affecting A&E Networks including Lifetime Movie, Lifetime, A&E, History, FYI,  and Viceland.  We are waiting on signal restoration.

Status Update:  All signals restored.

Start date and time, 8/24/18 4:41pm EDT
End date and time, 8/24/18 5:45pm EDT

We will not have airings during these times for the aforementioned networks.   We apologize for any inconvenience.

Wednesday – August 8th Update

Wow, we are having a slew of bad luck with our diginet feeds this month.  Unrelated to the problems that affected some of our diginet feeds earlier in the week, yesterday we lost one of our Charge! feeds due equipment failure issues at the broadcaster.  We are hoping to have the signal back in the next day or two as the broadcaster fixes their issues.  We apologize for the inconvenience.  More details to follow.

August 16th update – We finally got the Charge! feed issue resolved shortly after midnight this morning.  One of the two broadcast feeds we require went offline due to a lightning strike on the broadcast tower on 8/7.  It took them until today to get their broadcast tower back online.   We’re hoping no more acts of God for awhile and we thank everyone for your patience.

Tuesday – August 7th Update

From 1AM – 5AM this morning, our provider had a technical issue with one of their routers causing us to lose feeds on the following networks:

For AdSphere:  TBD, CREATE, ION, CHARGE, ION LIFE, QUBO, HEROS, CW.

For AVS: The above networks in addition to ABC, NBC, CBS, and FOX feeds.

We apologize for our inability to report airings on these networks during this time frame.