Systems are still in catch up mode following yesterday’s unexpected issues.  Not all networks are showing data per norm in network log view, some thumbnail images are offline, etc.  We understand that a lot of back data is still being processed as the servers are in catch up mode.  We anticipate a lot of progress will be made today and that hopefully data in AdSphere and AVS will quickly return to normal.  More details to follow.

Update 12:09AM ET – While many networks have returned to normal, along with airing level videos and thumbnail images, the system is still playing catch up on other networks that may not be showing recent airings.   We hope the system will finish processing back data and recent data on these lagging networks soon.

DRMetrix is experiencing issues this morning at our main co-location data center that are impacting our services and others.  We are working to get AdSphere back online.  Thank you for your patience this morning.  Updates will be made available as soon as we know more.

Update 12:17PM ET – AdSphere services are back online.  Our co-location center experienced a critical power issue within their infrastructure early this morning that took some of our equipment offline.  We are working to bring all of our capture servers back online and are still assessing the situation and what impacts it may have had on our video feeds.  We expect some impact on our airing detections for today.  More information as it becomes available.

Update 7:55PM ET – We regret to inform that today’s issues impacted some of our video streams from 7:30a to 11:30a EST.  This will result in lost detections on some networks during this period of time.  We are waiting on full details from our provider as to what happened that caused the issue and what remedial steps will be taken to prevent another occurrence.  It is hard to express the frustration and disappointment we experience when something like this occurs and we offer our humble apologies.

Some videos are not playing properly in AdSphere this morning.   There was a fix done overnight on another video related issue that is to blame and our development team is currently working to resolve the current issue as quickly as possible.

10:55AM PDT Update – Video playback issue has been resolved.  We apologize for any inconvenience.

Our ETL is offline due to a technical issue that our team is working to resolve.  Airings will be delayed getting into AdSphere and AVS until the matter is resolved and the system catches up.  We apologize for this inconvenience.  More information will follow.

2:45PM PDT Update – The issue has been resolved and the ETL is in catch up mode.  Should be caught up in the next couple of hours.

Airings data is delayed today due to an issue with some new ETL code that was deployed yesterday evening.  There have been isolated occurrences of our ETL crashing and causing data delays so our team has redesigned aspects of the ETL software to address such issues.  Yesterday’s deployment of the new code base ran into an issue very early in the AM.  The issue was documented and a software patch was created to address.  The system is currently playing catch up.  We assume it will be back to normal before end of day.  We apologize for any inconvenience.

8/19/20 Update at 7AM PT – The ETL is still playing catch up after completing importing all of the airings through end of calendar day 8/18.  Taking longer than expected to catch up.  More details to follow.

8/19/20 Update at 7:50PM PT – ETL is almost caught up at this hour.  The new ETL code seems to be working well!

Yesterday’s storms across the East Coast have taken a toll on some of the OTA (Over the Air) stations we monitor for broadcast digital networks such as Antenna TV.  The signals from more than one broadcast station across the region have been disrupted due to the storms and damage to equipment and broadcast antennas in some markets.  It may take a few days for us to get Antenna TV back on the air .  More information as it becomes available.

8/12  Update – Antenna TV is back online.  Grit and This TV are currently offline also due to storm damage.  Expect these will be back online in the next few days.  More details to follow.

8/13  Update – It appears that Grit and This TV are also back online.  As the region recovers from the recent storm damage, we’re hoping there will be no further interruptions in signal.

1:49AM ET – The ETL which updates DRMetrix’s airing database had a hiccup and is running about 8 hours behind at the present time.  Airings are being delayed getting into the AdSphere and AVS databases.  We expect the issue to be resolved shortly and the ETL will be in catch up mode for the remainder of the day.  This should not impact any of the data but will delay it’s reporting in our system today.  We apologize for any inconvenience.

Just after 5PM ET today, one of our vendors, Cloudflare, which protects DRMetrix.com, AdSphere™, and AVS™ from DOS (denial of service) attacks went offline.  This is a very rare event which likely is impacting thousands of websites that use Cloudflare’s service.  At the time of this writing, we can’t reach cloudflare.com but we did call their offices and they are aware of the problem and have their engineers working on it.  We’re sure this will make some news later today.  We hope Cloudflare is able to quickly resolve their issues!

5:36PM Update – DRMetrix sites appear to be back online.  During the outage, we checked other popular websites such as uber.com, authy.com, yelp.com, and many others were also offline.

Cloudflare DNS goes down, taking a large piece of the internet with it

The following is from Cloudflare’s website:

There is a sync issue that is impacting 2020 airings data in the new AVS dashboards.  Some duplicate airings are being pulled into the dashboards for 2020 causing the spend and airing counts to be inflated a bit higher than actual counts.  AVS Dashboard airing data for 2018 and 2019 is not affected and counts between AdSphere and AVS dashboard match.   The AVS Dashboard technical team is working on this issue and we expect resolution soon and we’ll update here.  A status message is being displayed when users log into AVS Dashboards to advise of the temporary airing count issue.

Update 7/15 – The sync issue impacting 2020 airings has been resolved.

Capture Server went down on 7/3 at 21:22 ET.  The issue was resolved on 7/5 at 16:10 ET.  The following networks were impacted:  Food, Fox Business, Fox Sports 1 & 2, Fusion, FX Movie, FXX, and Golf.

We apologize for this issue and we’re researching the root cause.