System Status – 8/11/20

Yesterday’s storms across the East Coast have taken a toll on some of the OTA (Over the Air) stations we monitor for broadcast digital networks such as Antenna TV.  The signals from more than one broadcast station across the region have been disrupted due to the storms and damage to equipment and broadcast antennas in some markets.  It may take a few days for us to get Antenna TV back on the air .  More information as it becomes available.

8/12  Update – Antenna TV is back online.  Grit and This TV are currently offline also due to storm damage.  Expect these will be back online in the next few days.  More details to follow.

8/13  Update – It appears that Grit and This TV are also back online.  As the region recovers from the recent storm damage, we’re hoping there will be no further interruptions in signal.

System Status – 8/4/20

1:49AM ET – The ETL which updates DRMetrix’s airing database had a hiccup and is running about 8 hours behind at the present time.  Airings are being delayed getting into the AdSphere and AVS databases.  We expect the issue to be resolved shortly and the ETL will be in catch up mode for the remainder of the day.  This should not impact any of the data but will delay it’s reporting in our system today.  We apologize for any inconvenience.

System Status – 7/17/20

Just after 5PM ET today, one of our vendors, Cloudflare, which protects DRMetrix.com, AdSphere™, and AVS™ from DOS (denial of service) attacks went offline.  This is a very rare event which likely is impacting thousands of websites that use Cloudflare’s service.  At the time of this writing, we can’t reach cloudflare.com but we did call their offices and they are aware of the problem and have their engineers working on it.  We’re sure this will make some news later today.  We hope Cloudflare is able to quickly resolve their issues!

5:36PM Update – DRMetrix sites appear to be back online.  During the outage, we checked other popular websites such as uber.com, authy.com, yelp.com, and many others were also offline.

Cloudflare DNS goes down, taking a large piece of the internet with it

The following is from Cloudflare’s website:

Cloudflare System Status

Investigating – Cloudflare is investigating issues with Cloudflare Resolver and our edge network in certain locations.

Customers using Cloudflare services in certain regions are impacted as requests might fail and/or errors may be displayed.

Data Centers impacted include: SJC, DFW, SEA, LAX, ORD, IAD, EWR, ATL, LHR, AMS, FRA, CDG
Jul 1721:37 UTC

System Status – 7/14/20

There is a sync issue that is impacting 2020 airings data in the new AVS dashboards.  Some duplicate airings are being pulled into the dashboards for 2020 causing the spend and airing counts to be inflated a bit higher than actual counts.  AVS Dashboard airing data for 2018 and 2019 is not affected and counts between AdSphere and AVS dashboard match.   The AVS Dashboard technical team is working on this issue and we expect resolution soon and we’ll update here.  A status message is being displayed when users log into AVS Dashboards to advise of the temporary airing count issue.

Update 7/15 – The sync issue impacting 2020 airings has been resolved.

System Status – 7/5/2020

Capture Server went down on 7/3 at 21:22 ET.  The issue was resolved on 7/5 at 16:10 ET.  The following networks were impacted:  Food, Fox Business, Fox Sports 1 & 2, Fusion, FX Movie, FXX, and Golf.

We apologize for this issue and we’re researching the root cause.

 

 

System Status – 6/30/2020

AdSphere Was Down for 60+ Minutes this Afternoon

We apologize for an issue that took AdSphere offline this afternoon.  No airings data was lost but our users were not able to reach and/or log into the system.  We’re still investigating what caused the problem.  We’ll post another update as soon as we have more details to share.

4:25PM PT Update – AdSphere is back online but the system is not stable at the moment nor operating normally.  We believe there is a software related issue creating a large number of database connections and overwhelming our web and database servers.  The development team will be isolating the issue in their new day and we expect to have resolution soon.  Adsphere should be back to normal by the morning, if not beforehand.  Again, we apologize for this inconvenience.

System Status – 5/28/2020

Our development team pushed a new build of AdSphere this morning which is causing airing detail reports to download with no data.  The team is currently working to rollback the latest build or fix the underlying issue.  We thank our clients who reported the issue!  We hope to have airing detail reports working properly in the short term.  Thank you for your patience while we fix this issue.

8:10AM Update – Rollback of AdSphere build has resolved this issue.  We apologize for any inconvenience.

System Status – 5/12/2020

We have a capture server that went offline end of last week that our technical team has been working to resolve.  It is impacting the following four networks that are currently offline.  Create, Comet, CW Network, and ION Network.  We apologize for any inconvenience and we will post an update when this situation is fully resolved.

5/15/20 Update – Underlying problem that caused the above issue was finally isolated and resolved today.  We apologize for the inconvenience and are glad to have these networks back online!

System Status – 3/2/2020

We are investigating an issue with network log view this morning where the same airings data is being shown for all recent dates. This may have something to do with the recent leap year. We apologize for the inconvenience. More details to follow.

8:18AM update. Network Log View issue has been resolved. Turns out it was indeed related to leap year.