AdSphere Was Down for 60+ Minutes this Afternoon

We apologize for an issue that took AdSphere offline this afternoon.  No airings data was lost but our users were not able to reach and/or log into the system.  We’re still investigating what caused the problem.  We’ll post another update as soon as we have more details to share.

4:25PM PT Update – AdSphere is back online but the system is not stable at the moment nor operating normally.  We believe there is a software related issue creating a large number of database connections and overwhelming our web and database servers.  The development team will be isolating the issue in their new day and we expect to have resolution soon.  Adsphere should be back to normal by the morning, if not beforehand.  Again, we apologize for this inconvenience.

Our development team pushed a new build of AdSphere this morning which is causing airing detail reports to download with no data.  The team is currently working to rollback the latest build or fix the underlying issue.  We thank our clients who reported the issue!  We hope to have airing detail reports working properly in the short term.  Thank you for your patience while we fix this issue.

8:10AM Update – Rollback of AdSphere build has resolved this issue.  We apologize for any inconvenience.

We have a capture server that went offline end of last week that our technical team has been working to resolve.  It is impacting the following four networks that are currently offline.  Create, Comet, CW Network, and ION Network.  We apologize for any inconvenience and we will post an update when this situation is fully resolved.

5/15/20 Update – Underlying problem that caused the above issue was finally isolated and resolved today.  We apologize for the inconvenience and are glad to have these networks back online!

DRMetrix was informed that all video streams will be off and on during an expansion maintenance window from 1am until 7am EST today.  DRMetrix will not be able to detect all airings at normal levels of accuracy during this time frame.  We apologize for the inconvenience.

We are investigating an issue with network log view this morning where the same airings data is being shown for all recent dates. This may have something to do with the recent leap year. We apologize for the inconvenience. More details to follow.

8:18AM update. Network Log View issue has been resolved. Turns out it was indeed related to leap year.

We had a capture server malfunction impacting four OTT networks from 4AM EST on 2/23/20 until 2AM EST on 2/24/20. The networks impacted were Heroes & Icons, Laff, Bounce, and Charge. We apologize for any inconvenience caused.

Most of DRMetrix’s systems were back online and functioning normally as of 3AM this morning. Due to a catastrophic controller failure on our large storage array, DRMetrix systems went offline and were unable to capture airings data from approximately 11PM on 2/14 (Friday evening) through 3AM this morning. There are a few network capture servers that are still offline and being worked on today. We will be speaking with our hardware vendors to determine why redundancy measures failed and what can be done to prevent a controller failure like this in the future.

We sincerely apologize for the loss of airings data this weekend.

DRMetrix encountered a catastrophic hardware failure late Friday evening. A controller on our main storage array malfunctioned and the system did not properly fail over to the redundant controller. We are still investigating why the built in redundancy failed. Service availability was affected through end of day Saturday. At this time, systems are back online and are catching up on the processing of airings. We are still determining the overall impacts of the hardware failure and whether or not any airings data was lost. More details will be posted when available.

We had a network outage on DIY for 50 minutes today between 13:00 to 13:50 EST. Our apologies for any inconvenience.

Our new February 2020 build was released today. The chat service we use, provided by a third party (Zoho), which allows our users to initiate chats for support inside of AdSphere, is not operating properly this morning. It appears Zoho is having an issue with their servers. While Zoho works to fix the issue, please email us at support@drmetrix.com if you have any issues with the new build and/or have questions. We apologize for any inconvenience and hope our chat feature will begin operating properly again soon!