System Status – 7/28/21

I’m happy to report that network reporting delays have largely been resolved. Additional improvements are coming this week when the ACR team implements a new “multi-thread” solution designed to further speed up processing and system stability. A lot of work is being done on infomercials and we’re planning to push additional enhancements over the weekend to get infomercial detections back to normal. Ad discovery has become a main focus so that our team can get caught up on bringing new creatives, brands, and advertisers into the system. Those of you who wish to upload new creatives via AVS can help short cut that process. Thank you for your continued patience as we continue to work to get our systems back to normal.

System Status – 7/27/21

An issue with OTA (Over-The-Air) networks has been identified. Since our ACR switch in July, it has caused local market airings to be incorrectly reported as national causing spend levels on some campaigns to be inflated. Our team will be pushing a fix for this tonight and AdSphere may be inaccessible for 30-60 minutes. When the fix for this issue is deployed, it will remove local airings across all OTA networks from our database since beginning of July. We sincerely apologize for this issue and not catching it sooner.

Update: The local OTA airings have been removed from the database and AdSphere reporting of units and spend for those impacted brands has been normalized.

System Status – 7/21/21

ACR team has been working on some updates to further increase system stability which were pushed during the 8AM ET hour this morning. This should translate to more airings processed per hour, more networks catching up to current day reporting, and a host of other benefits. In the hours since the update, we have witnessed an increase in airings processed per hour which is another positive step forward. On the infomercial detection front, the team reports they are making good progress and they plan on rolling out fixes in this area tomorrow. Lastly, ad discovery has come back online this week and our internal research team is once again processing new creatives and brands.

System Status – 7/17/21

Happy Saturday everyone!  I am providing a more comprehensive update today.  First and foremost, this ACR transition was not by choice but rather dictated by circumstances beyond our control.   We have been working to prepare for this transition since Q4 of last year and, despite our best efforts, our new ACR team was unable to deliver the seamless transition we were promised.  There have been numerous issues that our internal and ACR teams have spent countless hours trying to solve over the last two weeks.  I have spent many a late night myself, staying up with the teams, as I can't rest knowing they are working until 3-4AM most days. 

Initially, the focus was on network capture servers that were falling behind delaying reporting of airings.  Some were worse than others but many networks fell "days" behind at one point earlier in the month.  As of today, the vast majority of networks are reporting same day and within a few hours of real time, which is a huge improvement.  For the handful of networks still lagging behind (by hours not days) the gap has been closing.  Even over the past 24 hours we've seen significant improvements.  The ACR team is working to further optimize but this aspect is getting more stable by the day with significant and measurable improvements.  To help the systems catch up, we initially had to take infomercial detections offline.  This past week, one of our focuses has been to bring them back online.  Our team is working hard on this aspect and infomercial detections are starting to come into the system the past day. We're hoping to make significant gains in this department over the next few days.  Also, when we took the infomercials offline, the ACR system started to report short form detections within certain infomercials.  An initial fix for this was put into the system in the past 36 hours and we are currently working to eliminate the issue entirely while, at the same time, bringing back normal infomercial detections.  

With everything going on, the team has also been chasing down another issue where the system was matching the wrong creative length in certain instances.  A fix for this was pushed about 24 hours ago and we're already seeing a huge improvement.  The team will continue to investigate edge cases daily until this problem is resolved.  

As the systems catch up, the video and thumbnail creation routines are also playing catch up.  The team is focusing on getting these back online in lock step with detections so that these assets are more immediately available in AdSphere.  We hope to see the results of all of these improvements in the coming days so that we can begin to reprocess back days in July using the latest version of the updated ACR software.   

Normally, our network feeds expire after 10 days but we have created a new procedure to permanently store the prior day video feeds in July so that our ACR team will be able to reprocess all prior days once we are happy with how the system is processing current day airings.  The good news here is that the historic airings data in AdSphere for July will be updated/fixed in the coming days.  

Our ad discovery routines are also new and were brought online this past week.  Our team is playing catch up getting new July creatives into the system.  Those of you receiving tracking and alerts emails will start to see these new ads coming through.  In the interim, if you notice any of your ads are not being auto-detected, please consider uploading them using the AVS creative upload feature.   If you don't have access to AVS, please email me at joseph.gray@drmetrix.com   

As we enter this new media week, we can only hope that the worst of this transition is behind us.  Now we will move into a phase were we will be looking at data on a daily basis trying to find issues and edge cases so that our ACR team may continue to fine tune the new system.  

We continue to be amazed at the patience and understanding of our clients, many of whom have been sending well wishes to us during this difficult time.  Thank you so much.  We continue to be incredibly grateful for everyone's support.