System Status – Week #2 on Unified Platform – 10/30/23

This morning issues with crons were discovered that process weekly reports. The data is showing in Adsphere and these same reports can be manually run and exported as an interim solution. If anyone needs help to manually generate a critical report, please email support@drmetrix.com for assistance.

These cron issues should be resolved soon and automated delivery will resume and missing reports will be resent in the next day or two. There are a few other issues in AdSphere this morning which are being looked at on priority for resolution. One is that network airing totals at the bottom of network detail pages are inaccurate and don’t match the total airings shown at the top of page. There is a backend process that has been identified as the culprit and it will be resolved soon. Airing level videos are still a work in progress and there is a network mismatch error on video playback in ranking system (which is a minor issue not impacting video playback). A more serious issue is 28.5m long form reporting which is delayed from last week and is not caught up at this hour. These are some of the areas that our team will be focusing on this week for resolution. We apologize for these issues and we’ll be working hard to resolve them as quickly as possible.

A New Chapter Begins…

Dear Valued Customers,

Two years ago, iSpot.tv acquired DRMetrix and we began working toward a unified platform so that we could bring additional networks and features to both AdSphere and AVS. Not only will DRMetrix be able to offer more comprehensive coverage of both Angelo and Hispanic networks, but we will also be able to offer broadcast monitoring of ABC, CBS, NBC, and FOX affiliates in the top 40 markets! Under the unified platform, DRMetrix will also benefit from iSpot’s HD monitoring capability while enhancing our core automated content recognition (ACR) technology.

We will switch over to the new unified platform starting this weekend, midnight on 10/21. AdSphere will still be accessible during this time but there will be no new airing occurrences reported between 10/21 at 12A EST and 10/23 12A EST, although services may be restored sooner. We expect the overnight reports for Friday 10/20 data to run normally.

Note: No occurrences / airings will be lost. There will just be a delay in reporting. The goal is to have all of the data caught up by early Monday AM in time for weekly reports.

The changes happening behind the scenes are rather significant and include moving all of our systems and databases completely to the cloud via Amazon Web Services. These types of transitions are never easy and unexpected challenges can arise so we humbly ask our valued clients for their patience and understanding as we make this critical transition. Please continue to check this post for additional updates which will be provided in the coming days.

Warmest Regards,

The DRMetrix Team

Update 10/23/23 at 3:21AM EDT – Our team has been working all weekend and we have switched over to the new unified platform. Our systems are currently in catch up mode processing airings, videos, and thumbnails from over the weekend. No data was lost but there may be a delay with weekend and current airings until the system fully catches up. System performance may be also be impacted while the system is playing catch up which our team is currently working on. We’ll post another update in the next few hours.

Update 10/23/23 at 10:00AM EDT – System is still processing airings from this weekend and playing catchup so we didn’t achieve our stated goal of having all the data caught up by early Monday AM in time for weekly reports. Please accept our humble apologies. There seems to be an unexpected issue with slow processing of videos and image thumbnails that appears to be the root cause of the system struggling to catch up on processing. With the new unified system, these assets are now being created from HD video, rather than the old SD video, and also the backend processes are different. This will be a main point of focus for the team to resolve today. Once resolved, the entire system will run a lot faster allowing the airing detections to get caught up and return to normal. We apologize for these teething pains but once the system is caught up, we believe it will run considerably faster under the Amazon Web Services (AWS) platform.

We are getting feedback this morning of various issues which we’ll be fixing as quickly as possible. We are working to address all reported issues so if you run into an issue, please email support@drmetrix.com. We appreciate everyone’s help in identifying issues. Once we feel that the new system is stable, and running properly on all fronts, we’ll begin adding new networks and also broadcast stations! More updates will follow.

Update 10/23/23 at 10:25PM EDT – Team is still working very hard to resolve all outstanding issues. While a lot of progress was made today, the system is still pretty far behind and busy processing airings from the weekend. Our team is going to try a few tricks tomorrow to try and speed things up in hopes of getting the system caught up in the next 24 – 48 hours.

Update 10/24/23 at 9:24AM EDT – Encouraging update from the team early this morning. Changes were made to speed up the ETL which processes new airings with a 3X improvement in speed. That means the processing from this weekend should be completed within an estimated 4-6 hours and the system should be caught up within the next 20-24 hours. We appreciate everyone’s patience and understanding during this major transition. It will be exciting to experience all of the performance benefits, additional networks, and new features that will accompany our new unified platform!

Update 10/25/23 at 9:31AM EDT – Yesterday’s estimation of the system catching up in the next 20-24 hours was incorrect. Should have an update later today on the root cause of it. That said, there was progress made on that front and airings from this past weekend are now complete and the system is starting to process current week data. On the better news front, quite a few issues were resolved yesterday and AdSphere should be pretty stable today. If you notice any issues remaining, please email support@drmetrix.com

Update 10/25/23 at 1:32PM EDT – Last night’s update appears to have addressed a number of excel reporting related issues. There remains an issue with advertiser page links in both excel and tracking & alert emails not working properly. When clicked on, the system is throwing an error and crashing. We have asked the AdSphere development team to focus on this issue and fix it as soon as possible. There are also issues with video playback / network mismatch. This is being researched as well. The video playback feature has been enhanced. As before, the target video plays but now one can watch any part of the ad break by rewinding or forwarding through the entire ad break. It’s a really nice enhancement.

Update 10/26/23 at 9:31AM EDT – Root cause of advertiser page link errors has been identified and fix will be implemented in the next few hours. Issues with video playback / network mismatch will be a focus for the team today. Team created a 2nd ETL that is focusing on current day airings which will be caught up today while simultaneously catching up on prior day airings. This will allow regular daily reporting to resume tomorrow as well as access via API to daily airings on the AVS side.

Update 10/27/23 at 9:12AM EDT – Happy Friday! Suffice it to say, it’s been a tough week as this transition hasn’t gone as smoothly as we had hoped. Fortunately, the system has caught up with the processing of airings from prior days and is back to processing airings for the current day! Advertiser page link errors have been resolved. Video playback system is still being worked on and we’ll hopefully have the network mismatch issues resolved soon and the new and improved airing level videos online. We have to get the tracking & alert emails back online too which may take a few more days but progress is being made on all fronts. After we work though any and all remaining issues, our focus will turn to adding new Hispanic and Angelo national networks and to the debut of our new top 40 market broadcast monitoring solution!

System Status – 10/2/23

There is an issue with AdSphere this morning where proper data is not showing from last week. It’s currently 6:50AM so working to contact technical resources to assess the problem. More updates to follow. Apologies to everyone for the issue. Our team has been doing a lot of things in preparation for moving our systems to Amazon Cloud and expanding our coverage of networks, including the addition of broadcast stations in the top 40 markets. With all of the infrastructure changes going on, something unexpected may have happened.

9:00AM Update. Data from last week is now properly showing in AdSphere. Weekly reports will be re-run this evening for Tuesday AM delivery. If the need is urgent, one can log into AdSphere, visit the filters dropdown, find your weekly email reports, and re-run them as needed. We apologize for the inconvenience this morning.