Data Processing Delays
Resolved·Full outage

Normal operations restored.

Fri, Dec 13, 2024, 11:10 AM
(3 months ago)
·
Affected components
Updates

Resolved

Normal operations restored.

Fri, Dec 13, 2024, 11:10 AM

Monitoring

A fix has been implemented that we believe should resolve the incident; data volumes are rapidly returning to normal.

Fri, Dec 13, 2024, 01:28 AM(9 hours earlier)

Identified

Processing is resuming - we are starting to see data returned to the Visualizer and API. We expect to restore normal operations soon, and will then concentrate on backfilling any missing JA3 signatures.

Fri, Dec 13, 2024, 12:30 AM(57 minutes earlier)

Identified

We are continuing to work on a fix for this issue.

Fri, Dec 13, 2024, 12:28 AM

Identified

Data processing is working but is taking multiple hours, resulting in no data in the Visualizer and API for queries of "last_seen:1d" (data is shown for "last_seen:2d". We're updating our platform to improve data processing.

Thu, Dec 12, 2024, 11:42 AM(12 hours earlier)

Identified

We continue to troubleshoot with our vendor and are working to restore most data to our API and Visualizer tonight. Certain fields (like JA3) may not be populated until after a full recovery.

Wed, Dec 11, 2024, 10:08 PM(13 hours earlier)

Identified

Our database vendor has their engineering team collecting information for additional troubleshooting while we make changes to our platform to attempt to restart data processing. We'll continue to provide updates at least every few hours as we work.

Wed, Dec 11, 2024, 04:05 PM(6 hours earlier)

Identified

Data is processing but still has at least a few hours to catch up. We'll provide updates every few hours.

Wed, Dec 11, 2024, 11:03 AM(5 hours earlier)

Identified

We are seeing improved health of the data subsystem but still have a substantial backlog of data to process.

Wed, Dec 11, 2024, 04:20 AM(6 hours earlier)

Identified

Data processing is still stalled, resulting in no results for a GQNL query of "last_seen:1d". We continue to work with our data vendor to identify the root cause.

Wed, Dec 11, 2024, 02:11 AM(2 hours earlier)

Identified

Data processing is substantially delayed, resulting in reduced data available in the Visualizer and via the API.

Tue, Dec 10, 2024, 11:02 PM(3 hours earlier)

Identified

We're currently experiencing a degradation in our database performance that is impacting the availability of some bulk data files.

Mon, Dec 9, 2024, 10:03 PM(1 day earlier)
Powered by