Catch intermittent failures

Sampling determines what becomes visible

An intermittent fault comes and goes, so it may be gone when you look. Monitoring takes readings, called samples, at set times, or averages them over a period. A short problem can happen between two samples or get lost inside an average. Before you trust a normal-looking graph, check how often it samples and how it averages.

Logs, telemetry and service records may use different clocks, and some update later than others. Line up their times before you put events in order. Otherwise, an effect can seem to happen before its cause just because the times in each source do not match.

Conditions can matter more than elapsed time

An intermittent fault may depend on load, temperature, a scheduled event or a mix of conditions. Record the conditions each time the fault happens, and each time it does not. A healthy period under different conditions may not test the explanation you care about. For example, a cool morning does not test a fault that only appears on hot afternoons.

Seeing the pattern again makes it stronger, but it could still be coincidence. State the link you suspect and look for separate evidence that supports it. Do not keep causing a possibly hazardous failure just to make the pattern easier to see.

Observe within the approved envelope

The approved envelope means the normal operating limits your site has approved for the equipment. Watch the unit under the normal conditions linked to the failure, using existing telemetry or an authorized monitoring plan. Never try to cause the fault by bypassing or turning off protection, running beyond limits or creating an electrical hazard. Decide in advance who can stop the observation and what data will be kept.

Worked example: the misleading clock

A unit log shows a restart at 13:02. A facility record shows a shared event at 13:04. The unit’s clock is known to run three minutes behind.

Adding the three minutes the unit clock runs behind puts the restart at about 13:05. So the restart may have come after the facility event, not before it. Keep the original timestamps and record the correction separately. The offset is only approximate, so do not claim second-by-second accuracy from records that only show minutes.