The \ac{ALMA} telescope is composed of 66 high precision antennas, each antenna produces signals with 16[GHz] bandwidth (4 pairs or orthogonal linear polarization signals). Detecting the root cause of a loss of coherence issue between pairs of antennas can take valuable time which could be used for scientific purposes. This work presents an approach for quickly determine, and in a systematic way, the source of this kind of issues. Faulty sub-system can be detected using the telescope calibration software and the granularity information. In a sophisticated instrument, finding the cause of a loss of coherence issue can be a cumbersome task due to the several sub-systems involved on the signal processing (frequency down-converter, analog/digital filters, instrumental delay), the inter-dependencies between sub-system can make this task even harder. A method based on the information provided by the online calibration software will be used to aid the identification of the faulty unit or the wrong configuration, which is causing the loss of coherence issue. This method uses granularity information for finding the cause of the problem.
The signal path in the \ac{ALMA} telescope passes through a number of analog and digital hardware elements that process the received signal in a variety of bandwidths and signal levels. Although the development of these interconnections has proven to be efficient in delivering high-quality data, the study and search of the offending device can represent a challenging task even for an instrumentation specialist. A granularity exploration is a vital piece of information when the coherence between antennas is compromised, and provides an important clue about the nature of the problem. For example, if one \ac{IF} processor fails, the granularity of the loss of coherence problem will be polarization based.
The granularity information is obtained from the \ac{TelCal} subsystem. The \ac{ALMA} \ac{TelCal} subsystem consists of several calibration algorithms, which are executed and applied to the \ac{ALMA} datasets to ensure that the telescope is and remains in a proper state to observe the subsequent scheduled observations. The calibration result is notified by events, archived in the dataset and published through consumer interfaces to display it as a plot or formatted plain text. The calibration measurement is based on a series of steps, such as a standard interferometry measurement on strong quasar; translation of frequency turns into phase changes from a reference antenna; and adjustment of sub-scan duration to source flux. The result is the delay offset and its error, the error tells us “how much the phase values are correlated to the frequency values”, and it is this figure that specify if the signals are correlated (coherent) or there is a loss of coherence.
According to the expected calibration results, a flow chart breakdown is prepared in order to present which sub-system can be examined for finding the cause of the coherence problem. This chart contains a detailed explanation of which device represent the primary offending unit according to error published by the \ac{TelCal} subsystem. The error cases for this study are mostly centered in the loss of coherence. However, as an additional feature, we proposed to add delay analysis whenever coherence levels allow it since \ac{TelCal} subsystem also provides it during calibration measurements. The delay analysis integrates an extra layer of granularity study as the specific measured delays come from specific hardware elements.
The interaction with \ac{TelCal} relies on an event notification service once the results are ready to be retrieved, the tool for analyzing the data produced by \ac{TelCal} was developed through the use of the Python language. This software tool listens to the notification channel associated to the \textit{Delay Calibration}, and it analyzes the obtained data once \ac{TelCal} reports an event.
The original intention of the phase error value provided by the \textit{Delay Calibration} was to serve a measurement of how precise the time delay is. Furthermore, that value could be also used for analyzing the loss of coherence issues. Since the delay values are presented in a way which is also consistent with how the data is divided and processed by the \ac{ALMA} instrument, that information can be used for finding which sub-system is introducing problems, or which hardware is not correctly synchronized. Considering the number of antennas in the array for the \ac{ALMA} telescope case, it is a difficult task trying to find a loss of coherence problem while an observation is undergoing. As the calibration is a task which must be done on a regular basis, using the \ac{TelCal} for detecting the loss of coherence issues and for identifying the faulty hardware or configuration is a well-favored strategy.