The ETL monitoring script, bb-check-etl.sh, monitors the health of the ETL service.
Overview
The script resides in the [RUNTIME_DIR]/scripts directory and runs as an extension script in the Big
Brother framework. It reports to Big Brother based on the output of the etl_running script which checks whether the ETL service is running or not. The host on which the ETL service is configured to run is also displayed in Big Brother.
Status indicators
The ETL Monitoring script uses the Big Brother color codes for status indicators. The status codes can be green or red.
- Red
- The ETL service is down. This status requires immediate attention.
- Green
- The ETL service is running. Under normal operating conditions, the status is generally green. This condition requires no response.
Logs
The ETL service logs to two files in the [LOGGING_DIR]/debug directory:
- etl.log
- Information from ETL runs. This file may contain data transformation warnings and errors.
Note: Transformation errors do not constitute a failed ETL run. For example, if a corrupt row of data in one of the source tables causes transformation errors, this is treated as a "skipped record" and gets logged.
- etl-out.log
- This file contains logs from the ETL scheduler and logs from ETL runs. It is created by the babysit process on ETL startup.