Kicking off Quick Aggregation Jobs

Assumptions:

  • There are raw_v2_* tables containing yesterday’s data.
  • The are no records in meta_batch where batch_status is not COMPLETED or FAILED with a timestamp less than today. The aggregation won’t run if it finds incomplete batches.

Database Adjustment to Start Aggregation

This involves 3 database-modifying steps:

  1. Make sure that Analytics is running.

  2. reset yesterday’s aggregation status (‘reporting’ database):

    update meta_agg_daily
set status = 'UNRUN'
where date_key = to_days(now()) - 1;

  • If you don’t have a record, you can insert one:

    insert into meta_agg_daily (date_key)
values (to_days(now()) - 1);

  1. Kick off the aggregation job (‘reporting’ database):

    update QRTZ_TRIGGERS
set next_fire_time = now()
where trigger_name = "AggregateDataTrigger";

Using the Tools from ZenPacks.zenoss.psSelfMonitoring

  • Clone the repo from https://github.com/zenoss/ZenPacks.zenoss.psSelfMonitoring

  • On the Analytics server, copy the files as zenoss user:

    git clone git@github.com:zenoss/ZenPacks.zenoss.psSelfMonitoring.git
cd ZenPacks.zenoss.psSelfMonitoring/ZenPacks/zenoss/psSelfMonitoring
cp scripts/* /opt/zenoss_analytics/bin

  • Ensure you have /opt/zenoss_analytics/bin in your $PATH env variable. Once the files are in place you can use them. Of note are:

  • set_analytics_env.sh

    This one sets up all the DB settings for other scripts. Its worth looking at. In fact you can use a similar script to quickly enter the MySql db:

#!/bin/sh
# Name: enter_mysql.sh
# Function: Enter the analytics database
# Sets environment variables based on analytics configuration

if [[ -x $(which mysql) ]]; then
   MYSQL_PATH=$(which mysql)
else
   echo "No apparent MySQL client exists! exiting..."
   exit 1
fi

CONFIG_FILE='/etc/zenoss_analytics/zenoss_analytics.conf'
ETLHOSTNAME=`grep "^etl.jdbc.hostname" $CONFIG_FILE | cut -d"=" -f2`
ETLPORT=`grep "^etl.jdbc.port" $CONFIG_FILE | cut -d"=" -f2`
DBUSER=`grep "^etl.jdbc.username" $CONFIG_FILE | cut -d"=" -f2`
DBPASS=`grep "^etl.jdbc.password" $CONFIG_FILE | cut -d"=" -f2`
ETLDB=`grep "^etl.jdbc.dbname" $CONFIG_FILE | cut -d"=" -f2`
$MYSQL_PATH -u $DBUSER --password=$DBPASS -h $ETLHOSTNAME -P$ETLPORT $ETLDB -A
  • start_model_load_now.sh If you haven’t yet loaded your model onto the server, this will expedite. Sets triggering to 10 minutes in the future.
  • start_aggregation_now.sh If you are ready to aggregate and have setup batch jobs for it, this will speed up that process by setting the trigger time to 10 minutes.
  • remove_extractor.sh This one can remove some of your unwanted batches

The others we list without comment:

  • show_aggregation_detail.sh
  • show_aggregation_config.sh
  • show_aggregation_highlevel.sh
  • show_analytics_task_status.sh
  • show_batch_status.sh
  • show_processlist.sh
  • show_queries.sh
  • show_settings.sh
  • monitor_aggregation_counts.sh
  • monitor_analytics_counts.sh
  • monitor_db_connection_counts.sh
  • monitor_etl_batch_states.sh
  • monitor_etl_performance.sh
  • remove_metric.sh
  • remove_zenoss_instance.sh
  • retry_batch.sh

Dealing with Time: Forcing Aggregation

Make sure your current time in correct. Analytics is very sensitive to time in terms of aggregation triggering.

Things to remember:

  • If you started your Analytics server from a snapshot, you should IMMEDIATELY reboot that snapshot to ensure all services are running normally.
  • Check that NTP is running
  • Check the current data and insure it is correct.
  • Analytics will only create daily and hourly tables from yesterday’s data!
    • One dirty trick you can do to force faster aggregation is to set the system time to tomorrow’s date, making Analytics think the data you currently have is from yesterday.
    • Once aggregation is finished to your satisfaction, start/restart NTP.