Analytics trobleshooting guide

Based on docs by William Gerber.

What problem do you have?

I get no reports

Are your batches working?

My batches are not working

See at what state they are stuck in at the web UI:

Note

Bi-Directional DNS Resolution

Make sure your DNS on both the RM and Analytics side can fully resolve the each other. If required, add lines in /etc/hosts on the Analytics server and RM so that they can resolve whatever configuration URLs you have set.

Once you set /etc/hosts on Zenoss, make sure to install, configure, and restart dnsmasq so that containers can get the right DNS data.

Can you log into Zenoss Analytics at all?

Zenetl or zenperfetl daemon fails

Have you configured the 4 URLS for Analytics through the WEB UI?

If no, then go configure them.

Update each Hub and Collector so it gets the proper config properties

Check to see if the collector can hit the Analytics server. From the command prompt do the following: wget http://analytics_server_url/reporting

Did you get a text file without error?

Is there enough space on the hard drives to write files to?

If no then get more space or edit the config file for daemon to point to a drive with more space.

Is the log file complaining that it can’t hit the Hub? - Yes: Check daemon connection with the HUB - No: Does deamon try to upload a file at all?

Check analytics availability from collector

Double check that the Analytics server is running by open a browser to the URL http://analytics_server_url/reporting.

If you got a text file than your analytics server is not reachable from collector and you should check firewalss proxies and ports.

If you got an error, you should figure out why your Analytics server is not running.

Is there model data in staging?

  • If debugging just tail ALL the logs:

    cd /opt/zenoss_analytics/logs
tail -f *.log *.out
  --or--
tail -F -n100 *.out *.log |grep -iE '(failed|error)'

  • No: Check triggers

    • Also check the log /opt/zenoss_analytics/catalina.out
    • Ensure Analytics can resolve (via DNS or /etc/hosts) the Zenoss server!

  • Yes: Has it been in staging over 4 hours?

    • No: We only check for Model batches ready to load into DB once every 4 hours.
    • Yes: Check triggers

Check triggers

Log into the reporting database and run this query a few times over a couple of minutes:

select trigger_name, from_unixtime(next_fire_time/1000) as next_time,
from_unixtime(prev_fire_time/1000) as last_time,
trigger_state from reporting.QRTZ_TRIGGERS;

Are some of the triggers state stuck in blocked? If yes, then something caused one or more of the triggers to fail repeatedly and the system finally gave up. Fix the root cause and then update the QRTZ_TRIGGERS table to a state of WAITING and bounce the zenoss_analytics service.

Go to the Analytics application server and look through the logs:

/opt/zenoss_analytics/logs/zenoss_analytics.log:  file for glaring errors.

          **or**

/opt/zenoss_analytics/webapps/reports/WEB-INF/logs/jasperserver.log

After Importing Bundle, I Get this Error when I edit my bundle

  • Cannot get metadata for /organizations/zenoss/Data_Sources/zenoss_reporting: com.mysql.jdbc.Driver This means you imported your bundle without turning off the checkboxs on the import. The Zenoss_reporting datasource is now corrputed.

    To fix this, you must restore the zenoss_reporting datasource from a working install e.g. export it and reimport it into your broken install, or revert your Analytics server to a clean state.

    This can be done in a number of ways:

    1. Revert your Analytics instance to the default state: You’ll lose your dim tables though.
    2. This way restores zenoss_reporting without destroying Reporting db.
      • Stop Analytics
      • drop zenoss_analytics db from mysql
      • As zenoss, run update_db.py
      • Start Analytics

Check daemon connection with the HUB

–tbd–

Does deamon try to upload a file at all?

–tbd–