Datasources in Detail

How to Determine what Daemon Runs a DataSource

The ZenPack documentation should list of daemons and datasource types. If that is not quite enough, then try this:

• First look at the collector daemon’s class, it will specify a configurationService.
  • For a custom daemon, look in the daemons folder.
• Then look at that service, to see what kinds of datasources it includes.
  • For custom services, look in the services folder.
• If your datasource inherits from PythonCollector, it probably runs zenpython

How to Determine How Templates Set the DataSource

• Look for the sourcetypes variable in objects.xml or datasource.type in zenpack.yaml.
• Now compare that to existing datasources’ sourcetype variable.

General Questions and Answers

• Question: What’s the difference between a datasource and a datasource plugin?

  A datasource is an instance of RRDDataSource. It’s the object that’s part of a monitoring template and stored in ZODB. A datasource plugin is not a persistent ZODB object. It’s a class that is used to perform collection of a specific subclass of RRDDataSource instance: PythonDataSource.

  A PythonDataSource has a plugin_classname property that must be set to the fully qualified classname of a PythonDataSourcePlugin class such as “ZenPacks.zenoss.AWS.dsplugins.S3BucketPlugin”.(edited)

  Also datasource plugins are a zenpython concept. So all PythonDataSources have PythonDataSourcePlugins, but other kinds of RRDDataSources do not.

  So for instance, vsphere has a datasource, but not a plugin.

PythonCollector

When using PythonCollector there are two ways to configure collection as described in PythonCollector’s documentation.

1. Using the existing PythonDataSource. http://wiki.zenoss.org/ZenPack:PythonCollector#Using_the_Python_Data_Source_Type_Directly
2. Subclassing PythonDataSource. http://wiki.zenoss.org/ZenPack:PythonCollector#Extending_the_Python_Data_Source_Type

• How do you decide which method to use?

  It’s pretty easy to figure out. Do you expect that a Zenoss admin would ever be adding one of your datasources to a monitoring template? If so, the answer is to subclass PythonDataSource to create your own type. If the answer is no, the answer is to not create a PythonDataSource subclass, but to just deliver your own monitoring templates that use the Python data source type with the plugin_classname populated.

  Zenoss adds each PythonDataSource subclass as an option in the datasource type drop-down when creating a new datasource in a monitoring template. So don’t pollute that drop-down with things the user will never use, but don’t make the user create “Python” type datasources where they have to provide the correct value for plugin_classname.

  Using the Ceph ZenPack as an example, the suggestion would be that none of the PythonDataSourcePlugins implemented therein are generic enough to deserve a PythonDataSource subclass.

  Good examples of plugins that deserve a PythonDataSource subclass are “Windows Perfmon”, “Windows Event Log”, “Windows Process”.

Invalid config_keys() in Datasources: Clearing Redis

Sometimes if you work on datasource, you can accidentlly insert a bad entry into your config_key() method. This can get cached in redis and result your datasource using the cached value, even if you change the config_key() method later.

You may see a messages like this in your collector logs:

2017-08-17 13:12:08,839 DEBUG zen.collector.config: Fetching daemon configuration properties
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
   msg = self.format(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
   return fmt.format(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
   record.message = record.getMessage()
File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
   msg = msg % self.args
TypeError: %d format: a number is required, not str

This probably means that the cached config_keys are not in the proper order. Ensure your config_key returns items in the following order:

* context.device().id,
* datasource.getCycleTime(context),
* <whatever>
* <whatever>
* <whatever>

You must ensure that the first and second fields are (str, int).

If you get a broken config key n collectorredis or redis, you may have to flush redis’ cache :

• connect to the redis container(s)

• list the contents of table 2:

  (zenoss) [zenoss@6f913fd44dbb ~]$ redis-cli -n 2 --scan --pattern '*'
     collector_cache_zenpython-0_devices@@nutanix64
     collector_cache_zenpython-0_threshold_classes
     collector_cache_zenpython-0_property_items
     collector_cache_zenpython-0_thresholds
  

• Open up redis table 2 in redis-cli:

  (zenoss) [zenoss@6f913fd44dbb ~]$ redis-cli -n 2
  127.0.0.1:6379[2]>
  127.0.0.1:6379[2]> del collector_cache_zenpython-0_devices@@nutanix64
     (integer) 1
  127.0.0.1:6379[2]> del collector_cache_zenpython-0_property_item
     (integer) 1
  127.0.0.1:6379[2]> exit
  (zenoss) [zenoss@6f913fd44dbb ~]$
  (zenoss) [zenoss@6f913fd44dbb ~]$ redis-cli -n 2 --scan --pattern '*'
  

• Now retry your datasource