Bread and Butter: A Quick Primer on Pickles

Sometimes you need a way to preserve objects in their exact state to be reproduced later. This is especially useful for testing, or recreating customer issues. Fortunately, Python offers a facility to do just this: Pickles. For a bit more about pickles, check the Python docs: https://docs.python.org/2/library/pickle.html

Mix Up Some Brine: How to Make a Pickle

Any object in Python can be pickled simply by calling:

import pickle
with open(pickle_filename, 'wb') as pickle_file:
    pickle.dump(results, pickle_file, 2)

The following patch, if applied to zenmodeler.py, will create pickle files for all the datamaps generated during modeling:

diff --git a/Products/DataCollector/zenmodeler.py b/Products/DataCollector/zenmodeler.py
index 386847f..ef4eb54 100644
--- a/Products/DataCollector/zenmodeler.py
+++ b/Products/DataCollector/zenmodeler.py
@@ -634,6 +634,26 @@ class ZenModeler(PBDaemon):
                     try:
                         results = plugin.preprocess(results, self.log)
                         if results:
+###############################################################################
+                            import os
+                            if not os.path.isdir(device.id):
+                                os.mkdir(device.id)
+
+                            plugin_prefix_length = None
+                            if plugin.__module__.startswith('ZenPacks'):
+                                plugin_prefix_length = 5
+                            else:
+                                plugin_prefix_length = 0
+
+                            pickle_filename = '%s/%s.pickle' % (
+                                device.id, '.'.join(
+                                    plugin.__module__.split(
+                                        '.')[plugin_prefix_length:]))
+
+                            self.log.info("Dumping results to %s", pickle_filename)
+                            with open(pickle_filename, 'wb') as pickle_file:
+                                pickle.dump(results, pickle_file, 2)
+###############################################################################
                             datamaps = plugin.process(device, results, self.log)
                         if datamaps:
                             pluginStats.setdefault(plugin.name(), plugin.weight)

Opening the Pickle Jar

After you have save an object to a Pickle file, you can load it back again as a Python object as follows:

import pickle
datamaps = pickle.load("filename")

Once loaded, you can interact with the object just as you would any other object.

I Wanted Sweet, Not Dill: Modifying Pickles

Pickles are saved as binary files, which can be easily copied and transferred. However, they are not so easy to read, or edit. In some cases you may want to take an existing pickle and inspect its contents, or even update the object’s attributes. The following script is a sort editor for pickle files. It loads the pickle file from the specified path to an object, and then drops you to an interactive Python shell, where you can interact with the object however you need to. Once finished, you can write the object back to the original pickle file.

This was something I created for a specific purpose, but should be generally useful. However, it has not been tested extensively. Please update this doc if you find any bugs.

#!/usr/bin/env python

# from IPython import embed
from pprint import pprint
import pickle
import os.path
import os
import sys
from datetime import datetime

# Greeting
print "Welcome to edPick: a simple editor for pickle files\n"

# Get file path
path = sys.argv[1] if 1 < len(sys.argv) else ''
valid_path = os.path.isfile(path)
while not valid_path:
    path = raw_input('Enter file path [./modeling_data.pickle]: ')
    path = path or './modeling_data.pickle'
    if os.path.isfile(path) :
        valid_path = True
    else:
        print "Weird.  I can't find that file.  Try again.\n"

# Load pickled data
data = pickle.load( open( path, "rb" ) )
pprint(data)

# Modify data
print "Modify the <data> object as needed.\n"
print "Then hit Ctrl-D to exit interpreter and write out your new pickle file.\n"
print "Or hit Ctrl-Z to abort.\n"
# embed()
import readline # optional, will allow Up/Down/History in the console
import code
vars = globals().copy()
vars.update(locals())
shell = code.InteractiveConsole(vars)
shell.interact()

# Backup old file
suffix = '.'+datetime.fromtimestamp(os.path.getmtime(path)).strftime('%Y%m%d-%H%M')+'.old'
os.rename(path,path+suffix)

# Write out new pickle data
pickle.dump( data, open( path, "wb" ) )
print "<data> has been written in place of the file you specified.\n"
print "Your original pickle file has been moved to '%s%s'.  You can remove it if you do not need it.\n" % (path, suffix)

Just save it to your path, and you can run it whenever you need it.