Solr Catalogs (ModelIndex) Review

What is Solr?

http://lucene.apache.org/solr/

Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene™.

Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world’s largest internet sites.

What do we call it?

  • Solr:
    • Zenoss.resmgr/Infrastructure/Solr service in Control Center
    • Show Solr UI
  • modelindex:
  • model_catalog:
    • The Solr “collection” in which all of this is stored
    • Currently the only Solr collection we use
  • ModelCatalog:
    • Products.Zuul.catalog.model_catalog
    • Provides an API to index, unindex, and search Zenoss objects using modelindex
    • Adds ZODB transactional layer via two-phase commit
  • ModelCatalogTool:
    • Products.Zuul.catalog.model_catalog_tool
    • Provides an interface similar to ICatalogTool (IModelCatalogTool)

What catalogs are moved to Solr?

  • global_catalog (ICatalogTool still works)
  • deviceSearch (via LegacyCatalogAdapter)
  • layer2_catalog (via LegacyCatalogAdapter)
  • layer3_catalog (via LegacyCatalogAdapter)

What is LegacyCatalogAdapter?

  • A ZODB “SimpleItem” that behaves like ZCatalog, but uses ModelCatalog.

What is ZenPackLib going to do?

  • Move all “index_scope: global” catalogs into Solr using LegacyCatalogAdapter.
  • Migrate existing global catalogs to Solr? Maybe, but when?
  • Backwards compatibility can be maintained.
  • Going into ZenPackLib 2.1.0.

Other

  • Some things that indexed via obj.index_object() now required notify(IndexingEvent(object))

Performance

  • Queries in generally have a much more stable response time.
  • “Hot” queries are intrinsically slower due to networking overhead to Solr.
  • There are effectively no “cold” queries.

How is this different than CatalogSerice?

  • Same in respect to performance trade-offs: Reliable response times, but often slower.
  • CatalogService uses Lucene. ModelCatalog uses Solr.
  • Think of Solr as a clustered (distributed) Lucene.
  • Removes one of the remaining single points of failure.
  • Enables moving to a future of active/active HA.
  • ModelCatalog is in the Core platform. CatalogService is only in RM/UCS-PM.

Show LegacyCatalogAdapter usage in zendmd.