Windows ZenPack

Overview

• Methods of Monitoring
  • WMI enumeration over winrm
  • PowerShell using a remote shell over winrm
• Differences
  • WMI queries through DCOM
    • Mostly modeling. Service, IIS, Process, WinRMPing datasources
  • Powershell for everything else

Authentication

• Local Authentication
  • Simple user/password sent over clear text
  • Can be useful for tcpdump
• Domain Authentication (Kerberos)
  • Encrypted
  • Single user over multiple trusted domains

Kerberos Authentication

Process:

• Kinit asks for a krbtgt (kerberos ticket granting ticket) from the kdc
  1. a.Sends user/pass to kdc for authentication
  2. b.Any error messages you see that begin with kinit: only deal with authenticating to the kdc
• After successful login, we then create a context for the device to send our winrm messages
  1. c.We request from the kdc a ticket to use the HTTP/HTTPS/WSMAN(2.7.0) service on the device
     1. Server not found in kerberos database
     2. Usually reverse DNS mismatch
     3. HTTPS is the only spn that would need to be added
     4. zWinRMKrb5DisableRDNS
  2. b.Successfully get HTTP/HTTPS/WSMAN ticket
     1. i.Happily send/receive encrypted payloads to/from device
• Trusted realms
  1. d.If an org has a trust between other domains, a single user can be used from the main domain
  2. e.Set up zWinKDC, zWinRMServerName, zWinRMPassword normally
  3. f.Use zWinTrustedRealm/zWinTrustedKDC to specify the domain and kdc for that domain

Example:

• example.com is the main domain

• austin.example.com is a trusted subdomain

  • zWinKDC: dc01.example.com (recommend using ip for fewer dns lookups)
  • zWinRMUserName: zenny@example.com
  • zWinRMPassword: SomeCrazyPassword
  • zWinTrustedRealm: austin.example.com
  • zWinTrustedKDC: dc01.austin.example.com (or ip)

Miscellany

• Location is /opt/zenoss/var/krb5/krb5.conf
• Cache files are located in /opt/zenoss/var/krb5cc/user\@domain.com
• To verify user/pass, at a zenoss prompt:
  • [zenoss]$ kinit user@DOMAIN.COM
    • Note the domain in all caps
• To view cache for a user and verify we have a good ticket for the HTTP/HTTPS/WSMAN service:
  • [zenoss]$ klist -c /opt/zenoss/var/krb5cc/user\@domain.com
• Extra include directory
  • Default is /opt/zenoss/var/krb5/config
  • zWinRMKrb5inlcludedir
  • Can ONLY contain valid kerberos configuration files
  • Any name you want for the file(s)
• Enter multiple kdcs for zWinKDC
  • 10.10.10.10, 10.10.10.20, kdc.example.com
• What do various errors mean? And how to debug or solve?
  • https://technet.microsoft.com/en-us/library/bb463167.aspx
  • Check the wiki + comments section
  • Server not found in kerberos database
    • Usually reverse DNS mismatch
    • HTTPS is the only spn that would need to be added
    • zWinRMKrb5DisableRDNS

Performance Counters

• Windows Performance Counters
  • Uses get-counter PowerShell cmdlet
• MS SQL Performance Counters
  • Pulls counters directly from SQL Server, i.e. sql SELECT statement
• Debugging:
  • Counter corruption and how to deal with those?
    • lodctr: https://technet.microsoft.com/en-us/library/bb490926.aspx
• Missing/corrupted counters are occasionally an issue on the monitoring target
  • Can occur after installing software that includes its own counters, such as MS SQL Server
  • Another possible cause - disabled in Registry
  • May also need to investigate possible KB issues
• Example Resources:
  • https://support.microsoft.com/en-us/help/2554336/how-to-manually-rebuild-performance-counters-for-windows-server-2008-64bit-or-windows-server-2008-r2-systems
  • https://social.technet.microsoft.com/Forums/windowsserver/en-US/46a21b31-f5fc-4a44-bd4c-c9884a923943/performance-monitor-in-windows-server-2012?forum=winserver8gen
  • https://support.solarwinds.com/Success_Center/Server_Application_Monitor_(SAM)/Performance_Counters_are_not_working_or_missing_in_PerfMon
  • http://naftalyshprai.blogspot.com/2009/08/rebuild-procedure-for-corrupt-windows.html
• Different solutions exist depending on Windows version and other factors
• Debugging:
  • Missing counters
    • typeperf -qx
    • PowerShell - no need to sift through list
      • Get-Counter -listset process*
        • Returns counter sets for Processor Information, Processor, Process
      • (Get-Counter -listset objectname).pathswithinstances
        • Shows all currently installed counters for specific object
  • wbemtest
    • Available from run command
    • Connects to local windows host
      • WQL queries
      • CIM data
    • Can also check existence and security settings using WMI Control in MMC

Services Monitoring

• Options for Services Monitoring
  • ~130-200 service components per device
  • Templates
    • Making a change to DS can be expensive
  • Windows Services Page
    • Any change IS expensive
• How does the RegEx work?
• WinService template, Windows Service datasource
• Service State
  • Based on Windows services status
  • Running, Stopped, Paused
  • state_state datapoint
• Service Status
  • Based on zenoss events

Order of precedence for enabling/disabling monitoring

1. User manually sets monitoring
2. ‘DefaultService’ datasource from the template associated with the service
3. Datasource other than the DefaultService in the template associated with the service
4. Monitoring is enabled via the Infrastructure -> Windows Services page

Note: At least one Windows Service datasource must be enabled

Event Log

• PowerShell
  • Get-WinEvent for .NET >= 3.5
    • Can use xml generated by MS in Event Viewer
    • Returns EventLogRecord objects
    • $$_.Id
  • Get-EventLog for .NET < 3.5
    • Slower, no xml query
    • Returns EventLogEntry objects
    • $$_.EventID

Performance / Scale

• What to check for performance/scale issues?
  • Twisted uses file handles for open connections
  • ulimit
• What to avoid?
  • Very low cycle time for datasources
    • Perfmon ok to have lower interval as the get-counter cmdlet runs over the entire zenpython cycle

Tracing & Logging

What logs are relevant for supporting Windows ZenPack?

• Modeling:

  • zenmodeler and/or zenhub logs

• Monitoring:

  • zenhub, zenpython logs

• GUI

  • event.log

• Event Viewer:

  Event Viewer
     -> Applications and Services Logs
        -> Microsoft
           -> Windows
              -> Windows Remote Management
                -> Analytic
  

  • wevtutil: https://github.com/zenoss/microsoft.tools/blob/develop/scripts/zenoss-winrm-analytic-logs.ps1
  • Shows requests received by Windows and what the responses are

Pitfalls / Limitations

• Encourage users to test windows service datasources in a subclass
• See wiki
• Installing 2.7.x can take a long time

References

https://www.zenoss.com/product/zenpacks/microsoft-windows