Using AWS CloudWatch to monitor Centrify Audit Trail events in EC2 Linux instances

Background

As more and more organizations run infrastructure in IaaS platforms like Amazon AWS, there's an increased need to enhance security operations and prove effective implementation of security controls.  AWS provides a solution set that includes CloudWatch.  

About CloudWatch

As defined by Amazon "CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real time. You can use CloudWatch to collect and track metrics, which are variables you can measure for your resources and applications." 

For more information, check out the Getting Started guide for CloudWatch:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html

The goal of this article, is to provide some initial guidance to leverage AWS CloudWatch to collect, track and measure Centrify Audit Trail events in UNIX, Linux systems running in AWS.

For a companion article that describes the process for Windows instances, go here: http://centrifying.blogspot.com/2017/04/using-aws-cloudwatch-to-monitor_24.html

About Centrify Audit Events

Centrify Audit Events (CentrifyAuditTrail) is the cross-platform framework used by Centrify Server Suite to document and provide access, privilege and audit trail event data. When a Centrify-enabled service is invoked, an audit trail event is written to UNIX syslog or Windows event log.  These events are documented in the  Audit Events Administrator's Guide for the current version of Server Suite.  The types or content of the events vary depending on the edition (Standard or Enterprise)

For more information, check out the current guide for Server Suite 2017: https://docs.centrify.com/en/css/suite2017/centrify-audit-events-guide.pdf

Pre-Requisites

For this lab, you'll need:

• An AWS Account with the proper VPC setup, privileges in CloudWatch and IAM
• Active Directory (run by you or managed by Amazon) and the proper VPC name resolution and communications
• A Centrify zone, sample users and access/privilege setup
• At least one Linux system joined to Active Directory and the Centrify zone
• The Linux system should have some Centrify data (e.g. logins, privilege elevations) present in syslog.

Centrify AWS Lab:  You'll need to be at Standard Edition level to follow this lab, more info here:

http://community.centrify.com/t5/TechBlog/Labs-Setting-up-an-AWS-Test-Lab-for-Centrify/ba-p/27771

Implementation Overview

1. Set-up your AWS Linux Instances for CloudWatch Logs (use AWS's docs)
2. Verify Centrify Audit Trail events in the CloudWatch log group
3. Identify Access and Privilege-related Metrics provided by Centrify
4. Create the Filters and Assign a Metric
5. Create a Dashboard
6. Create an Alarm

Set-up your AWS Linux Instances for CloudWatch Logs

For information on this topic, please review AWS's documentation:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html

Once you have the /var/log/messages logs for your instances, please proceed to the next section.

Verify Centrify Audit Trail events in the CloudWatch log group

1. Go to your CloudWatch console: https://console.aws.amazon.com/cloudwatch/home
2. Click on Logs > Click on the log group for your Linux instances (e.g. "/var/log/messages"  or the group you are using for your Linux syslog)
3. Click on Search log group and in filter events, type "AUDIT_TRAIL"
4. Verify the results.
    
   If you have a system that was joined to the domain via Centrify, there will be event data about access, privileges and other activities.

Now you have verified that your systems are streaming syslog data with Centrify Audit Trail information.

Identify Access and Privilege-related Metrics provided by Centrify

Centrify DirectControl provides access control and role-based privileged elevation; this component is called DirectAuthorize.  DirectAuthorize controls how users access the system and what commands they can run. The implementation of privilege elevation leverages Centrify-enhanced sudo.

Example 

The metrics that you'll track will depend in your objectives and in your maturity level.  For illustration purposes, let's track successful and unsuccessful access and privilege elevation in my Linux EC2 instances. After reviewing the Centrify Audit events guide, I identify the following events:

Access Control

PAM Authentication Granted:  These events are related to the UNIX framework;  the PAM Auth module is used by any PAM-enabled application.  This can be a catch-all for any app using it (e.g. OpenSSH server, Switch User (su), etc);  the Centrify Event Id is 24100.

Centrify SSHD Denied:  My EC2 instances are running Centrify-enhanced OpenSSH.  I'm interested in looking at this metric, especially on instances with public IPs because it may denote attempts to break-in or move laterally. The Event Id is 27101.

Privilege Elevation

Centrify dzdo Granted:  Indicates successful privilege elevation using Centrify-enhanced sudo.  Event id: 30000.

Centrify dzdo Denied :  Indicates denied privilege elevation using Centrify-enhanced sudo.  It may allow to identiy attempts for privilege abuse.  Event id: 30001.

Create the Filters and Assign a Metric

1. Go to your CloudWatch console: https://console.aws.amazon.com/cloudwatch/home
2. Click on Logs and select the radio buttion next to your log group (e.g. /var/log/messages)
3. Click Create Metric Filter
   • In filter pattern, type: centrifyEventID=24100
   • Press "Assign Metric" 
4. In Filter Name, type a unique name for the filter
5. In Metric details, create a new namespace (e.g. CentrifyAuditTrail) or browse for it if you already have it.
6. In Metric name, give it a descriptive metric.
   
7. Press Assign Metric.
8. Repeat the process for all the metrics you've identified.

Create a Dashboard

Before creating a dashboard, you may want to plan how to visualize the data.  In some instances it's useful to see the aggregate data (# of events), in others it's useful to see a trend (graphs overlapped with time).

Once you have thought of how to visualize the data, it's time to get started with your Dashboard. 

1. Go to your CloudWatch console: https://console.aws.amazon.com/cloudwatch/home
2. Click on Dashboards > Create Dashboard and give it a name, then press Create Dashboard
3. To add aggregated information, select the Number widget
4. Select your Namespace, Dimension and check the metric(s) to be measured
5. Go to the graphed metrics tab, and select the proper statistic and period  (e.g. sum and 1 day) and press Update Widget.
6. Once you have the Widget in the dashboard, adjust the size and label.

Repeat the process with the trend using with a line or stacked area.

Below is a simple dashboard that includes the metrics above.

Create an Alarm

A meaningful alarm could be based on a pattern outside expected behavior, an availability issue or another event (or aggregation of events) based on the risk that wants to be corrected.  This example is for illustration purposes only.

Example:  The threshold for attempted abuse of Centrify-enhanced sudo is 3 or more attempts within a 5 minute period, when this happens, an email should be triggered to the members of the Security Operations distribution list.

1. Go to your CloudWatch console: https://console.aws.amazon.com/cloudwatch/home
2. Click Browse Metrics and next to Centrify-dzdo-Denied, click the alarm icon.
3. In create alarm:
   Name:  Alarm-Abuse-dzdo
   Whenever: is equal or greater than 3 for 1 consecutive period
   Period: 5 minutes
   Statistic: Sum
4. Actions
   Whenever this alarm state is Alarm
   Create a new list (secops@your-domain.com)

Trigger the alarm

1. Sign-in to your Linux instance with homer
2. Type 'dzdo su - root' and press enter
3. You should get this message:
   

   [homer@cdctest2 ~]$ dzdo su -
   Sorry, user homer is not allowed to execute '/bin/su -' as root on cdctest2.\

4. Repeat 3 more times.  This should trigger the alarm.
   
5. Review the Dashboard.  After a few minutes, the alarm will return to normal and you'll be notified
   

Conclusion

We have only scratched the surface of the capabilities provided by AWS CloudWatch, however in the context of Identity and Access Management, the enrichment of security operations via logs, alerts and dashboards should be done via standard tools; otherwise if each tool duplicates these capabilities, then security operations won't know where to go first.  Centrify provides native plugins for Splunk, IBM QRadar and HP ArcSight.  These tools provide both operational data as well as like the following privilege command pie chart.

Related Article

Using CloudWatch to Monitor Windows systems running Centrify:  http://centrifying.blogspot.com/2017/04/using-aws-cloudwatch-to-monitor_24.html