Troubleshooting: Understanding How Time Affects When Changes are Effective

The best way to summarize how time works in an AD environment that leverages Centrify agents for Unix, Linux and Mac OS X is to look at this formula:

Effective Changes = Σ(Provisioning, AD Replication, Cache Flush Interval)

Changes are additions, deletions or modifications of AD objects (LDAP).  This excludes real-time Kerberos transactions like authentication or password changes.

Provisioning

In Centrify for Unix/Linux, we call provisioning the action of assigning an existing AD principal (user or group) a UNIX identity (login,UID,GID,Home,GECOS,Shell).  In the case of users they also need a role to be able to log into systems; both actions can happen manually or automatically (via an Identity Management Solution that leverages the Centrify APIs, via the  Zone Provisioning Agent utility, or programmatically via Centrify PowerShell or Centrify adedit).
For example, in my environment I use ZPA and I assign the roles to AD groups so the management is simplified.  I've also nested my role-granting-group into the provisioning group.  This means that my modified equation is:

Effective Changes @ Contoso = AD Replication of Group Membership + ZPA Polling Interval + AD Replication of ZPA Provisioning + Cache Flush Interval)

Notice how my provisioning design has an impact on time.  I have potentially two actions that require AD replication and an agent that I've set up to poll every 15 minutes.

This means that a provisioning action can take as long as 15:30 from a provisioning + AD perspective in an intra-site scenario assuming (in the case of a user) that both the identity and the role were granted at the same time.

AD Replication

If you're relatively new to Active Directory and don't have an idea on how AD replication works, read these links and come back:

Basic Concepts: http://technet.microsoft.com/en-us/library/cc731537%28v=ws.10%29.aspx
How it works:  http://technet.microsoft.com/en-us/library/cc772726(v=ws.10).aspx 

In case you did not read the links above, the basic problem is that when you have a replicated database changes take time to propagate.  AD Sites (fast connected subnets) have internal (intra-site, shorter) and external (inter-site, longer) replication periods.

Intra-site replication is somewhat predictable.  A DC will notify to its nearest partner of a change within 15 seconds and this will cascade within a site.  Older versions (like Windows 2000) were set at 5 minutes.

Unfortunately in larger global environments AD inter-site replication times vary. It all depends how the AD team has tuned the environment based on the inter-network topology.   This is why to be an effective Centrify administrator you need to be in constant communication with your AD team. Replication affects availability for users and definitely affects your SLAs.  The best solution at a higher level is to use Microsoft's recommendations for AD replication in large environments and that they maintain current Subnet, Site and Domain Controller information.  This is very important.

That being said, there are things that you can do to make sure things happen faster. 

For example, if there is a new add/move or change and the target is a key server in a specific location, you can log into the server and find out what domain controller the server is currently talking to with the adinfo command (or adinfo --server).  If you're making the provisioning via ADUC, adedit or Access Manager, make sure you're talking to the same DC.  At that point you basically have eliminated AD time from the equation and you can issue an adflush when these changes are made.

If you're using The Windows PowerShell Centrify commandlets, you can use the echo %logonserver% command in a windows prompt to find out which domain controller you're currently talking to.

Centrify Agent Cache

We've talked about the cache in previous posts;  however, all you need to know that to improve performance and to provide high-availability the Centrify agent for Unix, Linux and Mac does not bother AD persistently to ask for changes;  this happens by default every hour.

Putting it all together

Looking back at my example, this means that in an intra-site scenario with two domain controllers like in my lab contoso environment, the length of time for an effective for a user change can be as long as 65:30 minutes. Because it takes up to 15 seconds for replication to happen on the provisioning/role granting action, up to 15 minutes for ZPA to poll, 15 additional seconds for the ZPA change to propagate and up to 60 minutes for a Centrified system to update its cache.

How to perform manual add/moves/changes in an effective matter

1. Determine the key system, and issue an adinfo command to determine the domain controller the agent is talking to.  (adinfo --server)
2. With that information, connect your ADUC or Access Manager consoles to the target DC.  (on AM, use the "Connect to remote forest" option; on ADUC use the "Change Domain Controller" option).
3. Perform your changes in AD (add/moves/changes)
4. Perform an adflush in the target system  (if it's a local override, you need to restart the agent)
5. Verify the changes with adquery, dzinfo, etc.

Note:  Flushing the cache (by interval or manual with adflush) is an expensive operation, I recommend that you keep the default cache flush interval of 3600 seconds (one hour) and try to establish a proper Service Level Agreement for these operations.

Basics: The Centrify Agent Cache

Background

In a previous post we described the Centrify agent (adclient) and we mentioned that the cache exists for two objectives:  performance and offline access (availability control in case AD is not accessible). Understanding the cache allows any system administrator to be more effective at using the solution.   The client caches DNS, object and authorization information.

The NSCD service

NCSD stands for name server cache daemon.  It is the best method to retrieve cached information since it's implemented and optimized for each version of UNIX/Linux.  As a basic computer science principle, retrieving items from memory is always preferable than retrieving from permanent storage. The rule of thumb is to always implement NSCD in your systems.  This is a very important design consideration.

The best thing you can do to improve performance of any application that uses Name Server Switch is to implement NSCD.  This applies to Centrify for Unix/Linux.

The Object cache

This cache stores the UNIX-enabled users and managed groups that belong to the client's zone. It's implemented as a database that is stored in the /var/centrifydc folder.  It is used so the client does not have to go to Active Directory for each query and provides support for offline access.

Object expiration and refresh intervals

When objects are retrieved they are tagged with a time-stamp and some of the AD information is retrieved (mainly the update sequence number), while the refresh interval is current, the object will be returned "as is" (no queries to AD are made), if the refresh interval expires, then the object's USN is compared to 

the one in the cache, only if the numbers are different the object will be refreshed, otherwise the time-stamp

is renewed.

The notable exception to this logic is when an end-user attempts to log in; at this point (among other things)
the validity of the account is checked (e.g. is the account disabled, locked, expired, allowed to log in at the time, etc) those checks are performed in real time.
Although there are more parameters to fine-tune the cache, the adclient.cache.expires interval governs when objects are expired. Traditionally, by default is set to 1 hour.

The Authorization Cache

This cache is used for the purposes of authorization (roles, rights etc) and it's separate from the object cache.
It uses the AD azman schema and Centrify algorithms to determine if there are changes to the group memberships (or principals) that have roles and rights tied to them. It uses the USN and the adclient.azman.refresh.interval (set to 30 minutes by default).

Depending on the complexity of the AD infrastructure, the operation can be expensive (in terms of LDAP queries) this is because corporate domains are complex and in certain scenarios it's possible that groups are merged.  Since the memberships are fully enumerated, this is something to think about.

How the Authorization Cache affects the User's ability to log in

Notice that before we have mentioned that for a user to log in they require two things: A UNIX identity, and a role that allows them to log-in. One component relates to identification, the other relates to authorization.
This means that it's possible to have a proper login, UID, GID, etc; but still be unknown to a system because the user does not have a role assigned or the role that is assigned doesn't have a PAM login right.

Note that users can be "listed" in a system.
There are other types of cache, but the two most relevant are the Object and Authorization caches.

What does this all mean?

It means that there's a fine balance to be understood in order to provide performance and high-availability. It also means that there are planning implications to the workings of the agent.  Questions like:

• What is this system used for?
  Very different settings for a shared developer's system than a slave node in a number-crunching system.
• What are the current provisioning/de-provisioning SLAs?
  This opens a great question.  The SLA dictates a lot of settings.  For example in a scheme in which users are provisioned using an IdM solution from an HR system, and there are workflows involved to approve access to UNIX/Linux systems, all components have to be accounted for, Including Active Directory replication intervals.

Adfush described

Adflush is a command line utility that allows system administrators to manipulate the cache.  It shouldn't be used without understanding how the cache works.  

usage: adflush [options]

options:

  -f, --force             flush cache even adclient is in disconnected mode

  -a, --auth               flush cached authorization data

  -d, --dns                flush the adclient dns cache and DC locator cache

  -e, --expire             expire everything in the DC and GC object cache

  -o, --objects            flush only the DC and GC object cache

  -t, --trusts             rediscover the trusted domains

  -b, --bindings           force refresh binding

  -v, --version            print version information

  -V, --verbose            print debugging information

  -h, --help               print this help information and exit.

The type of adflush can have implications from slowing things down a bit, to consuming resources to actually eliminating the ability to log-in offline.  This is important to know for roaming users of Macintosh platforms.  A user that runs a forced adflush while at home will have to wait to get back to the office to log in!!!

Basic Concepts: The Centrify Agent

Centrify's Active Directory Client
Typically known as DirectControl (or adclient) is Centrify's Active Directory integration service.  It leverages AD LDAP, Kerberos and Group Policy;  with over 10 years of maturity, the agent has evolved to support diverse platforms and has provides super user privilege management for UNIX/Linux and Windows platforms.

 

 The architecture of the client in UNIX is as follows:

• NSS Module:  Uses the NSS facility to present AD as a source of of identity for users (passwd), and groups.
• PAM Module:  The authentication against AD is implemented as a PAM module.  Centrify implements authentication, account, session and password modules.
• Kerberos Libraries:  MIT Kerberos compiled libraries with support for Microsoft's Kerberos implementation.  The location of these tools is /usr/share/centrifydc/kerberos/bin.
• Group Policy Engine:  Processes group policies from AD in the Unix/Linux and Mac platforms
• Centrify-enhanced sudo:  A version of sudo that leverages Roles and Rights defined in AD with Centrify.
• Command-line Tools:  Centrify has implemented command for the agent (ad commands), for privilege management (dz commands) and for auditing (da commands). In addition, there are modules for PowerShell for the DirectManage components, as well as an SDK.
• LDAP and NIS Proxies:  These proxies present AD information to clients that can't have the agent installed (like filers, appliances, or legacy systems)
• Offline credential cache:  Provides high-performance (by not requiring a persistent LDAP connections to AD) and high-availability (in case AD is not available or there's a network failure)
• External Modules:  provides SSO facilities for Apache, Tomcat, Websphere, Weblogic Java2EE application servers, SAP (GUI and Netweaver) and DB2
• Watchdog Process:  provides a backup mechanism for recovery and diagnostics in the case of a daemon failure.

Communicating with Active Directory

To talk to domain controllers, the Centrify agent uses the following communication ports:

Port	Description
TCP/UDP 389	LDAP
TCP 3268	Global Catalog Search
TCP 88	Kerberos TGT
TCP 464	Kerberos password changes (passwd, adpasswd)
UDP123	Optional: Network Time Protocol (NTP)(*)
TCP 53	DNS query for A and SRV records
TCP 445	Optional:  SMB to read Group Policies
Ephemeral Ports	Required for communication

Communications between the Centrify client are mutually authenticated and encrypted, just like when Windows clients communicate with DCs.

For more information, see this Technet article.

(*) By default, the Centrify AD client will make UNIX, Linux or Mac sync time with Active Directory Domain Controllers;  you can use any NTP service, however you need to make sure that it's within 5 minutes from the DC (that acts as a Kerberos KDC).  This is a Kerberos requirement to protect against replay attacks.