Troubleshooting: Understanding How Time Affects When Changes are Effective

The best way to summarize how time works in an AD environment that leverages Centrify agents for Unix, Linux and Mac OS X is to look at this formula:

Effective Changes = Σ(Provisioning, AD Replication, Cache Flush Interval)

Changes are additions, deletions or modifications of AD objects (LDAP).  This excludes real-time Kerberos transactions like authentication or password changes.

Provisioning

In Centrify for Unix/Linux, we call provisioning the action of assigning an existing AD principal (user or group) a UNIX identity (login,UID,GID,Home,GECOS,Shell).  In the case of users they also need a role to be able to log into systems; both actions can happen manually or automatically (via an Identity Management Solution that leverages the Centrify APIs, via the  Zone Provisioning Agent utility, or programmatically via Centrify PowerShell or Centrify adedit).
For example, in my environment I use ZPA and I assign the roles to AD groups so the management is simplified.  I've also nested my role-granting-group into the provisioning group.  This means that my modified equation is:

Effective Changes @ Contoso = AD Replication of Group Membership + ZPA Polling Interval + AD Replication of ZPA Provisioning + Cache Flush Interval)

Notice how my provisioning design has an impact on time.  I have potentially two actions that require AD replication and an agent that I've set up to poll every 15 minutes.

This means that a provisioning action can take as long as 15:30 from a provisioning + AD perspective in an intra-site scenario assuming (in the case of a user) that both the identity and the role were granted at the same time.

AD Replication

If you're relatively new to Active Directory and don't have an idea on how AD replication works, read these links and come back:

Basic Concepts: http://technet.microsoft.com/en-us/library/cc731537%28v=ws.10%29.aspx
How it works:  http://technet.microsoft.com/en-us/library/cc772726(v=ws.10).aspx 

In case you did not read the links above, the basic problem is that when you have a replicated database changes take time to propagate.  AD Sites (fast connected subnets) have internal (intra-site, shorter) and external (inter-site, longer) replication periods.

Intra-site replication is somewhat predictable.  A DC will notify to its nearest partner of a change within 15 seconds and this will cascade within a site.  Older versions (like Windows 2000) were set at 5 minutes.

Unfortunately in larger global environments AD inter-site replication times vary. It all depends how the AD team has tuned the environment based on the inter-network topology.   This is why to be an effective Centrify administrator you need to be in constant communication with your AD team. Replication affects availability for users and definitely affects your SLAs.  The best solution at a higher level is to use Microsoft's recommendations for AD replication in large environments and that they maintain current Subnet, Site and Domain Controller information.  This is very important.

That being said, there are things that you can do to make sure things happen faster. 

For example, if there is a new add/move or change and the target is a key server in a specific location, you can log into the server and find out what domain controller the server is currently talking to with the adinfo command (or adinfo --server).  If you're making the provisioning via ADUC, adedit or Access Manager, make sure you're talking to the same DC.  At that point you basically have eliminated AD time from the equation and you can issue an adflush when these changes are made.

If you're using The Windows PowerShell Centrify commandlets, you can use the echo %logonserver% command in a windows prompt to find out which domain controller you're currently talking to.

Centrify Agent Cache

We've talked about the cache in previous posts;  however, all you need to know that to improve performance and to provide high-availability the Centrify agent for Unix, Linux and Mac does not bother AD persistently to ask for changes;  this happens by default every hour.

Putting it all together

Looking back at my example, this means that in an intra-site scenario with two domain controllers like in my lab contoso environment, the length of time for an effective for a user change can be as long as 65:30 minutes. Because it takes up to 15 seconds for replication to happen on the provisioning/role granting action, up to 15 minutes for ZPA to poll, 15 additional seconds for the ZPA change to propagate and up to 60 minutes for a Centrified system to update its cache.

How to perform manual add/moves/changes in an effective matter

1. Determine the key system, and issue an adinfo command to determine the domain controller the agent is talking to.  (adinfo --server)
2. With that information, connect your ADUC or Access Manager consoles to the target DC.  (on AM, use the "Connect to remote forest" option; on ADUC use the "Change Domain Controller" option).
3. Perform your changes in AD (add/moves/changes)
4. Perform an adflush in the target system  (if it's a local override, you need to restart the agent)
5. Verify the changes with adquery, dzinfo, etc.

Note:  Flushing the cache (by interval or manual with adflush) is an expensive operation, I recommend that you keep the default cache flush interval of 3600 seconds (one hour) and try to establish a proper Service Level Agreement for these operations.

Labs: Centrify for Servers Enterprise Edition 2014 - Speedy Evaluation

I've posted this video playlist:



It covers Centrify Enterprise Edition 2014 and provides a full product evaluation experience for security access controls (Authentication, Authorization, Auditing).  Here are the evaluation objectives.

Category 1:  AD Integration and Authentication - Standard Edition

1. Does not require any schema extensions or software loaded in DCs.
2. Not all AD users have access to UNIX/Linux systems by default
3. Solution uses Standards and Frameworks (LDAP, Kerberos, PAM, NSS)
4. Solution does not synchronize identities or passwords
5. UNIX-enabled AD users can be controlled from AD  (enabling/desabling, logon hours, etc.)
6. Integrating UNIX/Linux systems can be performed from the GUI or command line.
7. AD Security policy is enforced in UNIX/Linux Platforms.
8. Multiple UNIX identities can be assigned to the same AD User
9. UNIX secondary group memberships can be managed from Active Directory as well
10. Authentication can be streamlined with SSO mechanisms.
11. Provisioning/Deprovisioning of UNIX users and groups can be automated.
12. Filer integration (Samba or NFS) - identity consistency for files and folders
13. High-availability:  Users should have access if AD is not available.

Category 2:  Access and Privilege Management - Standard Edition

1. Systems can be grouped using different criteria to enforce access.
2. Access can be limited based on groups of systems defined above (role-based access control)
3. Roles can be defined to control how the user signs into the system and the privileges that can be granted (Role-based access privileges)
4. Provides mechanisms to verify who used their privileges
5. Provides the ability to enforce separation of duties (operations vs. governance)
6. It's intuitive to find out who has access to what system and what can they do

Category 3:  Advanced Auditing - Enterprise Edition

1. Provides a detailed view of user's activities (user, system accessed, duration, client computer, activities)
2. Provides a contextual view of the actions performed (replay capability)
3. Provides the ability to search sessions based on queries.
4. Provides the ability to view when privileges are used across systems

Environment

Lab 16: Configuring ZPA for Automatic UNIX Group Provisioning

Background

Posting:  http://centrifying.blogspot.com/2014/03/basics-automatic-unix-profile.html
Automatic Provisioning for Users: http://centrifying.blogspot.com/2014/01/labs-centrify-zone-provisioning-agent.html

Part 1

1. We configure grant the proper rights to the Centrify ZPA AD service account
2. We create the Source Group
3. We create a test group (UNIX-HQ-webadmin) and nest it to the source group.
4. We test and verify that automatic provisioning is working

Part 2

1. We verify that add/moves/changes are working as expected.
2. We modify the UNIX group profile with account overrides
3. We modify the UNIX group provisioning to automatically set the unix name and GID based on our needs.
4. We add the dbaadmin group.
5. We verify that regardless of the membership, only the users who have been granted rights to the system are the ones in the NSS group.

Basics: Configuring the Centrify Zone Provisioning Service for Automatic Secondary UNIX group provisioning

Background

In a previous posting we discussed how to implement automatic provisioning of UNIX user identities using Centrify's Zone Provisioning Agent (ZPA).
As part of BP#2, one of the key requirements was to implement a provisioning model for secondary UNIX groups and to make it resemble the existing one for users.
This capability consists in the automatic creation of the secondary UNIX group as well as the lifecycle maintenance of members (add/moves/changes).

All the principles applied in the basic posing about ZPA apply here, but in the context of UNIX secondary groups.  Let's apply the Plan-Do-Check-Adjust methodology.

Planning for Secondary Group Provisioning

Is the secondary group needed
The first place to start is to ask "Why is this secondary group required?" -  remember that with Centrify there is no need to have these groups for Access or Privilege purposes, Zones and RBAC take care of all that. In these exercises, the typical answer "Because that's the way is done" is not a good answer so you may have dig deeper.

Naming Conventions
The same rules of UNIX systems apply here as well, but they are a bit more stringent;  the 8 character limit for users is platform-dependent;  the UNIX group case, it may be more of a hard requirement.

Active Directory
Basically the way UNIX Groups work in Centrify is as follows;  there is a security group in AD for each secondary group in UNIX;  the memberships are handled from AD (the process consolidator and time saver), however, governance and processes have to be taken into account.  Questions such as:

1. How are AD Security group are requested today?
2. How is the membership lifecyle managed (add/moves/changes)?
3. Are group memberships subject to any attestation process?

Need to be understood and agreed upon.  

Group Merging

Group Merging is the ability to mix members from other Name Server Switch groups (like /etc/group) with UNIX Enabled AD groups. For example, if an application requires the enumeration of all the members from the software group (UID 200) and the application uses local users (e.g. like the db2inst account) and AD users (like jessie.matthews), then this feature needs to be implemented as well.

Implementing (Do) Secondary Group Provisioning

The process is as follows:

1. Make sure the service account used for ZPA has the proper delegation to add/remove/modify group profiles.  This is done through the Delegation Wizard of the zone.
   
2. An Active Directory "Source" Group is created.
3. ZPA is configured to look at that group as the source group.  For consistency, this group is stored in the OU designated for provisioning 
4. The rules for GID are set, the options are:

• An unique value generated from the group's System ID (SID)
• The RFC 2307 attribute (has to be populated in the object manually or programatically)
• Using the defaults set in the zone (Group Defaults tab)
• Using Apple's scheme (this is new for version 2014)
  
5. The naming rules are set (truncation, character sets, prefixes, etc)
6. Once this is set, ZPA is ready to provision UNIX groups.

Verifying the Implementation (Check)

Verifying group creation:

1. Restart ZPA (so the new zone provisioning configuration is processed)
2. Create an AD Security group.
3. Nest the group (make it a member) of the source group in the previous step.
4. Wait the ZPA refresh cycle or restart the service.
5. Verify that the new group exists under Zone > UNIX Data > Groups.
6. Verify that the group exists in the adequate UNIX systems.  There are several ways:
   getent group | grep <unix name of group>
   adquery group | grep <unix name of group>
7. To verify the group's deletion, just remove the AD group from the source group and cycle ZPA;  the UNIX group will be removed from the zone.

Verifying Add/Moves or Changes

1. Add a user to the previously provisioned group
   Note:  this user needs to be at least listed in the system, otherwise he/she won't show. For example, if the user has a UNIX identity, but only has access to web servers, the user will not show in the Database Servers by virtue of the access rules enforced by Centrify.
2. Wait the ZPA refresh cycle or restart the service.
3. Verify that the user's UNIX name is in the list of users for that group.
   There are several ways:
   getent group | grep <unix name of group>
   adquery group | grep <unix name of group>
4. Remove the user from the AD group (go to step 2) and in step 3 verify that the user is gone.

Performing Adjustments

Adjustments may vary depending on the results.  They can be:

• Adjusting how the GID is generated (or reviewing the provisioning process)
• Adjusting how the UNIX name of the group is generated.
• Adjusting group merging options.

Labs: Centrify Zone Provisioning Agent - Installation, Configuration and Testing

Background

ZPA is a Centrify utility that can be used for automatic provisioning.  It can UNIX-enable Active Directory users and groups.  Read this post to learn about ZPA:
http://centrifying.blogspot.com/2014/01/basics-automatic-unix-profile.html

Installation

Configuration and Testing

Errata

- Identity overrides can happen at the zone level, child zone level and system level.
- It's recommended that you install in the Centrify common components (for troubleshooting purposes)
- The UNIX Super Users group also needs to be nested  in the provisioning group, otherwise they will lose their UNIX profile.
- The ZPA account needs to be made a local administrator in APP1 (if you want the troubleshooting components to work)
- The Provisioning tab under zone properties needs to be installed on CLIENT1 using ZPA setup.

Basics: Automatic UNIX Profile Provisioning Using Centrify's Zone Provisioning Agent

Background

In a previous posts we discussed what's in a UNIX profile, we have also discussed the concepts behind Centrify Zones, how Active Directory users can be UNIX-enabled using Centrify and what are some of the considerations and strategies for UNIX identities;  the problem has been that our labs have been focused in manual Identity manipulation.

The reason for this is simple:  understanding the basics is establishes the best foundation to address complex issues; but since we have gone over man manual scenarios, in this posting we will discuss automatic provisioning with the Zone Provisioning Agent (ZPA) utility.  The rest of the labs will use ZPA with a rationalized namespace.

Why Exceptions are Bad

In IT infrastructure, Security as well as in Identity and Access Management, designers need to strive for standardization and simplicity; sadly the reality is that organizations are complex today because problems are addressed in a reactive matter and rarely the root cause of issues are addressed, instead, organizations throw technology and resources at the problem.

Getting to a standard Identity Provisioning process is hard and the most challenging part is the ability to make the process standard and repeatable enough that exceptions are almost eliminated.   Exceptions introduce overhead and unnecessary risk; any system administrator with a certain degree of experience knows this and the sad thing is that some of them have figured out that complexity keeps them employed and enjoying the status quo.

Exceptions = More complexity = More stress = More work = More cost = Lower Satisfaction

In addition, to be completely fair, not all organizations can afford to completely aim and then shoot.  This is why solutions need to allow a certain degree of flexibility.

Centrify Zone Provisioning Agent

The Zone Provisioning Agent (or ZPA) is a utility included with Centrify Standard Edition that allows the automatic provisioning of UNIX identities leveraging Active Directory Security groups.
Yes, we get to reuse the same process again.
ZPA waits a preset number of minutes and searches in Active Directory for provisioning groups, and based on the zone defaults, will automatically UNIX-enable a user.

ZPA is implemented as a Windows Service or it can be run as a command utility called zoneupdate.exe.
Zoneupdate provides an interface to work with scripting or Identity Management solutions.

Like any other solution, ZPA has to subscribe to the Plan-Do-Check-Model.

Planning for ZPA

• Understand the impact of ZPA - once a zone has been enabled for automatic provisioning, AD users can't be UNIX-enabled manually, overrides are still possible at the zone, child zone and system levels.  The unavailability of manual provisioning, stresses that the process has to be clean of major exceptions.
• Determine how you're handling the Unix Name (RFC 2307 uid attribute).  This is important because some UNIXes need to be enabled for long name support and you may have naming conventions to consider.
• ZPA allows to handle any exceptions on the settings for the UNIX name.
  
• ZPA can use any attribute stored in the user object as the RFC2307 uid field, therefore if you have any scripts, IdM solution, etc that make transformations and write them into this field, they can be leveraged by zpa.  The default is the AD username:  ${u:sAMAccountName}
• ZPA requires a master provisioning AD security group per zone;  this means that we need to create and populate an AD Security group with the users that require provisioning.
  Sample Design (Nested Groups):  
  You can leverage Active Directory group nesting to kill two birds with one stone.   For example:  Jeremy is a member of the UNIX Database Servers Users group;  since this group grants the role in the appropriate number of systems, we can safely assume that he requires a UNIX profile, so if we have a Provisioning group called UNIX All Users, all we need to do is nest the role-granting AD group in to the provisioning group.
  
  
  This design simplifies operations because either manually, programmatically or by virtue of using an Identity Management tool, adding or removing the user to the role-granting group, has the effect of provisioning or deprovisioning access.
• ZPA high-availability:  All services need to be planned with HA in mind.  Running multiple ZPAs is not uncommon.
• Planning for Exceptions:  In this model, Centrify allows for identity overrides at the child or system levels.
• Planning for Migrations from existing Manual to ZPA zones:  In this case, the brief moment in which the accounts won't be available has to be accounted for.
  All users will lose their UNIX identities until the next polling interval kicks-in.  If well planned, this should be negligible in a medium to small environment.

Implementing ZPA (Do)

To Implement ZPA, at a high level:

1. Create the Provisioning group and populate it with the appropriate users and/or groups.
   If using the model outlined above, nest the role-granting groups into the provisioning group.
2. Request an Active Directory service account to run ZPA.
3. Delegate the proper permissions for Users (or Groups) for ZPA to be able to perform the functions desired.
4. Install ZPA in a highly-available system.
   Note:  If it hasn't been done, your existing Access Manager consoles have to be extended with ZPA setup to have the Provisioning tab enabled.
   
5. In Access Manager Configure the zone for automatic provisioning (if this zone had users, the UNIX-identities will be deleted)
6. Configure the account and the polling interval for ZPA
   (You may need to make sure that the service account has the proper log on locally rights)
7. When the service starts, all identities will be provisioned.

Verifying the Implementation (Check)

1. In Access Manager, verify that all UNIX Identities are accounted for.
2. In the UNIX systems, make sure that after an adflush all users for the corresponding system are accounted for.
3. Add a user to one of the role-granting groups.  Trigger a polling interval.  Verify that the user has a UNIX identity.
4. Remove the user from the role-granting group. Trigger a polling interval.  Verify that the user has lost his UNIX identity.

Adjusting the ZPA Implementation

There are multiple reasons to adjust a ZPA implementation, but the most common are:

• Provisioning SLAs:  Provisioning is composed of multiple parts:  from user creation or provisioning, to workflow/approvals, AD-replication, cache-flush intervals and now ZPA.  The polling interval may need to be adjusted for Service Level Agreements.
• Performance:  Network, domain controller availability, etc.
• Availability:  Redundancy, etc.

Videos on setup here: http://centrifying.blogspot.com/2014/01/labs-centrify-zone-provisioning-agent.html

Zoneupdate Help

Usage: ZONEUPDATE [options] zone1 [zone2 zone3...] Options

/z|SourceZone:Path  
Copy user attributes from the source zone (specified in canonical name format, for classic zone only).

/d|Domain:DomainName
The FQDN of the target domain.

/dc|DomainController:DomainControllerNameThe target domain controller to connect to.

/u|UserSource:ADGroup[@domain]

The AD group used to populate the users in the zone.

/g|GroupSource:ADGroup[@domain]The AD group used to populate the groups inthe zone.

/uu|UserUid:<Option>
The method to generate the UID for a user profile. Option can be one of the following values. Auto is the default if this option is not specified.

/us|UserShell:<Option>

The method to generate the UNIX shell for a user profile.

Option can be one of the following values. 

/uh|UserHomeDirectory:<Option>

The method to generate the home directory for a user profile.

Option can be one of the following values. 

/uc|UserGecos:<Option> (For hierarchical zone only)

The method to generate the Gecos in a user profile.

                     

/gg|GroupGid:<Option>

The method to generate the GID for a group profile.

/gn|GroupName:<Option>

The method to generate the UNIX name for a group profile.

Option can be one of the following values. SamAccountName

is the default if this option is not specified.

                     

Methods:  

SamAccountName is the default if this option is not specified (for login, UID, GID).

Auto is the default if this option is not specified.

Auto            - Auto-generate from the AD group's SID.

RFC2307         - Group's AD RFC 2307 gidNumber attribute.

ZoneDefault     - The GID of this group in the default value zone for the target zone. Use the next GID of the target zone if the default value zone is not available.

SourceZone      - The GID of this group in the source zone.

(for classic zone only)

<Empty>         - Leave empty option for undefined GID.

(for hierarchical zone only)

                     

/v|Verbose           Verbose output messages.

/p|Preview           Preview only. No provisioning changes will be made.

/l|Log:<Level>       

Turn on the log file. The log file will be created in'C:\Documents and Settings\<CurrentUser>\Application Data\Centrify\Zone Provisioning Agent\Log'

Log level can be Error, Warning, Information or Verbose.

Example:

    zoneupdate /u:"Domain Users" /g:"Domain Users" /d:centrify.dev zone1 zone2