Troubleshooting: Understanding How Time Affects When Changes are Effective

The best way to summarize how time works in an AD environment that leverages Centrify agents for Unix, Linux and Mac OS X is to look at this formula:

Effective Changes = Σ(Provisioning, AD Replication, Cache Flush Interval)

Changes are additions, deletions or modifications of AD objects (LDAP).  This excludes real-time Kerberos transactions like authentication or password changes.

Provisioning

In Centrify for Unix/Linux, we call provisioning the action of assigning an existing AD principal (user or group) a UNIX identity (login,UID,GID,Home,GECOS,Shell).  In the case of users they also need a role to be able to log into systems; both actions can happen manually or automatically (via an Identity Management Solution that leverages the Centrify APIs, via the  Zone Provisioning Agent utility, or programmatically via Centrify PowerShell or Centrify adedit).
For example, in my environment I use ZPA and I assign the roles to AD groups so the management is simplified.  I've also nested my role-granting-group into the provisioning group.  This means that my modified equation is:

Effective Changes @ Contoso = AD Replication of Group Membership + ZPA Polling Interval + AD Replication of ZPA Provisioning + Cache Flush Interval)

Notice how my provisioning design has an impact on time.  I have potentially two actions that require AD replication and an agent that I've set up to poll every 15 minutes.

This means that a provisioning action can take as long as 15:30 from a provisioning + AD perspective in an intra-site scenario assuming (in the case of a user) that both the identity and the role were granted at the same time.

AD Replication

If you're relatively new to Active Directory and don't have an idea on how AD replication works, read these links and come back:

Basic Concepts: http://technet.microsoft.com/en-us/library/cc731537%28v=ws.10%29.aspx
How it works:  http://technet.microsoft.com/en-us/library/cc772726(v=ws.10).aspx 

In case you did not read the links above, the basic problem is that when you have a replicated database changes take time to propagate.  AD Sites (fast connected subnets) have internal (intra-site, shorter) and external (inter-site, longer) replication periods.

Intra-site replication is somewhat predictable.  A DC will notify to its nearest partner of a change within 15 seconds and this will cascade within a site.  Older versions (like Windows 2000) were set at 5 minutes.

Unfortunately in larger global environments AD inter-site replication times vary. It all depends how the AD team has tuned the environment based on the inter-network topology.   This is why to be an effective Centrify administrator you need to be in constant communication with your AD team. Replication affects availability for users and definitely affects your SLAs.  The best solution at a higher level is to use Microsoft's recommendations for AD replication in large environments and that they maintain current Subnet, Site and Domain Controller information.  This is very important.

That being said, there are things that you can do to make sure things happen faster. 

For example, if there is a new add/move or change and the target is a key server in a specific location, you can log into the server and find out what domain controller the server is currently talking to with the adinfo command (or adinfo --server).  If you're making the provisioning via ADUC, adedit or Access Manager, make sure you're talking to the same DC.  At that point you basically have eliminated AD time from the equation and you can issue an adflush when these changes are made.

If you're using The Windows PowerShell Centrify commandlets, you can use the echo %logonserver% command in a windows prompt to find out which domain controller you're currently talking to.

Centrify Agent Cache

We've talked about the cache in previous posts;  however, all you need to know that to improve performance and to provide high-availability the Centrify agent for Unix, Linux and Mac does not bother AD persistently to ask for changes;  this happens by default every hour.

Putting it all together

Looking back at my example, this means that in an intra-site scenario with two domain controllers like in my lab contoso environment, the length of time for an effective for a user change can be as long as 65:30 minutes. Because it takes up to 15 seconds for replication to happen on the provisioning/role granting action, up to 15 minutes for ZPA to poll, 15 additional seconds for the ZPA change to propagate and up to 60 minutes for a Centrified system to update its cache.

How to perform manual add/moves/changes in an effective matter

1. Determine the key system, and issue an adinfo command to determine the domain controller the agent is talking to.  (adinfo --server)
2. With that information, connect your ADUC or Access Manager consoles to the target DC.  (on AM, use the "Connect to remote forest" option; on ADUC use the "Change Domain Controller" option).
3. Perform your changes in AD (add/moves/changes)
4. Perform an adflush in the target system  (if it's a local override, you need to restart the agent)
5. Verify the changes with adquery, dzinfo, etc.

Note:  Flushing the cache (by interval or manual with adflush) is an expensive operation, I recommend that you keep the default cache flush interval of 3600 seconds (one hour) and try to establish a proper Service Level Agreement for these operations.

Troubleshooting: My brand new users aren't available or can't log in

Background

This is probably the most common task to troubleshoot is user's availability in a system.  Granting access to a brand new user (or users) to a system.  The user may or may not be available to the system or can't log in.

In this initial article, we will cover the mechanisms to minimize replication and techniques on how to rule out issues with the user's AD account.

What you've probably done so far:

1. You set up a Centrify zone or child zone with the corresponding user and group defaults
2. You've configured authorization (computer roles, UNIX rights, roles and role assignments)
3. Installed the Centrify agent and joined one or two Unix or Linux systems
4. You're trying to test a user or two but have no luck

Symptoms:

Unfortunately, your user does not show up in the system (adquery user does not show the user or users), possibly the user shows up, but can't log in;  Maybe the user shows up and can log on after a long time (undetermined time).

Remember:

By default in zone mode (unlike in Express mode) no users have access to the Unix, Linux (or Windows) systems that belong to the zone.  

Troubleshooting Checklist

We can use a two-category technique to troubleshoot this issue.  We can divide it into time and configuration items.  Time can be proactively managed and configurations can be verified and ruled out.

• Enough time should have passed for
• Any Identity Management tool to work (IdM solution, ZPA or program)
• AD replication to complete
• The cache flush interval complete (or the adflush command issued in a target system)
• In order to access a Unix/Linux system, a user needs an identity and a role
• To get an identity: the AD user has to be added to the zone to get a login, UID, GID, GECOS, Home and Shell based on the zone defaults to get an identity
• To get a role:  the role has to be assigned to a user directly or to a group that the user is a member of.
• The role needs to have at least the intended PAM logon right for the user to be allowed to log in.
• The role assignment has to be properly scoped (Zone, Child Zone, Computer Role or System level)
• The role has to be properly created:  logon options, hours, auditing, etc.
• The user's AD account should not be subject to any restrictions (logon, computer)
• The user's AD account should be usable (not expired, disabled or locked)
• The user is not listed in the /etc/centrifydc/user.ignore file
• The user principal has to be readable on the correct side of the one-way trust

Where to go now?

• First, try to understand how provisioning, AD replication and the agent's cache can affect when changes are effective.
• After you've ruled out time, now learn how to rule out the user's AD account.
• Learn how to spot configuration issues
• Learn how to use the addebug tool
• WIP...

Introduction to Troubleshooting

Troubleshooting

This is my attempt to start a troubleshooting topic for Centrify Server Suite.  I use the word 'attempt' because people have two attitudes when it comes to problem solving.

• The wrong way (e.g. brute force):  I was tasked to do something (get from point A to point B) and suddenly I had an issue, so I use anything I can (help, Google, etc) to find references to the problem, attempt many approaches until I fix it and I move on.
  We've all used the wrong way.  The problem lies in that the issue I had wasn't relevant to the bigger goal I want to accomplish, so I pay very little or no attention.  All I want is to get from A to B.
• The right way (e.g. understand the issue, find the root cause, solve it so it does not happen again):  Here you go to the documentation, try to understand how things work, determine the issue and implement a mechanism that stops it from happening again (even if it means rethinking how to get from A to B, or maybe determining that the journey is not from A to B, but from A to D).
  The right way is typically used when there's value to what we're doing or that we'll be stuck doing this work, so we might as well learn to do it well.

Troubleshooting Ground Rules

• We will try to use "the right way"
• We will summarize the problem just in case you want to solve it the "wrong way"
• There is no structure, the troubleshooting topics will be selected at random.
• We will try to keep things simple, but sometimes we'll go deep and show you some logs :)
• Same rules of the blog:  no Express, no Deployment Manager, no Classic zones.

What do you need to know to be an effective Centrify SME 

1. You need to know the basic security principles and security controls:  Centrify for Servers is a Security product it deals with Authentication, Authorization and Auditing.  You need to understand why you're doing what you're doing.
2. You need to understand Active Directory:   I can't stress this enough.  If you only know about UNIX, Linux or Mac OS X you are missing  60 to 70% of the knowledge needed to be an effective Centrify subject-matter-expert.  If you have any doubts about these terms:  LDAP, Kerberos, Group Policy, Sites and Services, Domain Controllers, DNS, Global Catalog, FSMA Roles, SRV Records, Replication, UPN, SPN, sAMAccountName, PKI, SysVol, Built-in groups, Domain Trusts, Forest, Domain, Site, etc.
3. Basic understanding of TCP/IP including DNS, TCP/UDP ports, Ephemeral Ports
4. Understand Kerberos (you don't need to become an expert).
5. Do not pretend that Windows does not exist.
6. If you are supporting Linux/UNIX, know at least a base level of what you're doing.
7. Know what the UNIX  Name Server Switch (NSS) and Pluggable Authentication Modules (PAM) frameworks are about.
8. For User Suite:  Read a Mac Admin Book, Read about federation standards (federation <> authentication), Office 365, etc.

My Advice

The best advice I can give you when it comes to troubleshooting is to always step back and ask yourself (without any implementation details) - "what is it that I want to accomplish?" "does it make sense?"

If you are trying to do something and you don't know WHY you're doing it - you are in trouble.
Also, recognizing competency gaps is important; if you don't have a grasp on the knowledge outlined above you may be in over your head;  the good thing is that we can always learn.

To find troubleshooting topics, follow the category bar.

Good Luck!!

News: Making Strides!

I'm very proud to see that Centrify has been named as a Visionary in the Identity as a Service (IdaaS) providers.

Gartner
https://www.gartner.com/doc/2756517/magic-quadrant-identity-access-management
Centrify
http://www.centrify.com/lp/gartner-magic-quadrant-for-idaas/

Exciting times!!

Labs: Testing the Availability Controls of Centrify for UNIX/Linux/Mac

In this Lab:

1. We'll use the dns.block parameter to use the domain controller fail-over capability.
2. We'll simulate a network failure on UBU1 and use the offline cache.
3. We'll attempt to kill the adclient process as a regular user
4. We will corrupt the /etc/nsswitch.conf and krb5.conf files 
5. We'll simulate an abnormal ending of the client process to invoke the watchdog process
6. We'll describe the domain controller telemetry process

Security Corner: Centrify UNIX Agent's Availability controls

Availability is the security principle that states that information should be available when needed and authentication mechanisms are required to be highly available since they are the door to the information that may be needed to make a business decision.

The Centrify agent for UNIX has a great advantage since it enables those platforms to integrate to Active Directory.  Active Directory was created with high-availability in mind and when properly implemented(*) provides:

• Replication:  Changes in the AD database are replicated to other domain controllers.  This process is called convergence.  Replication applies to LDAP objects and files.
• Multi-master:  Unlike NTDS 4.0 that relied on a Primary Domain Controller (PDC) role for write operations, in AD all DCs are writable.  There are some exceptions in what are called Flexible Domain Master Operations roles.  
• Sites and Services and DNS SRV records:  AD leverages DNS to provide the closest-to-client services;  any services that rely on these capabilities will be able to access the best connected service based on network location.

(*) Sadly, on the field (especially on lab environments) we see a lot of single DC environments and improper configuration of AD Sites and Services.  When we see the infamous "Default-First-Site-Name" displayed it decreases the credibility of the environment's maintainer.

One of the biggest concerns for any UNIX/Linux Systems Administrator is not having the ability to do their job because authentication scheme is not available;  as a matter of fact, that is why many of them are biased on using shared accounts (like root), because those accounts are reliably available.  Unfortunately this perpetuates the poor security practice of sharing those sensitive accounts.  The benefit with Centrify is that the mitigation for HA happens automatically without the need to re-target LDAP services or reconfigure krb5.conf files.

How does Centrify mitigate the Availability question?

Any true risk is mitigated by preventative, detective and corrective controls.  The controls deployed by Centrify are:

1. AD Sites and Services compatibility:  this means that the agent will pick an alternative domain controller based on the AD site topology.  (Corrective/Preventative)
2.  Performance Optimizations:  the agent performs its own telemetry calculations to determine if it's talking to the most optimal domain controller. (Preventative)
3. No AD available:  In case of a network-level failure (inability to connect to any DC) the agent provides the offline credential cache. (Corrective)
4. Abnormal termination:  the cdcwatch process is a watchdog that will spawn a new agent process in case of an abnormal termination.  (Corrective)
5. System file corruption:  Any changes in name server switch (NSS) (nsswitch.conf), Kerberos (krb5.conf) or pluggable authentication module (PAM) config files is monitored and rolled-back to proper operational mode if needed. (Corrective/Preventative)
6. Process Protections:  All agent-related processes (and the watchdog) are owned by root. (Preventative)
7. Logs and Core Dumps:  Centrify integrates with the syslog facility and provides its own core dumps in case of an abnormal termination. (Detective)

As you can see, Centrify's agent implements a high-level of controls to ensure high-availability.

When will a privileged system or local account be needed?

In two instances:

a) Normal termination of the Centrify agent process:  The agent has been stopped, therefore there's no communication with AD or the cache.  The authentication stack will continue on.  For example, on your /etc/nsswitch.conf you may see a line for users (or groups like this):

passwd         centrifydc   files

In case the adclient process is not available, any local account will be granted access.  This is the same under normal operations.

b) Single-user mode:  If the system abends and falls into that mode, the only account that can access the system is root.

Command Line Tool tips

The best command to troubleshoot the agent is adinfo.  Use it with the --test option perform connectivity tests.  Sample output:

george@suse1:~> adinfo --test

Domain Diagnostics

  Domain: corp.contoso.com

  Subnet site: CorpHQ

    DNS query for: _ldap._tcp.corp.contoso.com

    Found SRV records:

      dc1.corp.contoso.com:389

  Testing Active Directory connectivity:

    Domain Controller: dc1.corp.contoso.com

      ldap:      389/tcp - good

      ldap:      389/udp - good

      smb:       445/tcp - good

      kdc:        88/tcp - good

      kpasswd:   464/tcp - good

      ntp:       123/udp - good

Tips about conducting Disaster Recovery tests

Disaster recovery with Centrify for Servers can piggyback on the AD infrastructure and work performed for Windows domain members.  Unix/Linux systems just become another "customer of AD" this means:

• If you're performing a total (from scratch) AD recovery, AD DCs and DNS go first, then the Zone data has to be rebuilt.  UNIX/Linux systems rebuild happens in parallel, once ready, load the agent and join the zone; at that point instead of using the root account, you can switch to dzdo.
• If you're performing a partial or restore of existing systems, restore the AD infrastructure first;  you can work in parallel with UNIX/Linux systems  (credentials will be cached up to the moment of the backup or snapshot) and access will be offline - once AD is online the agent will go in connected mode.
• If you're testing with a disaster recovery site that goes online during tests your strategy may vary. In some outfits its not desirable that the agent fails over to the DR site; so using the dns.block parameter (or the Blacklist DNS DC hostnames GPO) to have those DCs blocked during production is desirable.  However, during the disaster, it should be scripted (or automated) that the DR Site DCs will be unblocked so they are eligible for fail-over.  Another variation of this test is to block the production DCs and only allow the DR DC (that is a crude/forced DR test).
• In larger environments, it's quite common that not all trusted sysadmins have had logged on to all target systems, that is why Centrify has the ability to pre-validate (or pre-cache) user credentials.  Prevalidation will be the subject of another posting.

Description of the dns.block parameter

This configuration parameter specifies which DCs should be filtered out of the pool of existing DCs for the domain.  This is useful when a DC is behind a firewall, has been decommissioned but there's a stalled object or for DR tests.

The parameter in the /etc/centrifydc/centrifydc.conf file works by specifying the directive followed by the FQDNs of the DCs in question separated by commas.

dns.block: dc1.corp.contoso.com,exp1.corp.contoso.com

If you prefer to use GPOs, the path is:  Computer Configuration > Policies > Centrify Settings > DirectControl Settings > Network and Cache Settings > Blacklist DNS DC hostnames group policy.

Labs

10 years of Centrify straight from the founders