Let's break a few things....

In this video series we explore how different Centrify products respond to failure scenarios.

Centrify UNIX, Linux Mac Clients - Cached Credentials

Setting up the Account Prevalidation GPO

How Privileged Elevation for Windows handles AD Failures

DirectAudit High-Availability (playlist)

Identity Service HA for Cloud Connectors

Implementing the Centrify's Account Prevalidation Capability on UNIX, Linux and OS X using Group Policies

Requirement
In case of an Active Directory catastrophic failure, a limited set of trusted administrators shall be able to log in to UNIX, Linux or OS X systems and perform the actions defined in their privileged management roles.

Scenario
Contoso has 3,000 servers and many of them on branch deployments and public clouds, there are no DCs in the branches.  WAN connectivity can be lost and a mechanism to allow trusted administrators to perform actions should be available.  It's highly unlikely the set of trusted admins have logged in to all systems. This is why Centrify implemented the account prevalidation feature.

Tools and Targets

• Group Policy Management
• Active Directory Users and Computers
• CLI tools:  adgpupdate, adgpresult, setspn & PowerShell(windows)
• Configuration files:  /etc/centrifydc/centrifydc.conf

 Planning

• Target Group:  Account prevalidation can be assigned to users or groups.  Ideally this would be assigned to groups so the management can be delegated to the Helpdesk, or front ended by an ITSM or Worfklow solution.
  This means that an AD Group should be requested.  Sample Name:   Centrify-Prevalidated-Admins
• Process: 
  Onboarding:  When senior UNIX, Linux or Mac OS X administrators are onboarded, the user template or form should be updated so they are added to this group by default.
  One-offs:   When critical projects are going on and stakeholders need to be added to this group, our ITSM tool will front end a workflow request that ends on a PowerShell action that adds (or removes) the user from the target group.
  Attestation:   This group does not have any additional rights, however it has to be attested as part of the disaster recovery plan.

Implementation
What you'll need

• Centrify Standard Edition or Centrify Identity Service Mac Edition  (licensed)
• A Windows system with Group Policy Management and the Centrify Group Policy Extensions (GPOE) installed
• An Active Directory group pre-populated with the users that will be prevalidated
• A GPO tied to the OU of the target UNIX, Linux or OS X systems.
• Rights to edit the GPO
• Centrified system in the OU that has the GPO linked to verify.

First, Gather information about the members of the group

From a Centrified System 
$ adquery group -A Centrify-Prevalidated-Users | grep members
members:centrifyimage.vms/Staff/IT/Diana Wirth

From Windows with PowerShell
PS > get-adgroupmember Centrify-Prevalidated-Users | Select-Object samaccountname

samaccountname
--------------
dwirth
rpimentel

Second, Configure the GPO

1. Open Group Policy Management
2. Create or edit the GPO in the corresponding forest/domain/OU.
   
3. In group Policy Editor, navigate to Computer Configuration > Policies > Centrify Settings(*) > DirectControl Settings(**) > Account Prevalidation.  In the right pane, double-click on "Specify allowed groups for prevalidation" select Enabled and populate the dialog box with the group predefined for this purpose and press OK and close Group Policy Editor.
   
   (*) If you don't see a Centrify Settings section, you don't have the Centrify GPOE installed.
   (**) If you don't see a "DirectControl Settings" folder, you need to add the template.

Third, Add the Preval SPN to the members of the group and commit the changes in the Centrified system

a) To add the SPN

# From Windows with the setspn utility
C:\> setspn -A preval/rpimentel rpimentel
Checking domain DC=centrifyimage,DC=vms

Registering ServicePrincipalNames for CN=Diana Wirth,OU=IT,OU=Staff,DC=centrifyimage,DC=vms
        preval/dwirth
Updated object

# From Windows using PowerShell
PS > Get-ADUser dwirth | Set-ADUser -ServicePrincipalNames @{Add="preval/dwirth"}

# From a Centrified System Using adedit
## Authenticate as someone who can edit the target object
$ kinit administrator
Password for administrator@CENTRIFYIMAGE.VMS:

# Open adedit, bind to domain and use the add_object_value function
$ adedit
>bind centrifyimage.vms
>package require ade_lib
1.0
> add_object_value "CN=Diana Wirth,OU=IT,OU=Staff,DC=centrifyimage,DC=vms" serviceprincipalname "preval/dwirth"
1
> exit

# Manually using Active Directory Users and Computers

1. Open ADUC
2. In the View menu, check "Advanced Features"
3. Open your target user > Attribute Editor > servicePrincipalName
4. Click add, and type  preval/<user>, press OK twice.

b) To verify the SPN was added succesfully
# from a Centrified system using CLI

$ adquery user dwirth --attribute servicePrincipalName
preval/dwirth

# from Windows using PowerShell
PS > Get-ADUser dwirth -prop ServicePrincipalNames | Select-Object Name, ServicePrincipalNames

Name                                  ServicePrincipalNames
----                                  ---------------------
Diana Wirth                           {preval/dwirth}

c) To commit the changes to a system for the purposes of testing

## flush the cache - only if you want this immediately (will happen automatically after cache flush interval + gpo refresh
$ dzdo adflush
DNS cache flushed successfully.
Authorization cache store flushed successfully.
GC and DC caches expired successfully.
DA name cache flushed successfully.
DA installation information cache flushed successfully.


## perform a group policy update - again, this happens on the GPO refresh.
$ adgpupdate
Refreshing Computer Policy...
Success
Refreshing User Policy...
User Policy disabled on this machine.

## verify that the GPO inserted changes in the config file
$ adgpresult | grep Centrify-Prevalidated-Users
        adclient.prevalidate.allow.groups = Centrify-Prevalidated-Users,

Notes:  This process can also be implemented with a Management tool like Chef, Puppet, Spacewalk, etc.  Ultimately the parameter that will be adjusted is the "adclient.prevalidate.allow.groups" in the /etc/centrifydc/centrifydc.conf file.  This can also be part of a master config file that is enforced across the board, but watch out for conflicts with GPOs.  You must pick a tool.

Verify and Test
 To verify prevalidation you'll need a recently centrified system OR a system that you know one of your test subjects has never logged in.  You will also have to switch the agent to offline mode.  There are two ways to do it:
a) If this is a VM, go to the hypervisor console, and disconnect the network; OR
b) Force the agent to go offline by adding all your DCs to the dns.block parameter.
E.g. if I have 2 DCs  (dc1.corp.contoso.com and dc2.corp.contoso.com), the parameter would look like this:

dns.block:  dc1.corp.contoso,com, dc2.corp.contoso.com

c) If you have console access, you can also disable the network interface momentarily.

This video explains the installation and testing process:


Adjustments
 There are several areas of improvement for this process:

Automation
A really cool improvement has to do with the automation of the prevalidation SPN.  For example, you can implement both in a Centrified UNIX/Linux system or with Windows PowerShell script that checks if new users are added to your target group, and automatically will add (or remove) the preval SPN.

Service Management and Approvals
If you have an ITSM (like ServiceNow) or an IdM with workflow/approvals, you can front-end this process by adding references and approvals to be able to answer these questions:

• Why is this user a member of this group?
• Who approved?
• What's the reference number

Labs - Web-Mobile SSO Planning Session I Videos

The videos below correspond to the first planning session for the Web-Mobile SSO use cases.

Lab 1 - Centrify User Suite Branding and Domains



Lab 2a - Cloud Connectors and Active Directory Integration



Lab 2b - Cloud Connectors and Active Directory Integration



Lab 2c - Exploring the AD Integration using Centrify Cloud Connectors



Lab 2d -  Understanding Login Suffixes

Labs: Testing the Availability Controls of Centrify for UNIX/Linux/Mac

In this Lab:

1. We'll use the dns.block parameter to use the domain controller fail-over capability.
2. We'll simulate a network failure on UBU1 and use the offline cache.
3. We'll attempt to kill the adclient process as a regular user
4. We will corrupt the /etc/nsswitch.conf and krb5.conf files 
5. We'll simulate an abnormal ending of the client process to invoke the watchdog process
6. We'll describe the domain controller telemetry process

Security Corner: Centrify UNIX Agent's Availability controls

Availability is the security principle that states that information should be available when needed and authentication mechanisms are required to be highly available since they are the door to the information that may be needed to make a business decision.

The Centrify agent for UNIX has a great advantage since it enables those platforms to integrate to Active Directory.  Active Directory was created with high-availability in mind and when properly implemented(*) provides:

• Replication:  Changes in the AD database are replicated to other domain controllers.  This process is called convergence.  Replication applies to LDAP objects and files.
• Multi-master:  Unlike NTDS 4.0 that relied on a Primary Domain Controller (PDC) role for write operations, in AD all DCs are writable.  There are some exceptions in what are called Flexible Domain Master Operations roles.  
• Sites and Services and DNS SRV records:  AD leverages DNS to provide the closest-to-client services;  any services that rely on these capabilities will be able to access the best connected service based on network location.

(*) Sadly, on the field (especially on lab environments) we see a lot of single DC environments and improper configuration of AD Sites and Services.  When we see the infamous "Default-First-Site-Name" displayed it decreases the credibility of the environment's maintainer.

One of the biggest concerns for any UNIX/Linux Systems Administrator is not having the ability to do their job because authentication scheme is not available;  as a matter of fact, that is why many of them are biased on using shared accounts (like root), because those accounts are reliably available.  Unfortunately this perpetuates the poor security practice of sharing those sensitive accounts.  The benefit with Centrify is that the mitigation for HA happens automatically without the need to re-target LDAP services or reconfigure krb5.conf files.

How does Centrify mitigate the Availability question?

Any true risk is mitigated by preventative, detective and corrective controls.  The controls deployed by Centrify are:

1. AD Sites and Services compatibility:  this means that the agent will pick an alternative domain controller based on the AD site topology.  (Corrective/Preventative)
2.  Performance Optimizations:  the agent performs its own telemetry calculations to determine if it's talking to the most optimal domain controller. (Preventative)
3. No AD available:  In case of a network-level failure (inability to connect to any DC) the agent provides the offline credential cache. (Corrective)
4. Abnormal termination:  the cdcwatch process is a watchdog that will spawn a new agent process in case of an abnormal termination.  (Corrective)
5. System file corruption:  Any changes in name server switch (NSS) (nsswitch.conf), Kerberos (krb5.conf) or pluggable authentication module (PAM) config files is monitored and rolled-back to proper operational mode if needed. (Corrective/Preventative)
6. Process Protections:  All agent-related processes (and the watchdog) are owned by root. (Preventative)
7. Logs and Core Dumps:  Centrify integrates with the syslog facility and provides its own core dumps in case of an abnormal termination. (Detective)

As you can see, Centrify's agent implements a high-level of controls to ensure high-availability.

When will a privileged system or local account be needed?

In two instances:

a) Normal termination of the Centrify agent process:  The agent has been stopped, therefore there's no communication with AD or the cache.  The authentication stack will continue on.  For example, on your /etc/nsswitch.conf you may see a line for users (or groups like this):

passwd         centrifydc   files

In case the adclient process is not available, any local account will be granted access.  This is the same under normal operations.

b) Single-user mode:  If the system abends and falls into that mode, the only account that can access the system is root.

Command Line Tool tips

The best command to troubleshoot the agent is adinfo.  Use it with the --test option perform connectivity tests.  Sample output:

george@suse1:~> adinfo --test

Domain Diagnostics

  Domain: corp.contoso.com

  Subnet site: CorpHQ

    DNS query for: _ldap._tcp.corp.contoso.com

    Found SRV records:

      dc1.corp.contoso.com:389

  Testing Active Directory connectivity:

    Domain Controller: dc1.corp.contoso.com

      ldap:      389/tcp - good

      ldap:      389/udp - good

      smb:       445/tcp - good

      kdc:        88/tcp - good

      kpasswd:   464/tcp - good

      ntp:       123/udp - good

Tips about conducting Disaster Recovery tests

Disaster recovery with Centrify for Servers can piggyback on the AD infrastructure and work performed for Windows domain members.  Unix/Linux systems just become another "customer of AD" this means:

• If you're performing a total (from scratch) AD recovery, AD DCs and DNS go first, then the Zone data has to be rebuilt.  UNIX/Linux systems rebuild happens in parallel, once ready, load the agent and join the zone; at that point instead of using the root account, you can switch to dzdo.
• If you're performing a partial or restore of existing systems, restore the AD infrastructure first;  you can work in parallel with UNIX/Linux systems  (credentials will be cached up to the moment of the backup or snapshot) and access will be offline - once AD is online the agent will go in connected mode.
• If you're testing with a disaster recovery site that goes online during tests your strategy may vary. In some outfits its not desirable that the agent fails over to the DR site; so using the dns.block parameter (or the Blacklist DNS DC hostnames GPO) to have those DCs blocked during production is desirable.  However, during the disaster, it should be scripted (or automated) that the DR Site DCs will be unblocked so they are eligible for fail-over.  Another variation of this test is to block the production DCs and only allow the DR DC (that is a crude/forced DR test).
• In larger environments, it's quite common that not all trusted sysadmins have had logged on to all target systems, that is why Centrify has the ability to pre-validate (or pre-cache) user credentials.  Prevalidation will be the subject of another posting.

Description of the dns.block parameter

This configuration parameter specifies which DCs should be filtered out of the pool of existing DCs for the domain.  This is useful when a DC is behind a firewall, has been decommissioned but there's a stalled object or for DR tests.

The parameter in the /etc/centrifydc/centrifydc.conf file works by specifying the directive followed by the FQDNs of the DCs in question separated by commas.

dns.block: dc1.corp.contoso.com,exp1.corp.contoso.com

If you prefer to use GPOs, the path is:  Computer Configuration > Policies > Centrify Settings > DirectControl Settings > Network and Cache Settings > Blacklist DNS DC hostnames group policy.

Labs