Labs: Automating joins/removals with Kerberos Keytabs

This is the companion video to this post:  http://centrifying.blogspot.com/2014/08/using-kerberos-keytabs-and-centrify.html

Using Active Directory and Centrify to Accelerate your Linux-based Hadoop Big Data Deployments

Update

Updated whitepapers from August 2015:

• Cloudera (5.4.3 and Centrify 2015.1): http://goo.gl/zWoBZE
• Hortonworks: http://goo.gl/uzGPGG
• MapR (4.1 and Centrify 2015.1): http://goo.gl/0uLiHF

This playlist also explains the challenges at the OS Level:


*********************************************************************************
As of February 2015 Centrify is gearing-up to release Centrify Suite 2015, with it, enhancements specific to Hadoop deployments are being released.  To complement these efforts, integration guides for Cloudera, Hortonworks and MapR are being released, here are the first two:

Centrify Identity and Access Management for MapR
Centrify Identity and Access Management for Cloudera

Centrify Identity and Access Management for Hortonworks Data Platform

These are more robust and well-researched papers.  The article below from August 2014 will be left there for history purposes, but the post and the accompanying video outdated.
*********************************************************************************

Background

Big Data is one of the fastest spreading IT trends in the enterprise today, it's also a big reason why IT infrastructures have to provide elastic and secure infrastructures. Apache Hadoop (with value-added by Cloudera, Hortonworks, MapR, IBM and others) is gaining a lot of traction in enterprises that are looking to adopt Big Data.

What does this mean to the IT Infrastructure Manager?

• More Linux servers
• Need for more effective management
• More systems that need to be aligned with security practices
• Depending data classification,  a mature access management model that allows the enforcement of the principles of least access, least privilege, separation of duties, quick attestation and optionally session capture and replay is required.

Customers of Centrify know that all the bullets above are not an issue with properly Centrified (*) systems.

(*) A mature Centrify deployment has no systems in Express mode,  has discontinued the use of Classic zones and archaic provisioning methods like ZoneGen, and finally is using RBAC via Centrify-enhanced sudo.

Recommendations:

• If you are not familiar with terms like Kerberos authentication, keytabs, servicePrincipalName (SPN), userPrincipalName (UPN) and some of the differences between MIT Kerberos and Microsoft Kerberos, I recommend that you familiarize with those topics.
• From a Hadoop perspective, leverage Hortonworks/Cloudera/MapR/IBM's professional services organizations.  Hadoop is a broad topic and in my observation successful implementations have one thing in common:  they get expert help.  You are busy enough as it is just keeping-up with the day-by-day.
• Keep in mind, the reason why we introducing Active Directory as an alternative to a stand-alone MIT Kerberos is for a simple reason: to eliminate duplication of capabilities.  If you go the MIT Kerberos route, this means that you have to maintain two different technology environments + process + people implications. This just doesn't make sense.
• If you feel comfortable with the topics above, you at least need to understand what the adkeytab utility is.  This post covers adkeytab.

Centrify & AD = Faster Hadoop Implementation

If you've been following this blog, you probably realize the focus on Access Controls and to leverage your existing infrastructure (AD);  this extends to Hadoop deployments as well.  Because:

• You already would have solved the Access and Privilege governance model at the OS level.
• You don't need to add more complexity to your environment to enable Hadoop Security.

If you don't have a mature access and privileged model like local accounts or shared accounts, you don't want to continue to propagate the problem.

Hadoop Infrastructure Security 101

InfoQ does a much better job in this article discussing Hadoop's security evolution.  I will focus on a small piece: internal authentication.

Hadoop leverages MIT Kerberos to enable security by Kerberizing its services.  With Kerberos no passwords go through the wire and there are compensating controls for confidentiality, integrity and other threats.    

What is the impact?

Hadoop providers will ask you to stand-up an MIT Kerberos environment to support your BigData deployment;  a very simple ask, but with a large impact to the enterprise.  Things to take into account:

• Kerberos infrastructure - additional services that need to be highly available.
• Additional process - add/moves/changes of principals as the nodes expand/contract/change
• Potentially a Kerberos trust:  If you want to leverage AD, you may need into the business of a Kerberos trust into AD.  At least one provider has a recipe for this.
• Push-back:  Believe it or not, the idea of an additional authentication infrastructure is not going to be welcome by some IT groups.

This is where Centrify can help.  Centrify provides the Kerberos environment and tools, services and utilities to extend your access model and accelerate the Hadoop deployment.  With that in mind, let's apply the Plan-Do-Check-Adjust model.

Note:  Keep in mind, Hadoop follows the MIT kerberos implementation to the letter. This means that there's no concept of multiple service principal names tied to an account (Unlike AD), therefore as of today, that implies that some services will require one account per service per node.   I repeat now in AD terms:  One AD Account/Keytab, Per Service, Per Node.  This is where utilities like adkeytab can help in the automation process.

Number of Accounts (Keytabs) = Shared Service Accounts + (Nodes * Unique Service Accounts)

E.g.  If you have a 50-node cluster that has 2 services that can share the same account/keytab and 5 services that require unique accounts/keytabs, this means that you will have 2 + 50*5 = 252 AD accounts/unique keytabs.

Believe it or not, the easy part is the technology, the hard part is the process and the automation required.

Planning to Secure your Hadoop Services with Centrify

Infrastructure vs. User Apps: 

Kerberizing the servies like HDFS, MapReduce, Hive, Zookeeper, etc is just one piece of the puzzle, the user-level apps may have different access models.  Many use OS users and groups and Centrify will have you covered, but this is a different design session.

• What is the access model at the OS level? (may imply a different zone or set of computer groups)
• What are the privileges (commands) for a Hadoop operator? (remember, with Centrify there's no need to share privileged account credentials) - continue to enforce the least privilege model with Centrify RBAC
• Just like the Centrify agent, deploying Name Server Cache Daemon can improve performance.
• Have a rock-solid naming convention for:
• AD account names:  remember the 20 character limit.  Make sure that you can identify these accounts correctly in AD.   Use a dedicated OU if you have a large cluster.  Something like Environment-server-service general-to-specific can work.  E.g. "mktqa-had01-spnego"
• Keytab location and naming:  
• Keytab protocol:
• Secure based on requirements at rest.
• Always transport securely
• Keep keytabs where they are needed ONLY.
• Will the user accounts in AD have non-expiring randomized passwords or will the passwords be randomized after a period of time?
• Does the account used for AD Service account/keytab generation have the proper rights in AD to create those users/principals?
• What will be the automation strategy?  Remember, using cleartext passwords in scripts or other facilities is a NO-NO, but adkeytab is kerberized;  therefore you may need to provision a service account with the proper rights in AD and a keytab to be able to launch adkeytab in the automation script.
• How are you implementing Hadoop?  Is your team being trained?  Succesful implementations (just like with Centrify) make use of consulting services;  so leverage your Hortonworks, Cloudera or IBM experts.  After all, how many Hadoop (or Centrify) deployments have you implemented?  How much pure Kerberos knowledge do you have?  

Implementation

In this example, we'll use Hortonworks Hadoop Ambari 1.6.x to enable Kerberos in a cluster.  The domain is corp.contoso.com, the OU for service accounts is UNIX\Hadoop and the naming convention is servername-servicename.

1. Set up your system and join it to AD.  Make sure you don't register an SPN for http because it will be used by Hadoop.  This can be done:
   - Prior to the join:  by editing the /etc/centrifydc/centrifydc.conf and modifying the adclient.krb5.service.principals  directive and removing http.  E.g.
   - After the join: Remove the SPN with  ADUC or with adkeytab with the -x (delspn) option.
   adkeytab --delspn --principal http/hadoop1.corp.contoso.com -V
   Remember that you can check the service principals with the "adinfo -C" command.
2. Set up your cluster and make sure your services are running.
3. Go to Administration-Security and Click Enable Kerberos.
   
4. Click Next in the Getting started page.
5. In the Configure Pages page type the AD domain name in all caps in the Realm Name and  in the Kerberos tool path, type the Centrify-enhanced Kerberos tools path (/etc/centrifydc/kerberos/bin).
   
6. In the Create Principals and Keytabs page, you'll be shown a table with all the principals per host based on your configuration.
   
   Ambari provides a CSV file that can be consumed by a keytab-generating script.  In the case of Centrify, you can leverage adkeytab to perform these operations.  Here are a few examples of what needs to be done.  Note:  This example assumes that the person running adkeytab has the rights to create the user principals in Active Directory or in the respective container.
1. Shared Keytab Example (Ambari)
   Let's analyze this.
   a) We will be creating a new account (keep in mind that we can adopt an existing principal) (option --new)
   b) We will be specifying a UPN (because this keytab will be used to get a Kerberos TGT) (option --upn;  e.g. ambari-qa@CORP.CONTOSO.COM)
   c) We will create a keytab file (option --keytab /path/to/file)
   d) The AD account will be located in a particular OU  (option --container <x.500 notation>).  E.g. --container "ou=service-accounts,ou=Unix"
   e) Verbose output is always recommended (-V)
   f) The final parameter is the dn (distinguished name) of the account in AD.  (e.g. hadoop-ambari-qa)
   Sample command:
   adkeytab --new --upn ambari-qa@CORP.CONTOSO.COM --keytab /etc/security/keytabs/smokeuser.headless.keytab --container "ou=hadoop,ou=unix" -V hadoop-ambari-qa
   Then, the keytab has to be permissioned accordingly: 
   chown ambari-qa:hadoop /etc/security/keytabs/smokeuser.headless.keytab
   chmod 440 /etc/security/keytabs/smokeuser.headless.keytab
2. Individual Host Keytab Example (HTTP for hadoop2)
   Analysis
   a) We will be creating a new account (keep in mind that we can adopt an existing principal) (option --new)
   b) We will be specifying a UPN and an SPN (because this keytab will be used to get a Kerberos TGT and a TGS) (options --upn & --principal;  e.g. HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM)
   c) We will create a keytab file (option --keytab /path/to/file)
   d) The AD account will be located in a particular OU  (option --container <x.500 notation>).  E.g. --container "ou=service-accounts,ou=Unix"
   e) Verbose output is always recommended (-V)
   f) The final parameter is the dn (distinguished name) of the account in AD.  (e.g. hadoop-ambari-qa)
   Sample command:
   adkeytab --new --upn HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM --principal HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM --keytab /etc/security/keytabs/spnego.service.keytab --container "ou=hadoop,ou=unix" -V hadoop2-http
   Then, the keytab has to be permissioned accordingly: 
   chown root:hadoop /etc/security/keytabs/spnego.service.keytab
   chmod 440 /etc/security/keytabs/spnego.service.keytab
7. Verify that the principals have been created in AD and that the keytabs are in the selected directory with the proper ownership and permissions.  In addition, you can use the /usr/share centrifydc/kerberos/bin/kinit -kt command to test the principals for TGT or TGS requests.
   
   
8. Go back to Ambari's security wizard and Apply the changes.
   
9. Monitor the services, depending on the performance of your cluster, you may have to start some services manually.

Checking the Implementation

Aside from checking the cluster status, if you're using scripts to automate the add/move/changes of nodes, you need to make sure that those scripts are rock solid.  

Adjusting the Implementation

There are many improvements to be gained, and this depends on your security needs and the variety of environments.  The most notable is the user-facing apps.  Remember that as new environments are spun up and versions of Hadoop change, this process has to be revisited.

Using Kerberos keytabs and Centrify tools to automate UNIX/Linux/Mac AD domain joins or unjoins

The Problem

Dynamic environments expand and contract based on organizational needs; this means that Unix, Linux, Mac OS X servers and workstations are built and decommissioned frequently.  Most recently, public/private cloud elasticity accentuates this issue and having an access controls solution like may end-up adding unnecessary complexity if it's not designed for this reality.

When joining a computer to the domain, the computer name (hostname) may not be known to pre-create the AD account, therefore an an authoritative join is the only choice; the issue here is that an AD account that has the ability to join computers into the domain (OU) and into the Centrify zone is required.

Fortunately, Centrify designed the Server Suite solution tool set with this reality in mind.  In this post, we will discuss how to add or remove systems from AD securely leveraging Kerberos keytabs and tools like adjoin and adleave.  Like all small projects, we'll use the Plan-Do-Check-Adjust methodology.

Requirements

• Scripts should not store plain-text passwords
• Tools must be automation-friendly
• The least access and least privilege principles must be conserved
• The Separation of Duties (SoD) principle must be met

Tools

• Active Directory Users and Computers
• Centrify CLI tools: adjoin, adleave, adkeytab
• Kerberos tools:  kinit, kdestroy.

Planning

For a simple example, we'll consider the following planning steps:

• Incorporate the Centrify agent bits into the infrastructure image, the agent can be installed and not joined.
• An AD Service account with the ability to create, remove (or modify) computer objects to the target domain OU should be created.
• That same AD service account should have the rights to join, remove and modify objects in the target Centrify zone.
• A Kerberos keytab file needs to be created and securely put in a place where the script can use it.  Let's assume that the file will be securely copied to a local drive and deleted upon use.
  Note:  this is important, a Kerberos key table file needs to be to be treated with the same sensitivity as a private key.  They are not to be left behind on systems even if the account has been properly secured.
• In addition to the keytab file, a krb5.conf file with the correct settings needs to be deployed to the unjoined system so the Kerberos tools can find a KDC (AD DC)
• Naming conventions for the service account and the hostname need to be pre-established.  Keep in mind that computer account names have length limitations.
• An AD OU for Unix/Linux or Mac computers is required to limit the scope of where the join account can perform joins.

Implementation (Do)

AD Join service account setup (1-time steps)

1. Create the Active Directory service account by using ADUC.  Make sure that the account and its password do not expire.  In this example we'll use "ad-joiner" and the domain is corp.contoso.com
2. In Active Directory users and computers, use the Delegate Control wizard to delegate the ability to create and delete computer objects.
   
3. In the target Centrify zone, use the Delegate Zone Control wizard to give the service account the rights to join, remove and modify computers in the zone.
   
4. In a Centrified system, use the adkeytab with the adopt option to create the Kerberos keytab file and randomize the password.  This will be performed with the root account so it is protected by that account.
   
   # /usr/sbin/adkeytab --adopt --user jerry.seinfeld--keytab ad-joiner.keytab  -V ad-joiner
   ADKeyTab version: CentrifyDC 5.1.3-482
   Options
   -------
   use machine ccache: no
   domain: corp.contoso.com
   server: null
   user: dwirth
   container: null
   account: ad-joiner
   trust: no
   des: no
   jerry.seinfeld@CORP.CONTOSO.COM's password:
   Attempting bind to corp.contoso.com site:Demo-Site server:dc1.corp.contoso.com: ccache:MEMORY:0x566940
   Bind successful to server dc.centrifyimage.vms
   Searching for AD Object: filter = (samAccountName=ad-joiner), root = DC=corp,DC=contoso,DC=com
   AD Object found: CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp, DC=contoso,DC=com
   Key Version = 2
   Activating AD account: CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp, DC=contoso,DC=com
   Account 'CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp, DC=contoso,DC=com' All SPNs already present
   Adding managed account keys to configuration file: ad-joiner
   Changing account 'ad-joiner' password with user 'jerry.seinfeld@CORP.CONTOSO.COM' credentials.
   Searching for AD Object: filter = (samAccountName=AD-JOINER), root = DC=corp,DC=contoso,DC=com
   AD Object found: CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp,DC=contoso,DC=com
   Key Version = 3
   Success: Adopt Account: ad-joiner
5. Verify the keytab file with klist
   # /usr/share/centrifydc/kerberos/bin/klist -kt ad-joiner.keytab
   Keytab name: FILE:ad-joiner.keytab
   KVNO Timestamp         Principal
   ---- ----------------- ----------------------------------
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM

Checking the Implementation (Do)

In order to verify that the keytab file works and can join or remove a system, you need an unjoined Unix, Linux or Mac system with the Centrify agent installed.

1. Log into the system with a local account (that can elevate)
2. Make the keytab file securely accessible.
3. Copy the /etc/krb5.conf file from a working Centrified system to the local system:
   scp <account>@centrified.system:/etc/krb5.conf /etc/krb5.conf
4. Use the kinit command with the kt option to get a ticket-granting-ticket as the ad-joiner account.
   /usr/share/centrifydc/kerberos/bin/kinit ad-joiner -kt ad-joiner.keytab
5. Use the adjoin command without specifying the user option.  Adjoin is Kerberized and it will use the ad-joiner's ticket-granting ticket.
   # adjoin -z Model -c "ou=servers,ou=unix" corp.contoso.com
   Using domain controller:  dc2.corp.contoso.com writeable=true
   Join to domain: corp.contoso.com, zone: Model succesful
   
   Centrify DirectControl started
   Initializing cache
6. At this point the computer has joined the domain.
7. You can use it for the reverse with adleave.
   # adleave -r 
   Using domain controller:  dc2.corp.contoso.com writeable=true
   Left domain.
   Centrify DirectControl stopped.

Adjusting the Process

There are many opportunities to adjust this process and make it better.  Here are some examples:

• The process can be part of a script that specifies things like zones, computer roles, etc.
• ADEdit can be used to pre-create the account in the zone and move the computer to the proper computer role.
• The keytab can be deleted from the local host as part of the script termination process process

Video

Security Corner: What is a Kerberos Keytab and why should you use it?

Background

Active Directory provides a Kerberos environment.  We have written extensively about Kerberos capabilities in this blog.  We also have explained that Centrify provides tight integration with Microsoft's Kerberos AD implementation by way of their MIT-Kerberos libraries and tools.  The advantage of using Centrify's tool set is that the tools are extensively tested against Microsoft's implementation.

A Kerberos key table (or "keytab") file is "is a file containing pairs of Kerberos principals and encrypted keys (these are derived from the Kerberos password)."(1).

Kerberos keytab files can help overcome two major issues:

• The security requirement of not having plain-text passwords in scripts or helper files.
• The increasing IT Infrastructure requirement for automation.  The proliferation elastic computing by way of private and public clouds requires that IT adds or reduces capacity on demand in a secure way.

Kerberos Keytabs 101

• In a Kerberos environment, each system has at least one keytab table stored on disk.  The keytab table lists the service principals and provides at least one key for each of those service principals (/etc/krb5.keytab by default).  Remember that principals typically follow the "service@host/REALM" format.
• End-users also have Kerberos files, but they are temporary (by default 10 hours), they are stored as a Kerberos cache file (by default /tmp/krb5cc_<uid of the user>), they list any user principals.
• When working with a Centrified system, the location of the Centrify-enabled Kerberos tools is /etc/centrifydc/kerberos/bin.  
• Quick primer on basic Kerberos tools:
• kinit: is used to obtain a kerberos ticket-granting-ticket
• klist: is used to list the cached tickets.
• kdestroy: destroys kerberos tickets
• The Centrify Kerberos tools documentation is publicly available here.
• Keytabs are Sensitive/Confidential information: A Security Procedure needs to be established for the handling of keytabs, because whoever has the keytab can authenticate as the principal used to create it.  As a tip, this process should be viewed with the same sensitivity as the handling of private keys.

Using Kerberos Tools

Kerberos tools are great for troubleshooting purposes.

1. Testing end-to-end AD connectivity and account availability with kinit (e.g. accunt jerry.seinfeld)
   # /usr/share/centrifydc/kerberos/bin/kinit jerry.seinfeld
   Password for jerry.seinfeld@CORP.CONTOSO.COM:
   You are prompted for the user's password.  The benefit of this test that it bypasses the NSS, PAM and authorization modules.  This means that any user in AD can be tested with kinit.
2. Listing the contents of a keytab or cache file with klist
1. For systems
   # /usr/share/centrifydc/kerberos/bin/klist -kt krb5.keytab
   Keytab name: FILE:krb5.keytab
   KVNO Timestamp         Principal
   ---- ----------------- ----------------------------------
   3 05/17/14 22:22:55 host/cen1@CORP.CONTOSO.COM
   
   Note: By default, during a system join (with adjoin) Centrify will automatically register service principals for  http ftp cifs nfs, this parameter is controlled by the adclient.krb5.service.principals directive of the centrifydc.conf file.
2. For users (user root)
   /usr/share/centrifydc/kerberos/bin/klist
   Ticket cache: FILE:/tmp/krb5cc_0
   Default principal: jerry.seinfeld@CORP.CONTOSO.COM
   
   Valid starting     Expires            Service principal
   08/24/14 14:07:02  08/25/14 00:07:04  krbtgt/CORP.CONTOSO.COM@CORP.CONTOSO.COM
   renew until 08/25/14 14:07:02
   Notice that this is a Kerberos Ticket Granting Ticket (krbtgt).

Ticket-Granting-Tickets vs. Service Tickets

Depending on how the application is using the keytab, it's possible that it will request a ticket-granting ticket (TGT), or a service ticket.  Generally, a TGT is given upon a session log-in;  a service ticket is requested when a user attempts to access a service on the network.  The key here is that a principal without a servicePrincipalName (SPN) can't be used to request a service ticket.  This changes how the AD principal is created.

The example above shows how to request a TGT.  This example, shows how to test a keytab for a service ticket:

First, we check if the account has a servicePrincipalName (e.g. db2inst1)

$ adquery user -A db2inst1

dn:CN=db2inst1,CN=Service-Accounts,DC=centrifyimage,DC=vms

samAccountName:db2inst1

userPrincipalName:db2inst1@CENTRIFYIMAGE.VMS

servicePrincipalName:db2inst1/engcen5.centrifyimage.vms

Second, we attempt to get a service ticket (the keytab is db2inst1.keytab)

$ /usr/share/centrifydc/kerberos/bin/kinit -S db2inst1/engcen5.centrifyimage.vms -kt db2inst1.keytab db2inst1

[db2inst1@engcen5 ~]$ /usr/share/centrifydc/kerberos/bin/klist

Ticket cache: FILE:/tmp/krb5cc_1004

Default principal: db2inst1@CENTRIFYIMAGE.VMS

Valid starting     Expires            Service principal

11/24/14 11:24:46  11/24/14 21:24:47  db2inst1/engcen5.centrifyimage.vms@CENTRIFYIMAGE.VMS

        renew until 11/25/14 11:24:46

Notice that this is not a KRBTGT, but a service ticket.

Using Centrify Tools for Kerberos-related operations

adinfo (with the C option) can be used to review the system service principals, and encryption level. Remember that the latter is set by the domain controller functional level.
$ adinfo -C
Computer Account Diagnostics
  Joined as: cen1
  Trusted for Delegation: false
  Use DES Key Only: false
  Key Version: 6
  Service Principal Names: nfs/cen1.corp.contoso.com
                           nfs/cen1
                           ipp/cen1.corp.contoso.com
                           ipp/cen1
                           http/cen1.corp.contoso.com
                           http/cen1
                           host/cen1.corp.contoso.com
                           host/cen1
                           ftp/cen1.corp.contoso.com
                           ftp/cen1
                           cifs/cen1.corp.contoso.com
                           cifs/cen1
                           afpserver/cen1.corp.contoso.com
                           afpserver/cen1

Supported Encryption Type(s): RC4-HMAC
                              AES128-CTS-HMAC-SHA1-96
                              AES256-CTS-HMAC-SHA1-96

Operating System Version: 6.1:6.5 (Final)

Adkeytab

adkeytab, is arguably one the most powerful command line tools provided by Centrify.  When used in tandem with scripts and the Kerberos tools it can help to increase security and provide flexibility in automation scenarios.  It has the ability to provision, deprovision and modify server and user principals.  Aside from adedit, adkeytab contains the largest command reference.

adkeytab  -  create  and manage Kerberos key tables (*.keytab files)and coordinate changes with the  Kerberos  key  distribution center(KDC) provided by Active Directory.

I will try to highlight some benefits of using adkeytab:

• It can be used to create or adopt AD principals:  adkeytab can create the user in AD for you, or in a separation of duties scenario, the UNIX/Linux admin can adopt the account provided he has the appropriate rights to modify the AD account.
• It randomizes the AD account's password:  The major issue with service accounts it that they are considered a shared account since at least one person (the creator or the user) knows the password.  With adkeytab it's possible to randomize the password of the AD account, effectively minimizing the need for aggressive password rotation.
• Can be fully automated:  adkeytab can be used with your scripts to perform operations.  This helps with elastic or private clouds.
• It's Kerberized!  what that means is that you can use a keytab to use adkeytab - this is great for automation.
• Syntax can be tricky - there are so many options that it can be overwhelming for beginners; my tips are to use the long for options (e.g. --adopt) and verbose output.
• Fixes stale AD computer accounts:  It can be used to reset the machine computer account.  Just elevate to use the adkeytab command with the -C (--change-password).  The output below uses the local system's computer account (--machine) in AD to reset the computer's password in AD.

$ dzdo adkeytab --change-password --machine -V

Attempting bind to centrifyimage.vms site:Demo-Site server:dc.centrifyimage.vms: ccache:FILE:/etc/krb5.ccache

Bind successful to server dc.centrifyimage.vms

Searching for AD Object: filter = (samAccountName=engcen5$), root = DC=centrifyimage,DC=vms

AD Object found: CN=engcen5,OU=UNIX-Servers,OU=Unix,DC=centrifyimage,DC=vms

Key Version = 21

Changing account 'engcen5' password with machine credentials.

Searching for AD Object: filter = (samAccountName=engcen5$), root = DC=centrifyimage,DC=vms

AD Object found: CN=engcen5,OU=UNIX-Servers,OU=Unix,DC=centrifyimage,DC=vms

Key Version = 22

Success: Change Password: Default Key Tab

The posts below, cover scenarios that use Kerberos key table files:

• Big Data (Hadoop)
• DB2 AD Integration and SSO
• MongoDB AD-Integration with Centrify
• Automating system AD joins/removals

(1) Source: Indiana University IT, https://kb.iu.edu/d/aumh

Labs: Securing Multi-protocol File Access (NFS/CIFS) on NAS (NetApp) Using Centrify and Windows Security Groups

Background

In the previous post, we discussed how to overcome identity limitations in NAS sharing scenarios.  In heterogeneous environments the main issue is about Identity - the filer is unable to consistently translate the Windows identity to a UNIX identity.  With Centrify Zones and the LDAP Proxy we solved the issue providing a unified identity consistently across Windows, Unix, Linux and Macs.

In Summary, here's what we accomplished:

The LDAP Proxy present's user information for the filer with consistency.  The UNIX information corresponds to what the AD user has in the Centrify Zone.

Now, we can let the NetApp filer do it's magic.  Let's look at a very simple example:

Testing Mixed Share Permissions

Set up the NetApp share

1. Create a new qtree
   qtree create /vol/vol0/mixed
2. Change the security model as mixed
   qtree security mixed
3. Export the newly created qtree
   exportfs -p rw /vol/vol0/mixed

Create a New AD Security Group for the Mixed Share Access

1. Open ADUC browse to an OU for groups
2. Select New > Group
3. Give it a name  (e.g. "Mixed-Share-Access")
4. Add a test user to the newly created share.

Publish the share in Windows

1. Open Computer Managemetnt
2. Connect to your NetApp filer
3. Navigate to System Tools > Shared folders
4. Follow the wizard to create your share
5. In the Share Permissions, remove Everyone and add your previously-created AD Group

Now you have:

• A CIFS share that is a shared folder protected by a Security Group in AD.  Access is governed by group membership and the permissions on the share.  NTFS permissions will be assigned at file/folder creation based on the user's identity.
• An NFS share that can be mounted from any Unix, Linux or Mac system;  UNIX file permissions will be assigned at folder/file creation based on the user's identity.  We'll leverage the NETGROUPS option with Centrify-exposed options to accomplish this.

Let's see this in action

Improvements

As outlined in the latter part of the video, there's definitely improvements that can be implemented:

• On the NetApp side, Kerberos can be implemented for additional security.  Remember that Centrify already provides "hands-off" Kerberos environment optimized for Microsoft's Kerberos.
• The AD Group used to control access can be UNIX-enabled, this opens the possibility of leveraging the group's UNIX identity as a group owner on the UNIX side.

Key benefits

• Process consolidation - now a single AD group can be leveraged to control access.
• Better Security - there's a better grasp on who has access to what.

Business Problems: Conquering the NAS multiprotocol (CIFS/NFS) file sharing in a mixed (Unix, Linux, Windows, Mac) environment using Centrify

Background

Network Attached Storage (NAS) is a key part of any enterprise today; they provide flexibility and an abstraction layer from an infrastructure perspective.  In the heterogeneous enterprise, the easy part is the infrastructure piece;  the hard part has to do with the diversity of platforms and the multiple file-sharing protocols in use.  This post and a series of labs will illustrate how leveraging Active Directory, Centrify Zones, the Centrify agent for UNIX, Linux & Mac and the Centrify LDAP Proxy we can ensure consistency with multi-protocol CIFS/NFS shares and enhance operational efficiency.

In previous posts, we have addressed how to to solve the consistency issue individually:

• Centrify and NFS Basics  (video lab)
• Centrify and Samba (CIFS) basics  (lab1, lab2)

This posts uses a NetApp filer as an example(*).  The concepts outlined here can be used on any NAS device that supports LDAP.  We will assume that you understand that the fundamental problem is how to solve the issue of identity across multiple platforms and how we accomplish that with the Centrify Agent and Active Directory (I highly recommend that you read the previous links).

The Issues (or variations of it)

• We have a NAS device, we integrated it with AD and it works great as long as the user is performing file operations on Windows.  However, we have to keep separate NFS shares.  We also have to manage permissions in multiple groups (Windows or UNIX) and that takes a lot of effort.
  Note, in my example with NetApp, the appliance does a great job at discovering LDAP servers:
  
  netapp1> Sat Jul  5 11:26:52 EST [netapp1:auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- AD LDAP server address discovery for CORP.CONTOSO.COM complete. 2 unique addresses found.
  
  However, although it can resolve the AD portion, it is plain wrong about the Unix UID:
• We have a multi-protocol share (CIFS/NFS), everything is great on Windows, when the user wants to access their files over NFS he has to call to have the file/folder ownership taken care of
• We are tired of having to resolve file ownership issues.  Our organization is globally very diverse and each time we have a new visitor we spend significant amount of time helping them get access.
• We have a combination of Unix, Linux, Mac and Windows users and although we had it down in individual servers, when we moved to filers now we have all these new issues.
• I have a filer in my environment and we have Centrify as well, I know it can help but I don't now where to start.

Example of one of the issues

Let's look at an example.  In my environment with a domain-joined Netapp the appliance does a great job at discovering LDAP servers (this happened once I searched for one of my users):

netapp1> Sat Jul  5 11:26:52 EST [netapp1:auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- AD LDAP server address discovery for CORP.CONTOSO.COM complete. 2 unique addresses found.

netapp1> wcc -s george.constanza

(NT - UNIX) account name(s):  (CORP\george.constanza - pcuser)

        ***************

        UNIX uid = 65534

        NT membership

                CORP\george.constanza

                CORP\UNIX-Model-Samba Users

                CORP\UNIX-Model-All Users

                CORP\UNIX-Model-dbaadmin

                CORP\Domain Users

                CORP\UNIX-Model-Sysadmins

                CORP\UNIX-Model-webadmin

                CORP\UNIX-Model-All UNIX Groups

                BUILTIN\Users

        User is also a member of Everyone, Network Users,

        Authenticated Users

        ***************

This is great from a Windows perspective, however, look at the Unix UID (65534), when you look at the options, this means that the user is being assigned a "nobody" user from a UNIX perspective.  We can further confirm this with this command:

netapp1> wcc -u george.constanza

no passwd entry for george.constanza

The way I have my system configured, although I can derive the user's identity leveraging the DCs, I can't see the user's Unix identity.  It is stored in the zone. See the output from a centrified system:

george@ubu1:/etc/centrifydc/openldap$ adquery user george.constanza -A

unixname:george

uid:1149240406

gid:1149240406

gecos:George Constanza

home:/home/george

shell:/bin/bash

auditLevel:AuditIfPossible

isAlwaysPermitLogin:false

dn:CN=George Constanza,OU=Staff,DC=corp,DC=contoso,DC=com

samAccountName:george.constanza

displayName:George Constanza

sid:S-1-5-21-2180375406-786980114-1643973036-1110

userPrincipalName:george.constanza@corp.contoso.com

canonicalName:corp.contoso.com/Staff/George Constanza

passwordHash:x

accountExpires:Never

passwordExpires:Wed Aug 13 11:28:48 2014

passwordWillExpire:39

nextPasswordChange:Thu Jul  3 11:28:48 2014

lastPasswordChange:Wed Jul  2 11:28:48 2014

accountLocked:false

accountDisabled:false

zoneEnabled:true

unixGroups:dbaadmin,george,webadmin

memberOf:corp.contoso.com/UNIX/Provisioning/UNIX-Model-All UNIX Groups,corp.contoso.com/UNIX/Provisioning/UNIX-Model-All Users,corp.contoso.com/UNIX/RBAC/UNIX-Model-Samba Users,corp.contoso.com/UNIX/RBAC/UNIX-Model-Sysadmins,corp.contoso.com/UNIX/UNIX Groups/UNIX-Model-dbaadmin,corp.contoso.com/UNIX/UNIX Groups/UNIX-Model-webadmin,corp.contoso.com/Users/Domain Users

Our goal is to make sure that our filer (or application) gets the correct Identity information for the purposes of proper file/folder ownership, access and permissions.

Who's affected

As you can see, the variety of problems outlines have multiple constituents:

• Security needs to make sure that users have access to what they need and to comply with policies
• IT (business) need to make sure that users can do their jobs
• IT (operations) wants to be efficient and make sure that an army of people is not needed
• The End User just wants to get their job done

As with all Business Problems, we'll use the Plan, Do, Check, Adjust methodology.

Planning

As discussed in previous posts, the key to this issue is Identity.  In this case, the NAS appliance needs to know not only the user's AD information, but the extended attributes (RFC2307) like unixname, UID, GID, Home, GECOS, Shell, etc., and unlike other approaches, Centrify stores this information in the zone.  This makes the solution more flexible because an AD principal can have multiple Unix identities.

Note:  If you've read this blog, you know that I've been very vocal about overrides (what enables multiple identities and roles).  Every process needs to have constraints; constraints make solutions simpler.  Having a normalized UNIX namespace is the goal and destination of any enterprise, even if Centrify allows you to put off this goal.  

Identity and Access Planning

1. Do the best you can to have a normalized namespace.  Leverage Centrify's ability to assign users a unique UID by generating it from the AD SID (using Centrify or Apple's Scheme).
2. Perform a migration if necessary (this post talks about strategies and Centrify tools for that purpose)
3. Leverage the power of hierarchical zones:  provision all identities at the top-most zone.  This will ensure that all pertinent systems will know the user's identities.
4. Continue to enforce the least-access principle:  Use the "listed" role to make users known to the systems (e.g. the LDAP proxies) and ensure that they can't sign-in to the systems.

Planning your LDAP Proxies

The Centrify LDAP proxy leverages the Centrify client (adclient) to present AD information to clientless devices or programs.  The benefits of leveraging the proxy are:

• It flattens AD for the application:  depending on the enterprise, AD can be complex.  There may be multiple forests, child domains, one-way, two-way trusts, etc.
• It leverages the agent's built-in capabilities:  optimal DC selection, site-awareness and offline cache.

Plan for high-availability:  the NAS (or application) is going to require redundancy, plan to deploy and configure multiple proxies.

Plan for confidentiality:  make sure that only authenticated connections over secure transports (TLS, SSL, etc) in case your appliance or application does not do it natively.

Plan for high-performance:  As always, have NSCD on the Centrified system, and plan for co-location near the appliance, application and be close to Global Catalog servers in complex environments.

Plan for your file-sharing use case

This section varies based on what you want to accomplish.  Planning for home directories is not the same as planning for a workgroup share.  

Implement (Do)

1. Unix-enable all your relevant users (you don't need to Unix-enable users that won't ever access via NFS).  You can do this manually, with ZPA or with your own IdM solution.   
2. Assign roles as required:  keep the least-access principle intact by leveraging the "listed" role.
3. Install the LDAP Proxy(es):
1. Copy the Centrify LDAP proxy binaries to the platfrom.
   e.g. for Ubuntu on 2014:  centrifydc-ldapproxy-5.1.3-deb5-x86_64.deb
2. Log in and install the package
   e.g. dpkg - i centrifydc-ldapproxy-5.1.3-deb5-x86_64.deb
   The LDAP proxy binary is on /usr/share/centrifydc/libexec
   The config files and schemas are in /etc/centrifydc/openldap
   The LDAP utilities (like ldapsearch)  are on /usr/share/centrifydc/bin
3. Modify the RFC 2307 map
1. Edit the /etc/centrifydc/openldap/rfc2307.map file
2. Add the following line (for NetApp compatibility):
   # Added for NetApp
   posixAccount.userPassword: _userPassword
4. For testing only, start slapd manually:
   dzdo /usr/share/centrifydc/libexec/slapd -f /etc/centrifydc/openldap/ldapproxy.slapd.conf -h ldap://ubu1.corp.contoso.com
5. Check that the proxy is running
   $ ps -ef | grep slapd
   root     23192     1  0 Jul04 ?        00:00:00 ./slapd -f /etc/centrifydc/openldap/ldapproxy.slapd.conf -h ldap://ubu1.corp.contoso.com
6. Verify that the proxy is responding to queries (e.g. check the Administrator account)
   $ /usr/share/centrifydc/bin/ldapsearch -h ubu1.corp.contoso.com-x -b "dc=corp,dc=contoso,dc=com" "(cn=Administrator)"
   # extended LDIF
   #
   # LDAPv3
   # base <dc=corp,dc=contoso,dc=com> with scope sub
   # filter: (cn=Administrator)
   # requesting: ALL
   # with pagedResults control: size=100
   ## Administrator, Users, corp.contoso.comdn: cn=Administrator,cn=Users,dc=corp,dc=contoso,dc=comaccountExpires: 0
   (output truncated***)
4. Configure your NAS appliance (e.g. on NetApp Ontap v 8.1.2)
1. Housekeeping:  Make sure that the name of the appliance is resolvable (add an A record for it) and that all Centrified systems, Windows Systems can resolve it.  The NetApp system should resolve all systems as well.  E.g.
   netapp1> ping ubu1
   ubu1.corp.contoso.com is alive
   netapp1> options dns
   dns.cache.enable             on
   dns.domainname               corp.contoso.com
   dns.enable                   on
   dns.update.enable            off
   dns.update.ttl               24h
2. Housekeeping:  Make sure that CIFS and NFS are licensed, enabled and the NetApp filer is joined to the domain.  (use the license command, the cifs setup command )
3. Verify that there's a computer object for the netapp device and that you can (as a Domain Admin) connect to it via Computer Management.  E.g. using ldapsearch:
   george@ubu1:/etc/centrifydc/openldap$ /usr/share/centrifydc/bin/ldapsearch -h
   ubu1.corp.contoso.com -x -b "dc=corp,dc=contoso,dc=com" "(cn=NETAPP1)"
   <truncated>
   # NETAPP1, Appliances, corp.contoso.com
   dn: cn=NETAPP1,ou=Appliances,dc=corp,dc=contoso,dc=com
   <truncated>
4. Optional:  Make sure that Domain Administrators are not mapped to root the NetApp filer:
   netapp1> options wafl.nt_admin_priv_map_to_root off
5. Configure NetApp to use the LDAP Proxy (Example corp.contoso.com;  ubu1)
1. Set up the LDAP base container
   netapp1> options ldap.base dc=corp,dc=contoso,dc=com
2. Set up the LDAP server(s) to be used by the NetApp appliance
   netapp1> options ldap.servers ubu1.corp.contoso.com
3. Set up the name of the Active Directory Domain
   netapp1> options ldap.ADdomain corp.contoso.com
4. Map the userPassword attribute to avoid null results
   netapp1> options ldap.nssmap.attribute.userPassword cn
5. Enable LDAP
   netapp1> options ldap.enable on
6. Modify the /etc/nsswitch.conf to make LDAP the first source for user, group and shadow entries.
   netapp1> wrfile /etc/nsswitch.conf
   hosts:  files  dns
   passwd:  ldap files
   group: ldap files
   netgroup:  files nis
   shadow:  files nis
7. Verify the results
1. Query and note the UID from Windows
   wcc -s jerry.seinfeld 
2. Query and note the UID from UNIX
   wcc -u jerry.seinfeld
3. Advanced query using NSS
   priv set diag
   getXXbyYY getpwbyname_r jerry.seinfeld
   
   Compare the results with the zone information.  At this point it should match the UNIX identity on Access Manager

Check the Results

To verify the results, use a multi-protocol (CIFS/NFS) share and make sure that the identity information is consistent.

1. From a Windows system in the target domain, map or browse to the share via CIFS.  Create a file.
2. From a Unix/Linux or Mac system, mount the share as an NFS mount and issue the ls -la and ls -lan.  Compare the results from before.

Adjustments to the Implementation

Adjustments are based on your existing environment, however, based on ours here are some key modifications:

• There's only one LDAP proxy, we need several for high-availability
• The LDAP Proxy is accepting anonymous connections, we need to make sure that only authenticated connections are accepted.
• Ideally we would use Secure LDAP or implement TLS or IPSec for additional confidentiality controls.
• Operations needs to monitor the health and status of the LDAP Proxies

Lab Videos

What you'll need to follow along:

• Active Directory
• Centrify Standard Edition
• A Centrify zone
• Client for Unix, Linux or Mac
• Centrify's LDAP Proxy
• A Multi-protocol filer (we'll use NetApp's  ONTAP Simulator)

Installing, Configuring and Testing the Centrify LDAP Proxy

Configuring the NetApp appliance to use the Centrify LDAP Proxy


Verifying the NetApp CIFS/NFS share provides consistent identities

Mac OS X Extras: Computer Certificate Auto-enrollment

Background

Digital Certificates have an important place in a properly managed enterprise.  From an infrastructure perspective, they can enhance authentication and provide encryption for Ethernet and Wifi Networks.
Centrify for Mac OS X has built-in capabilities to enable 802.1x authentication leveraging Group Policy but it does require that the computer has a digital certificate.

PKI Disclaimer:  As in all PKI-related posts and videos, I make the caveat that Public Key Infrastructure is no joke.  There are policy, people, process, security and technology implications to your enterprise, so all PKI deployments need to provide a high-level of assurance.  You may have landed here due to a google search or reference, feel free to use these posts for testing purposes, but again, when it comes to PKI, any production deployment should conform to best practices.

Centrify uses GPOs to configure computer or user-based 802.1x settings on the Mac OS X platform

Configuring Computer AutoEnrollment for Mac OS X

The Centrify adclient is capable of leveraging Windows certificate auto enrollment with the Microsoft CA. The basic steps are:

On the AD side (with a Domain or Cert Admin)

1. Configure the certificate template based on your needs (using the Certificate Templates MMC)
• Subject  (typically common name based on the User Principal Name)
• Security (set it to an AD group containing your Mac Systems and check to Enroll and AutoEnroll)
• Extensions (add what you need)
  If using it for 802.1x - usage should be Client and Server Authentication.
2. Configure your CA to issue Certificates based on that template (using the Certificate Authority MMC)
3. Modify your GPO to enable the PKI policies for auto-enrollment
   Enabling the Computer Configuration > Windows Settings > Security Settings > Public Key Policies > "Certificate Services Client - Auto-Enrollment Settings" GPO

On the Mac (domain-joined)

1. Flush the cache with adflush (or wait the cache flush interval)  [sudo adflush]
2. Refresh the group policies (with adgpupdate) or wait for the GP refresh interval
3. Verify the Certificates on the Keychain Access app.

Video Lab