Setting-up a Kerberos keytab to Automate AD join/unjoin operations (DevOps)

This article is a republish of a blog post that I wrote for the Centrify Community Techblog.

Background

In order to automate Active Directory instance joins and unjoins, we need a keytab file corresponding to an AD user that has the proper rights in AD and in the Centrify zone.

Because a keytab is pretty much a credential, we are going to adhere to the following principles:

• The credential for the keytab shall have least minimum access
• The password for the credential shall be unknown
• The keytab file shall always be deleted once it's used.

What you'll need:

• You may need assistance from your Active Directory lead
  • Why? First and foremost separation of duties.  Also, in practicality, the UNIX/Linux/Mac teams in enterprises are different from the AD teams.
  • What do they need to do for me?  They need to create an AD account, provide delegation (see below) and they may have to type their credential when running the adkeytab command.
  • How many times do we need to do this?  Only once. 

Create an AD User

When you create an AD service account, you have to align with the security policy governing these accounts.  The overhead can come depending on the frequency of password changes.  Since adkeytab scrambles the password and it's effectively unknown, the account can be set with a Password Never Expires/User cannot change password flag, however for due-diligence you may rotate it once a year.  Each time the password is rotated, the key table file has to be generated again.

1. Open Active Directory Users and Computers
2. Navigate to the container or OU for this service account
3. Select New > User
4. In the New Object form, set the information based on your naming convention
   
   Note:  the "common name" of the user is typically the same as the display name.  If this contain spaces, you have to make note of this for when you use the adkeytab command (samaccountname vs cn).
5. In the New Object - User form, type a password and set the options according to your service account policy
   
   Remember, if you do the right thing, adkeytab will randomize the password and this can be a compensating control.  If you are required to change this password, you must re-generate the keytab and redistribute, otherwise your scripts or recipes will fail.

If you prefer PowerShell instead, you can use the New-ADUser commandlet.

Now you should have a service account.  Make note of the username (e.g. ad-joiner) vs. the cn (AD Joiner Service Account).

Delegate Permissions

There are 2 delegations needed to make sure the automation of joins/removals works.  The service account should be able to create/remove  computer objects in your designated AD container for UNIX/Linux or Mac systems, plus if you're using Centrify zones, the system has to have the ability to join, remove and modify computer profiles.

Optionally, there's a third delegation related to Computer Roles (contained in AD groups); for this you need to provide the "manage group membership" delegation to the target groups (or OU that contains the groups).

Let's illustrate the steps using this OU structure

 In this scenario, I plan to add the UNIX/Linux computers to the Servers SubOU under Centrify; this means that I have to delegate at that level to preserve the least privilege principle.  In a real-world deployment, you may have a different layout (perhaps based on sites), in that case you have to delegate in multiple places.

To delegate the computer object in the target OU 

1. In ADUC (as a privileged AD user), right click the Servers SubOU and select "Delegate Control"
2. Welcome Page > Next
3. Users or Groups > press Add, find the service account, select and press OK, then Next
4. Task to Delegate > Select "Create custom task to delegate" and press Next
5. Object Type > Select "Only the following objects in the folder"; check the Computer Objects box and check  Create and Delete.
   
6. Permissions tab > Check "Full Control" under permissions and press Next.
   You can dial this down, however we have this scoped down to the OU and type of object.
7. Completing page > Finish

 Now the service account can create/remove computer objects in the Servers container.

To delegate at the Centrify Zone level

You must know all the Centrify zones that the service account will be leveraged for automation.  In my example I have one zone (AWS). Centrify provides PowerShell to perform bulk delegations (see Set-CdmDelegation, in this link)

1. Open Centrify DirectManage Access Manager
2. If needed, open your target zone(s)
3. Right-click the zone and select "Delegate Zone Control"
4. Selected Objects > Press Add and find and select  your service account, press OK and Next
5. Tasks to Delegate > check join and modify computer operations to the zone (3 check boxes)
   
6. Completing Page > Press Finish.

At this point, if you're not using DirectAuthorize Computer Roles you are done.

Note:  You can always verify delegations by running the "Zone Delegation Report"

Optional:  To delegate for Computer Roles 

Computer Roles allow the grouping of systems as "teams of servers" this gives administrators the flexibility granting access/privileges to systems to  multiple user populations, the only operation required is the the system is a "Member" of the Computer role, and this can be accomplished any time or during system setup by automating add/removal of the computer account into the AD security groups that make-up the computer role.

In my example, I'm leveraging the Centrify recommended OU structure and all my AD Security groups for the purposes of Computer Roles are stored in the Computer Roles SubOU under the Centrify OU.  This means that I only need to make one delegation.  Based on your design, this may vary and you may have to perform multiple delegations.

1. In ADUC (as a privileged AD user), right click the Computer Roles" SubOU and select "Delegate Control"
2. Welcome Page > Next
3. Users or Groups > press Add, find the service account, select and press OK, then Next
4. Task to Delegate > Select "Modify Membership of a Group" and press Next
   
5. Completing > Press Finish

From this point on, the service account will be able perform add/removals of objects into any existing or new AD group in that container.

Create the Keytab File

Creating the keytab may require two individuals.  One individual can run privileged commands on UNIX/Linux/Windows, the other is an authorized AD user that can perform certain operations on the AD user (like changing it's password).

You need to know the account's samaccounname vs the common name.  You can quickly see this using the Attribute Editor in ADUC or PowerShell

This is important because the last parameter of the adkeytab command is the common name.  If you assume they are the same, you may hit this error:

AD Object found: CN=AD Joiner,OU=Service Accounts,OU=Centrify,DC=awsrealm,DC=centrifying,DC=net
Error: The account name does not match the SAM account name. You must supply both on the command line.
Adkeytab return code: 23
Failed: Adopt Account: ad-joiner

Here's a simple way of constructing an adkeytab command

• Elevation required:  root or credential for sudo or dzdo required:  sudo adkeytab
• Operation:  adopt  (needs the sammacount name, an authorized account and the common name):  --adopt
• Authorized user (e.g. AD administrator): --user admin
• AD user: --samname ad-joiner
• Keytab File name (e.g. login.keytab):  --keytab login.keytab  (the file will be owned by root)
• Common Name (if the CN is different from samaccount name):  "AD Joiner"  (since there are spaces, it has to be double-quoted)
• Verbose output recommended (-V)

Here's the command

dzdo adkeytab --samname ad-joiner --adopt --user admin --keytab login.keytab -V "AD Joiner"

Here's a sample output:

dzdo adkeytab --samname ad-joiner --adopt --user admin --keytab login.keytab -V "AD Joiner"
[dzdo] password for lisa:
ADKeyTab version: CentrifyDC 5.4.0-286
Options
-------
use machine ccache: no
domain: awsrealm.centrifying.net
server: null
user: admin
container: null
account: AD Joiner
trust: no
des: no
admin@AWSREALM.CENTRIFYING.NET's password:
Attempting bind to awsrealm.centrifying.net site:Default-First-Site-Name server:dc1.awsrealm.centrifying.net: ccache:MEMORY:0x644640
Bind successful to server dc1.awsrealm.centrifying.net
Searching for AD Object: filter = (samAccountName=ad-joiner), root = DC=awsrealm,DC=centrifying,DC=net
AD Object found: CN=AD Joiner,OU=Service Accounts,OU=Centrify,DC=awsrealm,DC=centrifying,DC=net
Key Version = 4
Activating AD account: CN=AD Joiner,OU=Service Accounts,OU=Centrify,DC=awsrealm,DC=centrifying,DC=net. Clearing existing SPNs: No
Account 'CN=AD Joiner,OU=Service Accounts,OU=Centrify,DC=awsrealm,DC=centrifying,DC=net' All SPNs already present
Adding managed account keys to configuration file: AD Joiner
Changing account 'AD Joiner' password with user 'rpimentel@AWSREALM.CENTRIFYING.NET' credentials.
Searching for AD Object: filter = (samAccountName=ad-joiner), root = DC=awsrealm,DC=centrifying,DC=net
AD Object found: CN=AD Joiner,OU=Service Accounts,OU=Centrify,DC=awsrealm,DC=centrifying,DC=net
Key Version = 5
Updated properties to config file /etc/centrifydc/centrifydc.conf.
Success: Adopt Account: AD Joiner

Verify the keytab is correct.  List the principals

 dzdo /usr/share/centrifydc/kerberos/bin/klist -kt login.keytab
Keytab name: FILE:login.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   5 04/20/2017 17:49:43 ad-joiner@AWSREALM.CENTRIFYING.NET
   5 04/20/2017 17:49:43 ad-joiner@AWSREALM.CENTRIFYING.NET
   5 04/20/2017 17:49:43 ad-joiner@AWSREALM.CENTRIFYING.NET
   5 04/20/2017 17:49:43 ad-joiner@AWSREALM.CENTRIFYING.NET
   5 04/20/2017 17:49:43 ad-joiner@AWSREALM.CENTRIFYING.NET

 Testing your Keytab for Join/Removal Operations

 You can test the keytab by removing and rejoining Active Directory.  You'll need to use kinit to authenticate with the key table file, then leverage adjoin or adleave to check the results.  Optionally, you can use the --computerrole switch of adjoin to check for those operations.

Authenticating using the key table file

1. Sign-in to your system with a privileged user (remember, the key table file is owned by root)
2. Change directories to the location of the key table file.
3. Run the kdestroy command
   /usr/share/centrifydc/kerberos/bin/kdestroy
4. Run kinit with the kt option to get a TGT for the service account and klist to verify the TGT
   

   sudo /usr/share/centrifydc/kerberos/bin/kinit -kt login.keytab ad-joiner
   [dzdo] password for lisa:
   $ /usr/share/centrifydc/kerberos/bin/klist
   $ dzdo /usr/share/centrifydc/kerberos/bin/klist
   Ticket cache: FILE:/tmp/krb5cc_cdc1702888528_Hw29E6
   Default principal: ad-joiner@AWSREALM.CENTRIFYING.NET
   
   Valid starting       Expires              Service principal
   04/20/2017 17:59:19  04/21/2017 03:59:19  krbtgt/AWSREALM.CENTRIFYING.NET@AWSREALM.CENTRIFYING.NET
           renew until 04/21/2017 17:59:20
   

Testing adjoin and adleave operations

Depending on the state of your system, you may test removal or joining first.  Since my system is already joined, I'm testing removal first (note that I switched to ec2-user since I was logged in as an AD user).  Note that I'm copying the centrify-populated krb5.conf file, since this file will be rolled-back once the system left the domain.

$ dzdo su ec2-user
$ sudo cp /etc/krb5.conf .
$ sudo adleave --remove
Using domain controller: dc1.awsrealm.centrifying.net writable=true
Left domain.
Centrify DirectControl stopped.

Attempting adjoin to the Zone an the "Utility-Servers" computer role.

$ sudo env KRB5_CONFIG=krb5.conf  /usr/share/centrifydc/kerberos/bin/kinit -kt login.keytab ad-joiner
$  sudo adjoin --zone AWS --container ou=servers,ou=centrify --computerrole Utility-Servers awsrealm.centrifying.net
Using domain controller: dc1.awsrealm.centrifying.net writable=true
Join to domain:awsrealm.centrifying.net, zone:AWS successful

Centrify DirectControl started.
Initializing cache
.
You have successfully joined the Active Directory domain: awsrealm.centrifying.net
in the Centrify DirectControl zone: CN=AWS,CN=Zones,OU=Centrify,DC=awsrealm,DC=centrifying,DC=net


You may need to restart other services that rely upon PAM and NSS or simply
reboot the computer for proper operation.  Failure to do so may result in
login problems for AD users.

What next?

At this point you have to distribute your key table file to your distribution points web server, repository, file share, etc. Note that this file needs to be deleted after it's used for added security. For example, you can upload this file to an AWS S3 bucket to use it with your AWS OpsWorks or CloudFormation scripts.

Business Cases: IBM DB2 Extra - Configuring GSSAPI/Kerberos SSO with the Centrify GSS Plugin

This is the continuation of our previous post about integrating DB2 to AD by using the Centrify IBM DB2 SSO module.  The IBM DB2 SSO module provides 3 basic plugins:

• The user/password plugin that leverages OS authentication (via PAM) to enable DB2 access to AD users.
• The group plugin that exposes AD group memberships to DB2
• The GSS (SSO) plugin that allows silent sign-on to DB2 and can be leveraged in AIX systems that are using LAM instead of PAM

The pre-requisites are the same as the previous post.  Because for the SSO plugin to work we need an AD account to create a keytab, we've added some additional steps.

A little bit of planning

Obtaining the AD Principal
A keytab file is usually tied to an AD principal;  all Centrified systems have one;  but in this case, we will have a service account and there are two ways to do this:

• If you have an AD account that can create user principals:  
• you can use the setupdb2.sh script, option 2 and follow the prompts
• you can use the adkeytab command with the --new option and perform everything in a single step.
• If you don't (which is quite common because of separation of duties), you need to request the account and then work in tandem with the AD folks to adopt the account.

Naming conventions

Because we're talking about AD accounts here, your organization may (hopefully) have a naming standard, if not, you have to keep in mind that you need to be able to identify the purpose of the AD account in AD.  A naming convention like this:

instanceName-server 

is simple enough and descriptive.  In this example, my instance is db2inst1 and the server is engcen8, therefore my AD account will be called db2inst-engcen8.

Container OU

If your AD admin is not messy, most likely they will have an OU designated for Service Accounts.  In fact, if you worked with Centrify Professional Services, there should be an OU under UNIX with the same name.  

Active Directory Account and Keytab

AD Account creation
When you request the account, the AD sysadmin will create an account like this:

Typically the best practices for service accounts are to set the password to never expire, and that the account can't change it.

Issue #1 - At this point, the password for this service account is known at least by one person.  This will go away once the keytab is adopted, the password will be randomized.

Now there are two things that can happen - the AD admin can grant you temporary full control of the account until you run adkeytab, or can just type in their password in the adequate moment.

Verify the account 
Use adquery user and the -A option and inspect the UPN and cn.  Here's the truncated output:

$ adquery user -A db2inst1-engcen8
dn:CN=db2inst1-engcen8,CN=Service-Accounts,DC=centrifyimage,DC=vms
samAccountName:db2inst1-engcen8
userPrincipalName:db2inst1-engcen8@centrifyimage.vms
canonicalName:centrifyimage.vms/Service-Accounts/db2inst1-engcen8

Issue # 2 the account is missing a servicePrincipalName (SPN), 

Issue # 3, the the userPrincipalName is in a non-MIT Kerberos friendly format. 

This means that we need 

• To adopt the account, this will randomize the password and create a keytab file (options A or --adopt and K or --keytab);  e.g.  /home/db2inst1/db2inst1-engcen8.keytab  (permissioned accordingly)
• To update the UPN (option -U or --upn);  e.g. engcen8@CENTRIFYIMAGE.VMS
• To add an SPN (option -P or --principal) with the format instance/server@suffix@REALM; e.g. db2inst1/engcen8.centrifyimage.vms@CENTRIFYIMAGE.VMS
• To specify an AD account that perform the adoption the keytab (-u --user). e.g. dwirth
  needs to be able to change account attributes like the UPN, SPN and password.
• We need to know the canonicalName (cn) of the account.  e.g. db2inst1-engcen8

Adopt the keytab and modify the account attributes

Based on the information gathered above, our adkeytab command looks like this (elevating with dzdo to write to the target folder):

$ dzdo adkeytab --adopt --principal db2inst1/engcen8.centrifyimage.vms@CENTRIFYIMAGE.VMS --upn db2inst1-engcen8@CENTRIFYIMAGE.VMS --user dwirth -V --keytab /home/db2inst1/db2inst1-engcen8.keytab db2inst1-engcen8

Here's the command output (truncated)

dwirth@CENTRIFYIMAGE.VMS's password:

Adding managed account keys to configuration file: db2inst1-engcen8

Changing account 'db2inst1-engcen8' password with user 'dwirth@CENTRIFYIMAGE.VMS' credentials.

Key Version = 3

Success: Adopt Account: db2inst1-engcen8

Inspect the account post-adkeytab

$ adquery user -A db2inst1-engcen8

dn:CN=db2inst1-engcen8,CN=Service-Accounts,DC=centrifyimage,DC=vms

userPrincipalName:db2inst1-engcen8@CENTRIFYIMAGE.VMS

servicePrincipalName:db2inst1/engcen8.centrifyimage.vms

Notice how now the UPN was modified and how the account has a servicePrincipalName.

Make sure the keytab file has the right permissions

The owner of the keytab should be the instance account.

$ chown db2inst1:db2iadm1 /home/db2inst1/db2inst1-engcen8.keytab

Configuration of the SSO Module and DB2

At this point, all we need to do is rerun the setupdb2.sh script and follow the prompts to add the SSO plugin;  because we don't want to deactivate the existing ones, we will run option 1 again and point to the previously created keytab.

 /usr/share/centrifydc/bin/setupdb2.sh inst=db2inst1

Is db2inst1 a DB2 server install?

Enter y for yes, n for no [y]: y

db2inst1 is a 64 bit instance

DB2 server and client setup will be done.

Is this DB2 version 9.5 or later?

Enter y for yes, n for no [n]: y

Which DB2 auth method do you want to use?

[1] Username/Password and Single sign-on

[2] Single Sign-on only

[3] Username/Password only

[4] Skip this step

Select a number from the menu [1]:

Select a number from the menu [1]: 1

Use the CentrifyDC group plugin?

Enter y for yes, n for no [y]: y

Do you want to configure the instance user db2inst1 as a service account?

You must do this step if you want to use the GSS-Plugin.  If you already did

this step for this instance, select the option to indicate the keytab file

name.

[1] Use adkeytab to create a service account in Active Directory and keytab

    file.  NOTE: You need to specify a user name with administrator privileges

    on the domain to use adkeytab.

[2] Provide the name of an already existing keytab file.

[3] Skip this step

Select a number from the menu [1]: 2

What is the name of the keytab file? Full path please.

[ /home/db2inst1/db2inst1.keytab ]: /home/db2inst1/db2inst1-engcen8.keytab                          

What group should be used as the group owner of this file? All DB2 instances

that you want to use the username/password plugin must be in this group.

[db2iadm1]:

*********** adkeytab setup (required for GSS-plugin) ***********

Using /home/db2inst1/db2inst1-engcen8.keytab for the keytab file for instance: db2inst1

# db2set DB2ENVLIST=KRB5_KTNAME

adkeytab setup successfully!

******* Installing the plugins into instance: db2inst1 *******

Installing client side auth plugin

# rm -f sqllib/security32/plugin/client/centrifydc_db2gsskrb5.so

# cp /usr/share/centrifydc/lib/libcentrifydc_db2gsskrb5.so sqllib/security32/plugin/client/centrifydc_db2gsskrb5.so

Continuing will stop the DB2 instance: db2inst1, update the configuration

and then start the instance.

Continue? y

New configuration:

 Group Plugin                             (GROUP_PLUGIN) = centrifydc_db2group

 GSS Plugin for Local Authorization    (LOCAL_GSSPLUGIN) = centrifydc_db2gsskrb5

 Server List of GSS Plugins      (SRVCON_GSSPLUGIN_LIST) = centrifydc_db2gsskrb5

 Server Userid-Password Plugin        (SRVCON_PW_PLUGIN) = centrifydc_db2userpass

 Server Connection Authentication          (SRVCON_AUTH) = GSS_SERVER_ENCRYPT

 Database manager authentication        (AUTHENTICATION) = SERVER

Starting instance

# db2start

SQL1063N  DB2START processing was successful.

The plugins for DB2 instance: db2inst1 were set up successfully!

I recommend a reboot at this point.

Verify that everything is working as expected

Log in with an AD user to the DB2 server

Issue the klist command to verify that you have a TGT (ticket-granting ticket), if not issue the /usr/share/centrifydc/kerberos/bin/kinit command and type your AD password when prompted.

$ klist

Ticket cache: FILE:/tmp/krb5cc_1627391058

Default principal: dwirth@CENTRIFYIMAGE.VMS

Valid starting     Expires            Service principal

11/16/14 22:19:10  11/17/14 08:19:13  krbtgt/CENTRIFYIMAGE.VMS@CENTRIFYIMAGE.VMS

        renew until 11/17/14 22:19:10

Attempt to access the sample database without specifying the username

$ db2 

db2  => connect to sample

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.5.0

 SQL authorization ID   = DWIRTH

 Local database alias   = SAMPLE

db2  => quit 

Now verify that you have a Kerberos service ticket requested via GSSAPI.

$ klist

Ticket cache: FILE:/tmp/krb5cc_1627391058

Default principal: dwirth@CENTRIFYIMAGE.VMS

Valid starting     Expires            Service principal

11/16/14 22:19:10  11/17/14 08:19:13  krbtgt/CENTRIFYIMAGE.VMS@CENTRIFYIMAGE.VMS

        renew until 11/17/14 22:19:10

11/16/14 22:22:36  11/17/14 08:19:13  db2inst1/engcen8.centrifyimage.vms@CENTRIFYIMAGE.VMS

        renew until 11/17/14 22:19:10

Video

(10 minutes, 15 seconds)

Labs: Using Centrify to Enable Kerberos Security on your Hadoop Linux Cluster

This video outlines the steps described in this post:  Using Active Directory and Centrify to Accelerate your Linux-based Hadoop Big Data Deployments

We use Hortonworks Ambari 1.6.x and a two-node cluster.

Labs: Automating joins/removals with Kerberos Keytabs

This is the companion video to this post:  http://centrifying.blogspot.com/2014/08/using-kerberos-keytabs-and-centrify.html

Using Active Directory and Centrify to Accelerate your Linux-based Hadoop Big Data Deployments

Update

Updated whitepapers from August 2015:

• Cloudera (5.4.3 and Centrify 2015.1): http://goo.gl/zWoBZE
• Hortonworks: http://goo.gl/uzGPGG
• MapR (4.1 and Centrify 2015.1): http://goo.gl/0uLiHF

This playlist also explains the challenges at the OS Level:


*********************************************************************************
As of February 2015 Centrify is gearing-up to release Centrify Suite 2015, with it, enhancements specific to Hadoop deployments are being released.  To complement these efforts, integration guides for Cloudera, Hortonworks and MapR are being released, here are the first two:

Centrify Identity and Access Management for MapR
Centrify Identity and Access Management for Cloudera

Centrify Identity and Access Management for Hortonworks Data Platform

These are more robust and well-researched papers.  The article below from August 2014 will be left there for history purposes, but the post and the accompanying video outdated.
*********************************************************************************

Background

Big Data is one of the fastest spreading IT trends in the enterprise today, it's also a big reason why IT infrastructures have to provide elastic and secure infrastructures. Apache Hadoop (with value-added by Cloudera, Hortonworks, MapR, IBM and others) is gaining a lot of traction in enterprises that are looking to adopt Big Data.

What does this mean to the IT Infrastructure Manager?

• More Linux servers
• Need for more effective management
• More systems that need to be aligned with security practices
• Depending data classification,  a mature access management model that allows the enforcement of the principles of least access, least privilege, separation of duties, quick attestation and optionally session capture and replay is required.

Customers of Centrify know that all the bullets above are not an issue with properly Centrified (*) systems.

(*) A mature Centrify deployment has no systems in Express mode,  has discontinued the use of Classic zones and archaic provisioning methods like ZoneGen, and finally is using RBAC via Centrify-enhanced sudo.

Recommendations:

• If you are not familiar with terms like Kerberos authentication, keytabs, servicePrincipalName (SPN), userPrincipalName (UPN) and some of the differences between MIT Kerberos and Microsoft Kerberos, I recommend that you familiarize with those topics.
• From a Hadoop perspective, leverage Hortonworks/Cloudera/MapR/IBM's professional services organizations.  Hadoop is a broad topic and in my observation successful implementations have one thing in common:  they get expert help.  You are busy enough as it is just keeping-up with the day-by-day.
• Keep in mind, the reason why we introducing Active Directory as an alternative to a stand-alone MIT Kerberos is for a simple reason: to eliminate duplication of capabilities.  If you go the MIT Kerberos route, this means that you have to maintain two different technology environments + process + people implications. This just doesn't make sense.
• If you feel comfortable with the topics above, you at least need to understand what the adkeytab utility is.  This post covers adkeytab.

Centrify & AD = Faster Hadoop Implementation

If you've been following this blog, you probably realize the focus on Access Controls and to leverage your existing infrastructure (AD);  this extends to Hadoop deployments as well.  Because:

• You already would have solved the Access and Privilege governance model at the OS level.
• You don't need to add more complexity to your environment to enable Hadoop Security.

If you don't have a mature access and privileged model like local accounts or shared accounts, you don't want to continue to propagate the problem.

Hadoop Infrastructure Security 101

InfoQ does a much better job in this article discussing Hadoop's security evolution.  I will focus on a small piece: internal authentication.

Hadoop leverages MIT Kerberos to enable security by Kerberizing its services.  With Kerberos no passwords go through the wire and there are compensating controls for confidentiality, integrity and other threats.    

What is the impact?

Hadoop providers will ask you to stand-up an MIT Kerberos environment to support your BigData deployment;  a very simple ask, but with a large impact to the enterprise.  Things to take into account:

• Kerberos infrastructure - additional services that need to be highly available.
• Additional process - add/moves/changes of principals as the nodes expand/contract/change
• Potentially a Kerberos trust:  If you want to leverage AD, you may need into the business of a Kerberos trust into AD.  At least one provider has a recipe for this.
• Push-back:  Believe it or not, the idea of an additional authentication infrastructure is not going to be welcome by some IT groups.

This is where Centrify can help.  Centrify provides the Kerberos environment and tools, services and utilities to extend your access model and accelerate the Hadoop deployment.  With that in mind, let's apply the Plan-Do-Check-Adjust model.

Note:  Keep in mind, Hadoop follows the MIT kerberos implementation to the letter. This means that there's no concept of multiple service principal names tied to an account (Unlike AD), therefore as of today, that implies that some services will require one account per service per node.   I repeat now in AD terms:  One AD Account/Keytab, Per Service, Per Node.  This is where utilities like adkeytab can help in the automation process.

Number of Accounts (Keytabs) = Shared Service Accounts + (Nodes * Unique Service Accounts)

E.g.  If you have a 50-node cluster that has 2 services that can share the same account/keytab and 5 services that require unique accounts/keytabs, this means that you will have 2 + 50*5 = 252 AD accounts/unique keytabs.

Believe it or not, the easy part is the technology, the hard part is the process and the automation required.

Planning to Secure your Hadoop Services with Centrify

Infrastructure vs. User Apps: 

Kerberizing the servies like HDFS, MapReduce, Hive, Zookeeper, etc is just one piece of the puzzle, the user-level apps may have different access models.  Many use OS users and groups and Centrify will have you covered, but this is a different design session.

• What is the access model at the OS level? (may imply a different zone or set of computer groups)
• What are the privileges (commands) for a Hadoop operator? (remember, with Centrify there's no need to share privileged account credentials) - continue to enforce the least privilege model with Centrify RBAC
• Just like the Centrify agent, deploying Name Server Cache Daemon can improve performance.
• Have a rock-solid naming convention for:
• AD account names:  remember the 20 character limit.  Make sure that you can identify these accounts correctly in AD.   Use a dedicated OU if you have a large cluster.  Something like Environment-server-service general-to-specific can work.  E.g. "mktqa-had01-spnego"
• Keytab location and naming:  
• Keytab protocol:
• Secure based on requirements at rest.
• Always transport securely
• Keep keytabs where they are needed ONLY.
• Will the user accounts in AD have non-expiring randomized passwords or will the passwords be randomized after a period of time?
• Does the account used for AD Service account/keytab generation have the proper rights in AD to create those users/principals?
• What will be the automation strategy?  Remember, using cleartext passwords in scripts or other facilities is a NO-NO, but adkeytab is kerberized;  therefore you may need to provision a service account with the proper rights in AD and a keytab to be able to launch adkeytab in the automation script.
• How are you implementing Hadoop?  Is your team being trained?  Succesful implementations (just like with Centrify) make use of consulting services;  so leverage your Hortonworks, Cloudera or IBM experts.  After all, how many Hadoop (or Centrify) deployments have you implemented?  How much pure Kerberos knowledge do you have?  

Implementation

In this example, we'll use Hortonworks Hadoop Ambari 1.6.x to enable Kerberos in a cluster.  The domain is corp.contoso.com, the OU for service accounts is UNIX\Hadoop and the naming convention is servername-servicename.

1. Set up your system and join it to AD.  Make sure you don't register an SPN for http because it will be used by Hadoop.  This can be done:
   - Prior to the join:  by editing the /etc/centrifydc/centrifydc.conf and modifying the adclient.krb5.service.principals  directive and removing http.  E.g.
   - After the join: Remove the SPN with  ADUC or with adkeytab with the -x (delspn) option.
   adkeytab --delspn --principal http/hadoop1.corp.contoso.com -V
   Remember that you can check the service principals with the "adinfo -C" command.
2. Set up your cluster and make sure your services are running.
3. Go to Administration-Security and Click Enable Kerberos.
   
4. Click Next in the Getting started page.
5. In the Configure Pages page type the AD domain name in all caps in the Realm Name and  in the Kerberos tool path, type the Centrify-enhanced Kerberos tools path (/etc/centrifydc/kerberos/bin).
   
6. In the Create Principals and Keytabs page, you'll be shown a table with all the principals per host based on your configuration.
   
   Ambari provides a CSV file that can be consumed by a keytab-generating script.  In the case of Centrify, you can leverage adkeytab to perform these operations.  Here are a few examples of what needs to be done.  Note:  This example assumes that the person running adkeytab has the rights to create the user principals in Active Directory or in the respective container.
1. Shared Keytab Example (Ambari)
   Let's analyze this.
   a) We will be creating a new account (keep in mind that we can adopt an existing principal) (option --new)
   b) We will be specifying a UPN (because this keytab will be used to get a Kerberos TGT) (option --upn;  e.g. ambari-qa@CORP.CONTOSO.COM)
   c) We will create a keytab file (option --keytab /path/to/file)
   d) The AD account will be located in a particular OU  (option --container <x.500 notation>).  E.g. --container "ou=service-accounts,ou=Unix"
   e) Verbose output is always recommended (-V)
   f) The final parameter is the dn (distinguished name) of the account in AD.  (e.g. hadoop-ambari-qa)
   Sample command:
   adkeytab --new --upn ambari-qa@CORP.CONTOSO.COM --keytab /etc/security/keytabs/smokeuser.headless.keytab --container "ou=hadoop,ou=unix" -V hadoop-ambari-qa
   Then, the keytab has to be permissioned accordingly: 
   chown ambari-qa:hadoop /etc/security/keytabs/smokeuser.headless.keytab
   chmod 440 /etc/security/keytabs/smokeuser.headless.keytab
2. Individual Host Keytab Example (HTTP for hadoop2)
   Analysis
   a) We will be creating a new account (keep in mind that we can adopt an existing principal) (option --new)
   b) We will be specifying a UPN and an SPN (because this keytab will be used to get a Kerberos TGT and a TGS) (options --upn & --principal;  e.g. HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM)
   c) We will create a keytab file (option --keytab /path/to/file)
   d) The AD account will be located in a particular OU  (option --container <x.500 notation>).  E.g. --container "ou=service-accounts,ou=Unix"
   e) Verbose output is always recommended (-V)
   f) The final parameter is the dn (distinguished name) of the account in AD.  (e.g. hadoop-ambari-qa)
   Sample command:
   adkeytab --new --upn HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM --principal HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM --keytab /etc/security/keytabs/spnego.service.keytab --container "ou=hadoop,ou=unix" -V hadoop2-http
   Then, the keytab has to be permissioned accordingly: 
   chown root:hadoop /etc/security/keytabs/spnego.service.keytab
   chmod 440 /etc/security/keytabs/spnego.service.keytab
7. Verify that the principals have been created in AD and that the keytabs are in the selected directory with the proper ownership and permissions.  In addition, you can use the /usr/share centrifydc/kerberos/bin/kinit -kt command to test the principals for TGT or TGS requests.
   
   
8. Go back to Ambari's security wizard and Apply the changes.
   
9. Monitor the services, depending on the performance of your cluster, you may have to start some services manually.

Checking the Implementation

Aside from checking the cluster status, if you're using scripts to automate the add/move/changes of nodes, you need to make sure that those scripts are rock solid.  

Adjusting the Implementation

There are many improvements to be gained, and this depends on your security needs and the variety of environments.  The most notable is the user-facing apps.  Remember that as new environments are spun up and versions of Hadoop change, this process has to be revisited.

Using Kerberos keytabs and Centrify tools to automate UNIX/Linux/Mac AD domain joins or unjoins

The Problem

Dynamic environments expand and contract based on organizational needs; this means that Unix, Linux, Mac OS X servers and workstations are built and decommissioned frequently.  Most recently, public/private cloud elasticity accentuates this issue and having an access controls solution like may end-up adding unnecessary complexity if it's not designed for this reality.

When joining a computer to the domain, the computer name (hostname) may not be known to pre-create the AD account, therefore an an authoritative join is the only choice; the issue here is that an AD account that has the ability to join computers into the domain (OU) and into the Centrify zone is required.

Fortunately, Centrify designed the Server Suite solution tool set with this reality in mind.  In this post, we will discuss how to add or remove systems from AD securely leveraging Kerberos keytabs and tools like adjoin and adleave.  Like all small projects, we'll use the Plan-Do-Check-Adjust methodology.

Requirements

• Scripts should not store plain-text passwords
• Tools must be automation-friendly
• The least access and least privilege principles must be conserved
• The Separation of Duties (SoD) principle must be met

Tools

• Active Directory Users and Computers
• Centrify CLI tools: adjoin, adleave, adkeytab
• Kerberos tools:  kinit, kdestroy.

Planning

For a simple example, we'll consider the following planning steps:

• Incorporate the Centrify agent bits into the infrastructure image, the agent can be installed and not joined.
• An AD Service account with the ability to create, remove (or modify) computer objects to the target domain OU should be created.
• That same AD service account should have the rights to join, remove and modify objects in the target Centrify zone.
• A Kerberos keytab file needs to be created and securely put in a place where the script can use it.  Let's assume that the file will be securely copied to a local drive and deleted upon use.
  Note:  this is important, a Kerberos key table file needs to be to be treated with the same sensitivity as a private key.  They are not to be left behind on systems even if the account has been properly secured.
• In addition to the keytab file, a krb5.conf file with the correct settings needs to be deployed to the unjoined system so the Kerberos tools can find a KDC (AD DC)
• Naming conventions for the service account and the hostname need to be pre-established.  Keep in mind that computer account names have length limitations.
• An AD OU for Unix/Linux or Mac computers is required to limit the scope of where the join account can perform joins.

Implementation (Do)

AD Join service account setup (1-time steps)

1. Create the Active Directory service account by using ADUC.  Make sure that the account and its password do not expire.  In this example we'll use "ad-joiner" and the domain is corp.contoso.com
2. In Active Directory users and computers, use the Delegate Control wizard to delegate the ability to create and delete computer objects.
   
3. In the target Centrify zone, use the Delegate Zone Control wizard to give the service account the rights to join, remove and modify computers in the zone.
   
4. In a Centrified system, use the adkeytab with the adopt option to create the Kerberos keytab file and randomize the password.  This will be performed with the root account so it is protected by that account.
   
   # /usr/sbin/adkeytab --adopt --user jerry.seinfeld--keytab ad-joiner.keytab  -V ad-joiner
   ADKeyTab version: CentrifyDC 5.1.3-482
   Options
   -------
   use machine ccache: no
   domain: corp.contoso.com
   server: null
   user: dwirth
   container: null
   account: ad-joiner
   trust: no
   des: no
   jerry.seinfeld@CORP.CONTOSO.COM's password:
   Attempting bind to corp.contoso.com site:Demo-Site server:dc1.corp.contoso.com: ccache:MEMORY:0x566940
   Bind successful to server dc.centrifyimage.vms
   Searching for AD Object: filter = (samAccountName=ad-joiner), root = DC=corp,DC=contoso,DC=com
   AD Object found: CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp, DC=contoso,DC=com
   Key Version = 2
   Activating AD account: CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp, DC=contoso,DC=com
   Account 'CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp, DC=contoso,DC=com' All SPNs already present
   Adding managed account keys to configuration file: ad-joiner
   Changing account 'ad-joiner' password with user 'jerry.seinfeld@CORP.CONTOSO.COM' credentials.
   Searching for AD Object: filter = (samAccountName=AD-JOINER), root = DC=corp,DC=contoso,DC=com
   AD Object found: CN=ad-joiner,OU=Service Accounts,OU=Unix,DC=corp,DC=contoso,DC=com
   Key Version = 3
   Success: Adopt Account: ad-joiner
5. Verify the keytab file with klist
   # /usr/share/centrifydc/kerberos/bin/klist -kt ad-joiner.keytab
   Keytab name: FILE:ad-joiner.keytab
   KVNO Timestamp         Principal
   ---- ----------------- ----------------------------------
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM
   3 08/2414 22:22:55 ad-joiner1@CORP.CONTOSO.COM

Checking the Implementation (Do)

In order to verify that the keytab file works and can join or remove a system, you need an unjoined Unix, Linux or Mac system with the Centrify agent installed.

1. Log into the system with a local account (that can elevate)
2. Make the keytab file securely accessible.
3. Copy the /etc/krb5.conf file from a working Centrified system to the local system:
   scp <account>@centrified.system:/etc/krb5.conf /etc/krb5.conf
4. Use the kinit command with the kt option to get a ticket-granting-ticket as the ad-joiner account.
   /usr/share/centrifydc/kerberos/bin/kinit ad-joiner -kt ad-joiner.keytab
5. Use the adjoin command without specifying the user option.  Adjoin is Kerberized and it will use the ad-joiner's ticket-granting ticket.
   # adjoin -z Model -c "ou=servers,ou=unix" corp.contoso.com
   Using domain controller:  dc2.corp.contoso.com writeable=true
   Join to domain: corp.contoso.com, zone: Model succesful
   
   Centrify DirectControl started
   Initializing cache
6. At this point the computer has joined the domain.
7. You can use it for the reverse with adleave.
   # adleave -r 
   Using domain controller:  dc2.corp.contoso.com writeable=true
   Left domain.
   Centrify DirectControl stopped.

Adjusting the Process

There are many opportunities to adjust this process and make it better.  Here are some examples:

• The process can be part of a script that specifies things like zones, computer roles, etc.
• ADEdit can be used to pre-create the account in the zone and move the computer to the proper computer role.
• The keytab can be deleted from the local host as part of the script termination process process

Video

Security Corner: What is a Kerberos Keytab and why should you use it?

Background

Active Directory provides a Kerberos environment.  We have written extensively about Kerberos capabilities in this blog.  We also have explained that Centrify provides tight integration with Microsoft's Kerberos AD implementation by way of their MIT-Kerberos libraries and tools.  The advantage of using Centrify's tool set is that the tools are extensively tested against Microsoft's implementation.

A Kerberos key table (or "keytab") file is "is a file containing pairs of Kerberos principals and encrypted keys (these are derived from the Kerberos password)."(1).

Kerberos keytab files can help overcome two major issues:

• The security requirement of not having plain-text passwords in scripts or helper files.
• The increasing IT Infrastructure requirement for automation.  The proliferation elastic computing by way of private and public clouds requires that IT adds or reduces capacity on demand in a secure way.

Kerberos Keytabs 101

• In a Kerberos environment, each system has at least one keytab table stored on disk.  The keytab table lists the service principals and provides at least one key for each of those service principals (/etc/krb5.keytab by default).  Remember that principals typically follow the "service@host/REALM" format.
• End-users also have Kerberos files, but they are temporary (by default 10 hours), they are stored as a Kerberos cache file (by default /tmp/krb5cc_<uid of the user>), they list any user principals.
• When working with a Centrified system, the location of the Centrify-enabled Kerberos tools is /etc/centrifydc/kerberos/bin.  
• Quick primer on basic Kerberos tools:
• kinit: is used to obtain a kerberos ticket-granting-ticket
• klist: is used to list the cached tickets.
• kdestroy: destroys kerberos tickets
• The Centrify Kerberos tools documentation is publicly available here.
• Keytabs are Sensitive/Confidential information: A Security Procedure needs to be established for the handling of keytabs, because whoever has the keytab can authenticate as the principal used to create it.  As a tip, this process should be viewed with the same sensitivity as the handling of private keys.

Using Kerberos Tools

Kerberos tools are great for troubleshooting purposes.

1. Testing end-to-end AD connectivity and account availability with kinit (e.g. accunt jerry.seinfeld)
   # /usr/share/centrifydc/kerberos/bin/kinit jerry.seinfeld
   Password for jerry.seinfeld@CORP.CONTOSO.COM:
   You are prompted for the user's password.  The benefit of this test that it bypasses the NSS, PAM and authorization modules.  This means that any user in AD can be tested with kinit.
2. Listing the contents of a keytab or cache file with klist
1. For systems
   # /usr/share/centrifydc/kerberos/bin/klist -kt krb5.keytab
   Keytab name: FILE:krb5.keytab
   KVNO Timestamp         Principal
   ---- ----------------- ----------------------------------
   3 05/17/14 22:22:55 host/cen1@CORP.CONTOSO.COM
   
   Note: By default, during a system join (with adjoin) Centrify will automatically register service principals for  http ftp cifs nfs, this parameter is controlled by the adclient.krb5.service.principals directive of the centrifydc.conf file.
2. For users (user root)
   /usr/share/centrifydc/kerberos/bin/klist
   Ticket cache: FILE:/tmp/krb5cc_0
   Default principal: jerry.seinfeld@CORP.CONTOSO.COM
   
   Valid starting     Expires            Service principal
   08/24/14 14:07:02  08/25/14 00:07:04  krbtgt/CORP.CONTOSO.COM@CORP.CONTOSO.COM
   renew until 08/25/14 14:07:02
   Notice that this is a Kerberos Ticket Granting Ticket (krbtgt).

Ticket-Granting-Tickets vs. Service Tickets

Depending on how the application is using the keytab, it's possible that it will request a ticket-granting ticket (TGT), or a service ticket.  Generally, a TGT is given upon a session log-in;  a service ticket is requested when a user attempts to access a service on the network.  The key here is that a principal without a servicePrincipalName (SPN) can't be used to request a service ticket.  This changes how the AD principal is created.

The example above shows how to request a TGT.  This example, shows how to test a keytab for a service ticket:

First, we check if the account has a servicePrincipalName (e.g. db2inst1)

$ adquery user -A db2inst1

dn:CN=db2inst1,CN=Service-Accounts,DC=centrifyimage,DC=vms

samAccountName:db2inst1

userPrincipalName:db2inst1@CENTRIFYIMAGE.VMS

servicePrincipalName:db2inst1/engcen5.centrifyimage.vms

Second, we attempt to get a service ticket (the keytab is db2inst1.keytab)

$ /usr/share/centrifydc/kerberos/bin/kinit -S db2inst1/engcen5.centrifyimage.vms -kt db2inst1.keytab db2inst1

[db2inst1@engcen5 ~]$ /usr/share/centrifydc/kerberos/bin/klist

Ticket cache: FILE:/tmp/krb5cc_1004

Default principal: db2inst1@CENTRIFYIMAGE.VMS

Valid starting     Expires            Service principal

11/24/14 11:24:46  11/24/14 21:24:47  db2inst1/engcen5.centrifyimage.vms@CENTRIFYIMAGE.VMS

        renew until 11/25/14 11:24:46

Notice that this is not a KRBTGT, but a service ticket.

Using Centrify Tools for Kerberos-related operations

adinfo (with the C option) can be used to review the system service principals, and encryption level. Remember that the latter is set by the domain controller functional level.
$ adinfo -C
Computer Account Diagnostics
  Joined as: cen1
  Trusted for Delegation: false
  Use DES Key Only: false
  Key Version: 6
  Service Principal Names: nfs/cen1.corp.contoso.com
                           nfs/cen1
                           ipp/cen1.corp.contoso.com
                           ipp/cen1
                           http/cen1.corp.contoso.com
                           http/cen1
                           host/cen1.corp.contoso.com
                           host/cen1
                           ftp/cen1.corp.contoso.com
                           ftp/cen1
                           cifs/cen1.corp.contoso.com
                           cifs/cen1
                           afpserver/cen1.corp.contoso.com
                           afpserver/cen1

Supported Encryption Type(s): RC4-HMAC
                              AES128-CTS-HMAC-SHA1-96
                              AES256-CTS-HMAC-SHA1-96

Operating System Version: 6.1:6.5 (Final)

Adkeytab

adkeytab, is arguably one the most powerful command line tools provided by Centrify.  When used in tandem with scripts and the Kerberos tools it can help to increase security and provide flexibility in automation scenarios.  It has the ability to provision, deprovision and modify server and user principals.  Aside from adedit, adkeytab contains the largest command reference.

adkeytab  -  create  and manage Kerberos key tables (*.keytab files)and coordinate changes with the  Kerberos  key  distribution center(KDC) provided by Active Directory.

I will try to highlight some benefits of using adkeytab:

• It can be used to create or adopt AD principals:  adkeytab can create the user in AD for you, or in a separation of duties scenario, the UNIX/Linux admin can adopt the account provided he has the appropriate rights to modify the AD account.
• It randomizes the AD account's password:  The major issue with service accounts it that they are considered a shared account since at least one person (the creator or the user) knows the password.  With adkeytab it's possible to randomize the password of the AD account, effectively minimizing the need for aggressive password rotation.
• Can be fully automated:  adkeytab can be used with your scripts to perform operations.  This helps with elastic or private clouds.
• It's Kerberized!  what that means is that you can use a keytab to use adkeytab - this is great for automation.
• Syntax can be tricky - there are so many options that it can be overwhelming for beginners; my tips are to use the long for options (e.g. --adopt) and verbose output.
• Fixes stale AD computer accounts:  It can be used to reset the machine computer account.  Just elevate to use the adkeytab command with the -C (--change-password).  The output below uses the local system's computer account (--machine) in AD to reset the computer's password in AD.

$ dzdo adkeytab --change-password --machine -V

Attempting bind to centrifyimage.vms site:Demo-Site server:dc.centrifyimage.vms: ccache:FILE:/etc/krb5.ccache

Bind successful to server dc.centrifyimage.vms

Searching for AD Object: filter = (samAccountName=engcen5$), root = DC=centrifyimage,DC=vms

AD Object found: CN=engcen5,OU=UNIX-Servers,OU=Unix,DC=centrifyimage,DC=vms

Key Version = 21

Changing account 'engcen5' password with machine credentials.

Searching for AD Object: filter = (samAccountName=engcen5$), root = DC=centrifyimage,DC=vms

AD Object found: CN=engcen5,OU=UNIX-Servers,OU=Unix,DC=centrifyimage,DC=vms

Key Version = 22

Success: Change Password: Default Key Tab

The posts below, cover scenarios that use Kerberos key table files:

• Big Data (Hadoop)
• DB2 AD Integration and SSO
• MongoDB AD-Integration with Centrify
• Automating system AD joins/removals

(1) Source: Indiana University IT, https://kb.iu.edu/d/aumh