In the previous post we discussed how to leverage the Kerberized environment provided by Centrify's UNIX/Linux client and the computer's AD account to access a read-only Windows (CIFS) share. By leveraging Kerberos, the mount command (or filesystems table) does not have to use a cleartext password.
Windows (CIFS) shares are all over the Enterprise (for files and printers) and most likely a read-only share is not a very practical example, however, a departmental share that is used by a workgroup to share files is much more common. This time, we will implement an example.
The Marketing Share
In our mock example we have a share in APP1 set for the Marketing department. Both Elaine and J.Peterman are members of this department, however they consistently access these shares from OS X, Unix and Windows machines. Today a Marketing Share Access AD group is used to control access by way of memberships.
Requirements:
Marketing users should have the ability to exchange files (read/write)
The current heterogeneous environment consists of Windows, Mac OS X and UNIX/Linux stations
Users should be able to access the information from any type of platform
Controlling access to the Windows share should not deviate from the current process.
Users should alingn with the security policy and don't use cleartext passwords when mounting CIFS shares.
Challenges:
This would not be an issue in a homogeneous environment
Identity is key, UNIX identities should be uniform across platforms (for users and groups)
Users should not have access to the root account to use the mount command
The Proposed Solution:
Centrify the non-Windows platforms (Mac OS X, Unix/Linux) - this provides the AD Integration, Kerberos environment and Privilege Elevation
Leverage a common UID/GID scheme by leveraging Zones and the Autozone
UNIX-enable the Matkeing Share Access AD group.
On UNIX Platforms, allow the Marketing users to use the mount command with privileges
Video Labs
Lab Part I - Initial Setup (13.30 min): https://www.youtube.com/watch?v=flp4L200o4c
Lab Part II - Verification (11 min): https://www.youtube.com/watch?v=M-fdtZeYYtk Elaine's mount command (CentOS 6.5): dzdo mount -t cifs //app1.corp.contoso.com/marketing /mnt/share -o sec=krb5,user=elaine.benes,uid=1149240407,gid=99998,group=marketing-share-access,umask=0002,file_mode=0775,dir_mode=0755 --verbose Share Permissions: (this can be improved since the rest of the world would get read access given that ACL) NTFS Permissions: Role created for Marketing Users
We can improve on this by using Automount to access the user's Windows home.
In the previous post "What is a Kerberos Keytab and why should I use it?" we discussed that once you have a Centrified system, Kerberos-enabled applications can leverage this authentication mechanism. The key benefit being that the Centrify agent (adclient) automatically maintains the Kerberos environment (e.g machine keytabs and krb5.conf files are updated based on the AD sites and services info) and both the OS and Centrify offer an array of "Kerberized" apps to help the most seasoned Sysadmins.
In this post I will discuss some popular use cases:
Using the Centrify DNS utility (addns) to leverage Active Directory DNS Dynamic Updates
Eliminating cleartext passwords from the mount command or from /etc/fstab
Using AD DNS Dynamic Updates and addns to keep your DNS entries up to date
One of the coolest Centrify tools is addns. As discussed in an earlier post, this utility can help in performing dynamic or manual updates to DNS registrations for Centrified systems. Think about the "ipconfig /registerdns" command on Windows. This is very useful especially when servers are being built with static IP addresses an in a Mixed UNIX/Linux and Windows environment.
How does it work?
As per TechNet: "Domain Name System (DNS) client computers can use dynamic update to
register and dynamically update their resource records with a DNS server
whenever changes occur. This reduces the need for manual administration
of zone records, especially for clients that frequently move or change
locations and use Dynamic Host Configuration Protocol (DHCP) to obtain
an IP address.
Dynamic updates can be secure or nonsecure. DNS update security
is available only for zones that are integrated into Active Directory
Domain Services (AD DS). After you directory-integrate a zone, access
control list (ACL) editing features are available in DNS Manager so that
you can add or remove users or groups from the ACL for a specified zone
or resource record."
This means that in cases in which there are workstations (like Macs or Linux) that change networks frequently or when servers are being provisioned and there's no available Windows DNS administrator, addns can be a very useful tool.
The addns utility supports the -m (or --machine) switch, this means that instead of providing a valid AD account to update the DNS records for a computer, you can use the computer account's credentials instead. The key benefit here is that there's no need to type credentials.
[root@engcen7 ~]# cat /etc/redhat-release CentOS Linux release 7.0.1406 (Core) [root@engcen7 ~]# adinfo -v adinfo (CentrifyDC 5.2.0-218) [root@engcen7 ~]# addns -U -m Updating host records for engcen7.centrifyimage.vms on 192.168.81.10. Updated host records engcen7.centrifyimage.vms.
Updating reverse lookup records for engcen7.centrifyimage.vms on dc.centrifyimage.vms.
Updated reverse lookup record 167.81.168.192.in-addr.arpa.
Alternatively, if you know the credentials of an AD DNS administrator (or have a keytab), you can kinit and launch addns without the need to specify the --user (or -u) switch:
[root@engcen7 ~]# /usr/share/centrifydc/kerberos/bin/kinit dwirth Password for dwirth@CENTRIFYIMAGE.VMS: [root@engcen7 ~]# addns -U Updating host records for engcen7.centrifyimage.vms on 192.168.81.10. Updated host records engcen7.centrifyimage.vms. Updating reverse lookup records for engcen7.centrifyimage.vms on dc.centrifyimage.vms. Updated reverse lookup record 167.81.168.192.in-addr.arpa.
Potential Uses:
Troubleshooting DNS connectivity for a roaming computer (Linux or Mac)
Automated provisioning of IP addresses during Server provisioning.
Keep in mind, all the Centrify "ad" tools that take the -u (--user) option are Kerberized, including adjoin, adleave, addns, adcert, adsmb, adkeytab, etc; so your scripts don't require interactive attention or the use of cleartext passwords!
Eliminate Cleartext Passwords from the Mount Command or the filesystems table
The mount command traditionally lends itself for the use of cleartext passwords. Although Centrify has nothing to do with this command (there's a great degree of variability based on the distribution); the key continues to be Kerberos.
Let's pick on this example. The idea is to use the mount command from a Unix or Linux system to mount a CIFS share that is on a NAS device or Windows Share.
Both the Mount Syntax # mount -t cifs //ntserver/download -o username=vivek,password=myPassword /mnt/ntserver
and the fstab entry
specify the use of a cleartext password. Let's improve on this example by leveraging a Centrified system in the modified Microsoft Test Lab Guide. The idea is to leverage the AD Kerberos environment, the Centrify client, the machine account and the user account to eliminate the use of the cleartext password.
Using the Machine Account to access a Read-only Share
On APP1 - Assign permissions to the Unix/Linux system
Open Windows Explorer and go to the root of the C: drive.
Create a new folder and call it 'mysmb'
Right-click the folder > Properties > Sharing > Advanced Sharing > Permissions and set: Authenticated Users : Read Domain Admins : Full Control
Right-click the folder > Properties > Security > Edit > and Add Select Type: Computers CEN1$ (or your centrified system) : Read
Create a a text file in the folder (myfile.txt) and put some text in it.
On CEN1 (or your Centrified system) - Test your mount
Note: Make sure the cifs-utils package is installed on RHEL derivatives.
Log on with a user that has the ability to elevate.
Use the following command: mount -t cifs //app1.corp.contoso.com/mysmb //mnt/share -o sec=krb5
Test your share.
This mapping is done right without the need of cleartext passwords.
The active session comes from the Linux computer.
Because the mapping has been done with the Kerberos ticket of the machine account and typically per policy, these only last 10 hours; a mechanism to renew the ticket for the lifetime of the mount has to be implemented. For example, a cron job that renews the ticket before it expires like this:
Runs every day at midnight, 8AM and 4PM. This assures
there’s a fresh ticket every 8 hours.
Crontab entries can be controled using Group Policy
In addition, and in support of Hadoop, Centrify has come up with another set of GPOs to help with Kerberos ticket renewal. This parameter can be specified in the configuration file or via GPOs:
Infinite ticket renewal GPOs
This sets up the building blocks for our next entry. Leveraging UNIX-enabled AD groups to control access to a Windows share leveraging Centrify.
I'm very excited about the new update to Centrify Suite 2014. This is a maintenance update that brings in support for RHEL7 and Debian 7.5, 7.6 and derivatives of these popular platforms. It includes additional updates, but what really makes it exciting to me is a feature available in the agent for Mac OS X called Combo Join.
Combo join marks an interesting milestone that affects this blog. The merging of Centrify on-premise and Cloud offerings. Combo join addresses the needs of employees with the top-of-line AD integration and Bring Your Own Mac Scenarios for third parties (like contractors or partners).
A simple check-box = more management capabilities
Combo-join means having the ability to register a Mac both as an AD domain member and with the Centrify Cloud Service. This allows for two management possibilities:
The AD Group Policy-based Centrify agent GPOs (400+)
The Centrify Cloud Service policies that can be delivered via GPOs or via the cloud.
Architecturally they are different, one uses the mappers and the other users profile manager. Combo join is a simple check-box that extends the AD Join assistant:
Once joined to the cloud service, a profile is delivered that permits management via the cloud:
This means that now both IT and end-users (via the Centrify User Portal) can perform device management along with their iOS and Android devices and IT will be able to deliver configuration management in a different vehicle without requiring a hard domain join.
Users will be able to perform self-service device management
IT will have more flexibility delivering configuration management
Why is this a game changer?
From this point on, we need to include Centrify User Suite content in this blog and if I'm correct, this is just the beginning. Part of the reason why Centrify solutions are so easy to implement and are so cohesive is because they are developed in-house. Not having to deal with Frankenstein-like solution packages like some of the traditional vendors translates into faster deployments and an easier learning curve.
Expect a Cloud section with info about Server Suite plus sections on synergy between Mac, Server and User Suites.
This playlist also explains the challenges at the OS Level:
*********************************************************************************
As of February 2015 Centrify is gearing-up to release Centrify Suite 2015, with it, enhancements specific to Hadoop deployments are being released. To complement these efforts, integration guides for Cloudera, Hortonworks and MapR are being released, here are the first two:
These are more robust and well-researched papers. The article below from August 2014 will be left there for history purposes, but the post and the accompanying video outdated.
*********************************************************************************
Background
Big Data is one of the fastest spreading IT trends in the enterprise today, it's also a big reason why IT infrastructures have to provide elastic and secure infrastructures. Apache Hadoop (with value-added by Cloudera, Hortonworks, MapR, IBM and others) is gaining a lot of traction in enterprises that are looking to adopt Big Data.
What does this mean to the IT Infrastructure Manager?
More Linux servers
Need for more effective management
More systems that need to be aligned with security practices
Depending data classification, a mature access management model that allows the enforcement of the principles of least access, least privilege, separation of duties, quick attestation and optionally session capture and replay is required.
Customers of Centrify know that all the bullets above are not an issue with properly Centrified (*) systems.
(*) A mature Centrify deployment has no systems in Express mode, has discontinued the use of Classic zones and archaic provisioning methods like ZoneGen, and finally is using RBAC via Centrify-enhanced sudo. Recommendations:
If you are not familiar with terms like Kerberos authentication, keytabs, servicePrincipalName (SPN), userPrincipalName (UPN) and some of the differences between MIT Kerberos and Microsoft Kerberos, I recommend that you familiarize with those topics.
From a Hadoop perspective, leverage Hortonworks/Cloudera/MapR/IBM's professional services organizations. Hadoop is a broad topic and in my observation successful implementations have one thing in common: they get expert help. You are busy enough as it is just keeping-up with the day-by-day.
Keep in mind, the reason why we introducing Active Directory as an alternative to a stand-alone MIT Kerberos is for a simple reason: to eliminate duplication of capabilities. If you go the MIT Kerberos route, this means that you have to maintain two different technology environments + process + people implications. This just doesn't make sense.
If you feel comfortable with the topics above, you at least need to understand what the adkeytab utility is. This post covers adkeytab.
Centrify & AD = Faster Hadoop Implementation
If you've been following this blog, you probably realize the focus on Access Controls and to leverage your existing infrastructure (AD); this extends to Hadoop deployments as well. Because:
You already would have solved the Access and Privilege governance model at the OS level.
You don't need to add more complexity to your environment to enable Hadoop Security.
If you don't have a mature access and privileged model like local accounts or shared accounts, you don't want to continue to propagate the problem.
Hadoop leverages MIT Kerberos to enable security by Kerberizing its services. With Kerberos no passwords go through the wire and there are compensating controls for confidentiality, integrity and other threats.
What is the impact?
Hadoop providers will ask you to stand-up an MIT Kerberos environment to support your BigData deployment; a very simple ask, but with a large impact to the enterprise. Things to take into account:
Kerberos infrastructure - additional services that need to be highly available.
Additional process - add/moves/changes of principals as the nodes expand/contract/change
Potentially a Kerberos trust: If you want to leverage AD, you may need into the business of a Kerberos trust into AD. At least one provider has a recipe for this.
Push-back: Believe it or not, the idea of an additional authentication infrastructure is not going to be welcome by some IT groups.
This is where Centrify can help. Centrify provides the Kerberos environment and tools, services and utilities to extend your access model and accelerate the Hadoop deployment. With that in mind, let's apply the Plan-Do-Check-Adjust model.
Note: Keep in mind, Hadoop follows the MIT kerberos implementation to the letter. This means that there's no concept of multiple service principal names tied to an account (Unlike AD), therefore as of today, that implies that some services will require one account per service per node. I repeat now in AD terms: One AD Account/Keytab, Per Service, Per Node. This is where utilities like adkeytab can help in the automation process. Number of Accounts (Keytabs) = Shared Service Accounts + (Nodes * Unique Service Accounts) E.g. If you have a 50-node cluster that has 2 services that can share the same account/keytab and 5 services that require unique accounts/keytabs, this means that you will have 2 + 50*5 = 252 AD accounts/unique keytabs.
Believe it or not, the easy part is the technology, the hard part is the process and the automation required.
Planning to Secure your Hadoop Services with Centrify
Infrastructure vs. User Apps:
Kerberizing the servies like HDFS, MapReduce, Hive, Zookeeper, etc is just one piece of the puzzle, the user-level apps may have different access models. Many use OS users and groups and Centrify will have you covered, but this is a different design session.
What is the access model at the OS level? (may imply a different zone or set of computer groups)
What are the privileges (commands) for a Hadoop operator? (remember, with Centrify there's no need to share privileged account credentials) - continue to enforce the least privilege model with Centrify RBAC
Just like the Centrify agent, deploying Name Server Cache Daemon can improve performance.
Have a rock-solid naming convention for:
AD account names: remember the 20 character limit. Make sure that you can identify these accounts correctly in AD. Use a dedicated OU if you have a large cluster. Something like Environment-server-service general-to-specific can work. E.g. "mktqa-had01-spnego"
Keytab location and naming:
Keytab protocol:
Secure based on requirements at rest.
Always transport securely
Keep keytabs where they are needed ONLY.
Will the user accounts in AD have non-expiring randomized passwords or will the passwords be randomized after a period of time?
Does the account used for AD Service account/keytab generation have the proper rights in AD to create those users/principals?
What will be the automation strategy? Remember, using cleartext passwords in scripts or other facilities is a NO-NO, but adkeytab is kerberized; therefore you may need to provision a service account with the proper rights in AD and a keytab to be able to launch adkeytab in the automation script.
How are you implementing Hadoop? Is your team being trained? Succesful implementations (just like with Centrify) make use of consulting services; so leverage your Hortonworks, Cloudera or IBM experts. After all, how many Hadoop (or Centrify) deployments have you implemented? How much pure Kerberos knowledge do you have?
Implementation
In this example, we'll use Hortonworks Hadoop Ambari 1.6.x to enable Kerberos in a cluster. The domain is corp.contoso.com, the OU for service accounts is UNIX\Hadoop and the naming convention is servername-servicename.
Set up your system and join it to AD. Make sure you don't register an SPN for http because it will be used by Hadoop. This can be done: - Prior to the join: by editing the /etc/centrifydc/centrifydc.conf and modifying the adclient.krb5.service.principals directive and removing http. E.g. - After the join: Remove the SPN with ADUC or with adkeytab with the -x (delspn) option. adkeytab --delspn --principal http/hadoop1.corp.contoso.com -V Remember that you can check the service principals with the "adinfo -C" command.
Set up your cluster and make sure your services are running.
Go to Administration-Security and Click Enable Kerberos.
Click Next in the Getting started page.
In the Configure Pages page type the AD domain name in all caps in the Realm Name and in the Kerberos tool path, type the Centrify-enhanced Kerberos tools path (/etc/centrifydc/kerberos/bin).
In the Create Principals and Keytabs page, you'll be shown a table with all the principals per host based on your configuration. Ambari provides a CSV file that can be consumed by a keytab-generating script. In the case of Centrify, you can leverage adkeytab to perform these operations. Here are a few examples of what needs to be done. Note: This example assumes that the person running adkeytab has the rights to create the user principals in Active Directory or in the respective container.
Shared Keytab Example (Ambari) Let's analyze this. a) We will be creating a new account (keep in mind that we can adopt an existing principal) (option --new) b) We will be specifying a UPN (because this keytab will be used to get a Kerberos TGT) (option --upn; e.g. ambari-qa@CORP.CONTOSO.COM) c) We will create a keytab file (option --keytab /path/to/file) d) The AD account will be located in a particular OU (option --container <x.500 notation>). E.g. --container "ou=service-accounts,ou=Unix" e) Verbose output is always recommended (-V) f) The final parameter is the dn (distinguished name) of the account in AD. (e.g. hadoop-ambari-qa) Sample command: adkeytab --new --upn ambari-qa@CORP.CONTOSO.COM --keytab /etc/security/keytabs/smokeuser.headless.keytab --container "ou=hadoop,ou=unix" -V hadoop-ambari-qa Then, the keytab has to be permissioned accordingly: chown ambari-qa:hadoop /etc/security/keytabs/smokeuser.headless.keytab chmod 440 /etc/security/keytabs/smokeuser.headless.keytab
Individual Host Keytab Example (HTTP for hadoop2) Analysis a) We will be creating a new account (keep in mind that we can adopt an existing principal) (option --new) b) We will be specifying a UPN and an SPN (because this keytab will be used to get a Kerberos TGT and a TGS) (options --upn & --principal; e.g. HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM) c) We will create a keytab file (option --keytab /path/to/file) d) The AD account will be located in a particular OU (option --container <x.500 notation>). E.g. --container "ou=service-accounts,ou=Unix" e) Verbose output is always recommended (-V) f) The final parameter is the dn (distinguished name) of the account in AD. (e.g. hadoop-ambari-qa) Sample command: adkeytab --new --upn HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM --principal HTTP/hadoop2.corp.contoso.com@CORP.CONTOSO.COM --keytab /etc/security/keytabs/spnego.service.keytab --container "ou=hadoop,ou=unix" -V hadoop2-http Then, the keytab has to be permissioned accordingly: chown root:hadoop /etc/security/keytabs/spnego.service.keytab chmod 440 /etc/security/keytabs/spnego.service.keytab
Verify that the principals have been created in AD and that the keytabs are in the selected directory with the proper ownership and permissions. In addition, you can use the /usr/share centrifydc/kerberos/bin/kinit -kt command to test the principals for TGT or TGS requests.
Go back to Ambari's security wizard and Apply the changes.
Monitor the services, depending on the performance of your cluster, you may have to start some services manually.
Checking the Implementation
Aside from checking the cluster status, if you're using scripts to automate the add/move/changes of nodes, you need to make sure that those scripts are rock solid.
Adjusting the Implementation
There are many improvements to be gained, and this depends on your security needs and the variety of environments. The most notable is the user-facing apps. Remember that as new environments are spun up and versions of Hadoop change, this process has to be revisited.
Dynamic environments expand and contract based on organizational needs; this means that Unix, Linux, Mac OS X servers and workstations are built and decommissioned frequently. Most recently, public/private cloud elasticity accentuates this issue and having an access controls solution like may end-up adding unnecessary complexity if it's not designed for this reality.
When joining a computer to the domain, the computer name (hostname) may not be known to pre-create the AD account, therefore an an authoritative join is the only choice; the issue here is that an AD account that has the ability to join computers into the domain (OU) and into the Centrify zone is required.
Fortunately, Centrify designed the Server Suite solution tool set with this reality in mind. In this post, we will discuss how to add or remove systems from AD securely leveraging Kerberos keytabs and tools like adjoin and adleave. Like all small projects, we'll use the Plan-Do-Check-Adjust methodology.
Requirements
Scripts should not store plain-text passwords
Tools must be automation-friendly
The least access and least privilege principles must be conserved
The Separation of Duties (SoD) principle must be met
Tools
Active Directory Users and Computers
Centrify CLI tools: adjoin, adleave, adkeytab
Kerberos tools: kinit, kdestroy.
Planning
For a simple example, we'll consider the following planning steps:
Incorporate the Centrify agent bits into the infrastructure image, the agent can be installed and not joined.
An AD Service account with the ability to create, remove (or modify) computer objects to the target domain OU should be created.
That same AD service account should have the rights to join, remove and modify objects in the target Centrify zone.
A Kerberos keytab file needs to be created and securely put in a place where the script can use it. Let's assume that the file will be securely copied to a local drive and deleted upon use. Note: this is important, a Kerberos key table file needs to be to be treated with the same sensitivity as a private key. They are not to be left behind on systems even if the account has been properly secured.
In addition to the keytab file, a krb5.conf file with the correct settings needs to be deployed to the unjoined system so the Kerberos tools can find a KDC (AD DC)
Naming conventions for the service account and the hostname need to be pre-established. Keep in mind that computer account names have length limitations.
An AD OU for Unix/Linux or Mac computers is required to limit the scope of where the join account can perform joins.
Implementation (Do)
AD Join service account setup (1-time steps)
Create the Active Directory service account by using ADUC. Make sure that the account and its password do not expire. In this example we'll use "ad-joiner" and the domain is corp.contoso.com
In Active Directory users and computers, use the Delegate Control wizard to delegate the ability to create and delete computer objects.
In the target Centrify zone, use the Delegate Zone Control wizard to give the service account the rights to join, remove and modify computers in the zone.
In a Centrified system, use the adkeytab with the adopt option to create the Kerberos keytab file and randomize the password. This will be performed with the root account so it is protected by that account.
In order to verify that the keytab file works and can join or remove a system, you need an unjoined Unix, Linux or Mac system with the Centrify agent installed.
Log into the system with a local account (that can elevate)
Make the keytab file securely accessible.
Copy the /etc/krb5.conf file from a working Centrified system to the local system: scp <account>@centrified.system:/etc/krb5.conf /etc/krb5.conf
Use the kinit command with the kt option to get a ticket-granting-ticket as the ad-joiner account. /usr/share/centrifydc/kerberos/bin/kinit ad-joiner -kt ad-joiner.keytab
Use the adjoin command without specifying the user option. Adjoin is Kerberized and it will use the ad-joiner's ticket-granting ticket. # adjoin -z Model -c "ou=servers,ou=unix" corp.contoso.com Using domain controller: dc2.corp.contoso.com writeable=true Join to domain: corp.contoso.com, zone: Model succesful
Centrify DirectControl started Initializing cache
At this point the computer has joined the domain.
You can use it for the reverse with adleave. # adleave -r Using domain controller: dc2.corp.contoso.com writeable=true Left domain. Centrify DirectControl stopped.
Adjusting the Process
There are many opportunities to adjust this process and make it better. Here are some examples:
The process can be part of a script that specifies things like zones, computer roles, etc.
ADEdit can be used to pre-create the account in the zone and move the computer to the proper computer role.
The keytab can be deleted from the local host as part of the script termination process process
