Skip to content

Commit 69e27fc

Browse files
tristanDave BeechJim Halfpennykrishna kalyanDima Fadeyev
authored andcommitted
Merge master onto main (#176)
* sample definition for 5.16 build * updated 7.1 GBNs * initial README contents * add wrapper around uri module for CM API calls added built-in wait for commands but maybe needs timeout?? * add auto-tls role and playbook ported auto-tls role to new CM API plugin syntax version check and cluster loop moved auto-tls into security subfolder * pre-requisites updates * merged rngd role into OS prereqs * ignoring virtualenv * ignoring selinux failures for ycloud * cluster template fixes * custom repository role * deploy multiple clusters * fixed enterprise license upload Fixes #6 * added helper playbook for ycloud * added retry loop for kdc creds import * ignore inventories inside definition folders for now * HDFS transparent data encryption (#9) * Recreate the PostgreSQL template1 database and set charset to UTF-8 * re-organise definitions into subfolders * don't force config keys to lowercase fixes #10 * removed useless database configs * updated GBNs for 7.1.1 * Single node cluster for CDP-DC 7.1.1 (#12) * Fix example path in README (#17) * Updates for Active Directory integration (#15) * workaround for this issue: ansible/ansible#47838 * I've been getting this error when using 'raw' module: '/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory' * Added new variable krb5_ad_kdc_domain to pass the AD organizational unit * Changes suggested by @dbeech * "Express Wizard" style cluster definitions (#14) * PostgreSQL UTF-8 template (plus other misc bits) (#18) * Recreate the PostgreSQL template1 database and set charset to UTF-8 * Adding new cluster definition with as many services as possible enabled * added ca_server.yml * added default value for krb5_kdc_ad_domain * Ranger database config is now less "special" https://jira.cloudera.com/browse/OPSAPS-53943 * Added functionality to add messages into history and rollback (#20) * Fix for non-Kerberized deployments * Added new (very) basic cluster templates Only HDFS, YARN and ZOOKEEPER services included * Added custom_banner_html, custom_ia_policy, custom_header_color options (#16) * Added custom_banner_html, custom_ia_policy, custom_header_color options * Commented out the null values so it doesn't trigger the cosmetic customisations * reworked * Added custom_banner_html, custom_ia_policy, custom_header_color options * Commented out the null values so it doesn't trigger the cosmetic customisations * reworked * Set api_configs variable in install_cloudera_manager to import additional extra_vars configurations * Adjusted api_configs variable to enable additional cloudera_manager_options to be configured when CM is being configured * Updated extra vars with some minimally disruptive values * Fix for sending null values to config keys not previously set (#23) * DAS database type parameter added in 7.1 * Documentation updates (#19) * updates following testing * Active Directory integration (kerberos and CM external auth) (#24) plus work-in-progress LDAP integration for cluster services * High availability cluster definitions (#26) * Enable admin login for Atlas (#21) * CA server role updates * Example inventories for HDFS encryption installs * set hostnames to inventory name for ycloud deployments (#27) * Fix for when CM external auth role mappings are not defined * Config merging updates for mgmt service * cleanup * Privilege escalation for non-root playbook execution (#28) * simplified and fixed mysql connector role * added privilege escalation flags * added non-root user to ycloud node prep * fixed hdfs encryption parts * Inventory examples and docs * Updated 7.1 GBNs for testing * Updated 7.1 KTS GBN for testing * Example templates for Accumulo (#30) * added role mappings for accumulo * added example accumulo cluster definitions * TLS configuration (#32) * Fixed messed up variable precedence * Fixes for TLS configuration on 7.1 (#34) * Fixes for 7.1 TLS * Added TLS configs for stream processing * New stream processing example cluster (#35) with fixes for CSP database configuration * updates for providing manually signed certs * added TLS details in readme * Added Spark 1.6 to CDH 5.x templates Needed for Hive on Spark as reported by Bentley during testing * Added missing keystore ACL * Updated 7.1.1 GBNs to RC version * Added missing keystore ACLs * Fix for Sentry database on MariaDB/MySQL * Cloudera Streams Processing on CDH 6.x (#39) * re-worked role mappings * re-worked database configs for differences in CSP * added symlink workaround for SMM home dir * example template for CSP with CDH6 * Added back missing database config template for 7.1 * Updates to default templates 7.1 RC2 GBNs Added extra license configs where missing, and updated license file name Added default custom CM banner text * Fixes to CSP configs for 6.x * Updates for CDP-DC 7.1 GA Removed internal archive urls Changed default install CM version to 7.1.1 Added paywall creds for 7.1 examples Added vault encrypted file for passwords * Updates for CDP-DC 7.1 GA Updated parcel repositories * Removed invalid group from example hosts file * Cluster out-of-memory behaviour config slice (#40) * Configs and example template for CSA 1.2 (Flink) (#41) * Configs and example template for CFM 1.0.1 (NiFi) (#43) * Parcel repo and CSD updates for CFM 1.1 Fixes #44 * Added missing Flume service into C5 and 6 role mappings * added flatten_dict_list filter to root of project * removed show_intermediate bool from flatten_dict_list as it was unintuitive * removed unused dependencies in filters.py * Added pre-deployment verifications (wdyson/verify) (#45) * created structure for verify role * added group_by for host templates before verify/definition * added assertion checking that all templates in inventory match the definition * added check for KTS cluster when a kts_active group is present in the inventory * abstracted parcel metadata retrieval into a role and added assertions for cluster services and configurations * added host template service role checks * removed use of equalto as this is jinja2 version specific * Updates to verification playbook (#47) * Added assertion success and fail messages to help with debugging * Moved task names to assertions If put on "block" they don't actually get printed ansible/ansible#33581 * added checks for kts groups * fixed kts verification rules and added one for kms_servers * added verification rules to fail empty groups * added checks for kts groups * fixed kts verification rules and added one for kms_servers * added verification rules to fail empty groups * made verify permissive of undefined cluster.configs and cluster.services and added test for services when type is base * added tls check * added verifcation comparing ranger sentry and kerberos * Fixes following full verification run on example templates * Readme updates * Extra verification case for host template services * Fix broken check - skip for KTS clusters * PS-207 document verify (#51) * added basic docs for verify * added examples to parcels_and_roles.md and adjusted wording of definition.md * PS-203 kerberos verify (#50) * fixed bug causing host template service check to fail on kts clusters * added kerberos verifications * added success and fail messages to some asserts * README updates * Updated CSA 1.2 repository urls for GA * Configs and example template for WorkloadXM "on-prem" installation (#42) * WorkloadXM on-prem updates and example template Config params impala_daemon_host and phoenix_queryserver_host need somehow calculating from host_templates * mandatory parameters WIP * added new host grouping and updated config * added missing line break * Improve the documentation around Ansible Vault (#53) * Improve the documentation around Ansible Vault * remove decrypted comments * typo * corrected bool expression (#54) * Fixed typos in kerberos checks * Removed conflicting TLS variables * Added Sqoop2 for completeness * Added C5 specific wording to license check * Fixes for TLS deployment in 'become' mode * Security deployment fixes for C5.x (#58) * Fixes for standalone database server (PS-167) (#56) * Should only try to create a database and database user when database is being provisioned by the playbook itself * The 'groups.db_server | first' check breaks when 'db_server' group is not defined * Changed DB host default from omit to 'localhost'. When there is no 'db_server' group specified, and a default DB host is omitted, one would have to specify it in his cluster.yml even for services that are not being deployed (or install_cluster.yml fails) * Documentation updates (#55) * Split out getting started, security and troubleshooting guides * added new common issue to troubleshooting section * Fix to parcel metadata download - remove auth unless challenged * Fixed missing pre-reqs for non-cluster, tls enabled hosts * Reformatted WorkloadXM config slice template for readability * Fixed kerberos config when krb5_server inventory group is missing * Improvements to C6 secure cluster definition (#57) * New example of full secured 10 node cluster * typo fixed * replace secure folder with the new example including tls * Fix cluster definition path for 6.3.3/secure * removed full-secure directory * Verify user groups (#60) * moved user_account.yml out of role and added user groups verification * fixed smm home directory check * added tags to verify_definition.yml * added users verification tag * changed around labels so users verifcation tag is shown * added connection: ssh where required to fix ansible issue #70184 * Role mapping updates (#64) Added some CDH5 roles for completeness. Removed obsolete data context connector. Removed WebHCat role from Hive as per OPSAPS-57279 * Setting correct home directory permissions (#65) and gecos comment field like CM does, because why not? * New example template with external database config (#68) * Added example definition for using external databases * updates * Fixes for Cruise Control with TLS (#67) * recovery from transient network issues while polling * Example cluster definition for Spark 3 (#71) * Tls and database host (#74) * made changes to allow for different db hosts and external dbs * added check for CMS TLS * added flag to disable inventory tls check * corrected problems in definitions and fixed jmespath expression in group_hosts.yml * split verify to allow for some predeployment checks * added checks for the intermediate ca already existing * added agent validation and changed cms to configure all roles * fixed kts config overlay * fixed kts config overlay * changed mgmt template to configure all CMS roles even if they are not included * more connection: ssh changes * added check for existing kts cluster (similar to other cases) * fixed checks for intermediate ca files * added flatten_safety_valves filter * fixed merge * added additional kerberos and tls configs based on sec-scan * added secure-ha definition * added changes from sec-score run * fixed task message * corrected secure-ha * removed unused flatten_safety_valves filter * fixed typos * added aggregated playbook to run all verifications together * fixed more typos * added any_errors_fatal to daemons installation and removed explicit installation of the daemons packaged in the cloudera_manager/agent role * removed unfinished secure-ha definition * captialized every mention of CA * Version number updates for CDP-DC 7.1.2 (#75) * update CDP-DC version to 7.1.2 in example templates * removed obsolete CDP 7.0 template * renamed definition version directories * updated cluster def vars and docs * Fix for certs being created without SAN (#78) * Fixed task failure when importing Kerberos credentials with special characters * Add missing 'auth_provider' variable into extra_vars.yml for basic-ad-integration template to resolve validation failure in verify_inventory_and_definition play * Restart SCM agent post-install just in case it has already been installed. Additionally move agent TLS configuration up prior to changes to server_host in agent config file. In testing this resolved an issue where the agents were not restarted post-TLS config changes * Fix issue where if the mgmt services start command was accepted by the API, but failed immediately, the API command status was not being validated * Fix for custom log base on CDP-DC 7.x (#81) Some log-related parameters present in C6 have been removed in C7. This patch fixes the cluster templates. * Fixed issue with CM services start where if the mgmt service was already started the command would fail with 'Start not available for Execution error' (#82) * Stop setting kernel flags on ycloud (#87) * stop setting kernel flags on ycloud * also don't run these things... * Added config slice for Ranger defaults (#86) * Added Infra Solr service (#83) * Added Infra Solr service A new pseudo-service INFRA_SOLR which acts as a placeholder to configure a second SOLR service in CM template, with configurations matching those used in CM's own autoconfiguration rules engine * moved up infra solr config slice ... to make it possible to override logdirs in a later merge * removed buggy load balancer config * explicitly set service dependencies to most appropriate Solr instance * somehow missed a merge conflict * Delete cluster on failed template import (#89) ... if it it safe to do so, i.e. no services were created or the services that were created have not yet been started. * install ntp or chrony only if neither of them is installed (#90) * install ntp or chrony only if neither of them is installed * Added the condition to set ntp_service property as chronyd if chrony is already installed * Add Private Cloud base sample definition with Ozone (PS-219) (#88) * Added definition for Private Cloud base with Ozone * Edit for removing Ranger HA from Private Cloud base * Added gateway to HDFS * Added Ozone ranger, admin related safety valve properties * Removed safety valve property ozone.acl.authorizer.class. Will be automatically added by CM * Remove become:true statements from actions performed on the control host (#95) * Minor database-related updates (#96) * update default database to Postgres 10 * fix possible crash when db_server group is not defined * Updates for release of CDP Private Cloud Base 7.1.3 (#99) * Fixes for Ubuntu 18 support (#100) * remove cloudera-scm from wheel group discussion: https://cloudera.slack.com/archives/CDN78QR8E/p1597141859004800 * fixes ca_server role for Ubuntu Ubuntu uses cryptography backend by default rather than pyOpenSSL we should aim to migrate the RHEL version to this too, since support for pyOpenSSL is deprecated and will be removed soon. * fixes TLS ACL setting for Ubuntu acl package was not present by default, at least not in the ycloud image i'm testing with * moved postgres template fix to run on all OS types * added required python package on postgres server for Ubuntu * fixed hardcoded kerberos state directory in config * decoupled agent install and config * further kerberos KDC config fixes * fixed httpd install for custom repo * other minor tweaks * these tasks are postgres only * there's no wheel group on ubuntu by default * Fixes for Ranger running on MariaDB database type (#101) * fixes for Ranger with mariadb database type * fixed variable name * Minor fix to CM agent TLS config (#103) * Only-kafka with no security implemented (#97) * New example of full secured 10 node cluster * typo fixed * replace secure folder with the new example including tls * Fix cluster definition path for 6.3.3/secure * removed full-secure directory * kafka only definitions * initial configurations for SRM * add vault for fast testing * fixed configurations for testing * Definition working with hardcoded values * for loop for retreiving host names * SMM configuration + all hosts in replication manager * fix hostnames to example.com * rollback site.yml for pull request * CDH version and name convention fix * dynamic groups fetching for kafka brokers * Remove ozone.acl.enabled (#105) It doesn't seem to exist anymore * Kafka POC: Bidirectional replication (no security) (#104) * New example of full secured 10 node cluster * typo fixed * replace secure folder with the new example including tls * Fix cluster definition path for 6.3.3/secure * removed full-secure directory * kafka only definitions * initial configurations for SRM * add vault for fast testing * fixed configurations for testing * Definition working with hardcoded values * for loop for retreiving host names * SMM configuration + all hosts in replication manager * fix hostnames to example.com * rollback site.yml for pull request * CDH version and name convention fix * dynamic groups fetching for kafka brokers * two clusters definition * test for cluster children in inventory * Config order and hostname examples fixed * definition name and host list updated * rename directory * Move definitions out of main repo (#106) * deleted definitions folder * some docs updates * missed one (#107) * File permission updates to handle recent Ansible default behaviour change (#108) ref https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.9.html#change-to-default-file-permissions * Ansible 2.10 updates (#102) * reworked requirements.yml for collections * doc updates * couple of trivial updates * Adding AD configs for PS-270 (#110) * Teardown (#98) * split up create_{base,kts}.yml into two pieces to allow a role to generate configs * created teardown role to remove old cluster files * moved cm database defaults to group_vars/all * moved dependency on api_hosts inside tasks in deployment/services/mgmt * moved dependency on api_hosts inside tasks in deployment/cluster * split out config generation in deployment/services/mgmt * modified config overlay conditions to work with kts clusters * moved the replace condition for mariadb->mysql to the source of the problem * fixed teardown to remove databases, directories, services and packages * set logdir overlay to always run with a sensible default * added service directory teardown * added check_mode: no to repometa uri call to allow for teardown --check * mgmt logdirs overlay set to always run with sensible defaults * limited cleanup tasks to the correct nodes * fixed directory cleanup and added cms log directories * added comments warning to keep variable names static * added temporary logic to delete kts directories * added log directory configs to teardown and fixed typo * disabled gather_facts for teardown * added tasks to teardown the kms * separated cluster and service config generation from the provisioning * fixed yaml formatting problem * changed teardown to use the merged cluster configs * added default directories to some services and added the configs to teardown * added explicit default fs paths for cms and added them to the teardown * fixed dependency problems and fixed run directory cleanup during agent teardown * removed erroneous condition on pheonix overlay * fixed logic problem with role check bypass in teardown * added option to stop cluster before deletion * added cluster_membership dict to definition (meta enrichment) * added teardown flags and restricted cluster deletion to nodes in cluster * replaced host cluster map with more scalable template cluster map * fixed problem with kms teardown * added cluster and cms deletion to teardown * added checks for existance and fixed stopped cluster check * naming and syntax changes * moved cloudera_manager_database_ variables into cloudera_manager/common * fixed stop call in delete_cms to pass when already stopped * moved config roles out of deployment * removed group hosts task from teardown * changed order of tasks, added teardown_skip_cluster_deletion and setup service groups as required by deployment/definition * added dynamic host groups generation to teardown.yml as required by deployment/definition * Updates for release of CDP Private Cloud Base 7.1.3 (#99) * added ignore_errors to agent and server stop service calls * Fixes for Ubuntu 18 support (#100) * remove cloudera-scm from wheel group discussion: https://cloudera.slack.com/archives/CDN78QR8E/p1597141859004800 * fixes ca_server role for Ubuntu Ubuntu uses cryptography backend by default rather than pyOpenSSL we should aim to migrate the RHEL version to this too, since support for pyOpenSSL is deprecated and will be removed soon. * fixes TLS ACL setting for Ubuntu acl package was not present by default, at least not in the ycloud image i'm testing with * moved postgres template fix to run on all OS types * added required python package on postgres server for Ubuntu * fixed hardcoded kerberos state directory in config * decoupled agent install and config * further kerberos KDC config fixes * fixed httpd install for custom repo * other minor tweaks * these tasks are postgres only * there's no wheel group on ubuntu by default * Fixes for Ranger running on MariaDB database type (#101) * fixes for Ranger with mariadb database type * fixed variable name * Minor fix to CM agent TLS config (#103) * Only-kafka with no security implemented (#97) * New example of full secured 10 node cluster * typo fixed * replace secure folder with the new example including tls * Fix cluster definition path for 6.3.3/secure * removed full-secure directory * kafka only definitions * initial configurations for SRM * add vault for fast testing * fixed configurations for testing * Definition working with hardcoded values * for loop for retreiving host names * SMM configuration + all hosts in replication manager * fix hostnames to example.com * rollback site.yml for pull request * CDH version and name convention fix * dynamic groups fetching for kafka brokers * Remove ozone.acl.enabled (#105) It doesn't seem to exist anymore * Kafka POC: Bidirectional replication (no security) (#104) * New example of full secured 10 node cluster * typo fixed * replace secure folder with the new example including tls * Fix cluster definition path for 6.3.3/secure * removed full-secure directory * kafka only definitions * initial configurations for SRM * add vault for fast testing * fixed configurations for testing * Definition working with hardcoded values * for loop for retreiving host names * SMM configuration + all hosts in replication manager * fix hostnames to example.com * rollback site.yml for pull request * CDH version and name convention fix * dynamic groups fetching for kafka brokers * two clusters definition * test for cluster children in inventory * Config order and hostname examples fixed * definition name and host list updated * rename directory * Move definitions out of main repo (#106) * deleted definitions folder * some docs updates * missed one (#107) * File permission updates to handle recent Ansible default behaviour change (#108) ref https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.9.html#change-to-default-file-permissions * Ansible 2.10 updates (#102) * reworked requirements.yml for collections * doc updates * couple of trivial updates * Added docs for the teardown playbook (#111) * Cloudera Manager database fixes (#113) * fixed db name vs user mix-up * moved message about what will be done to before actually doing the thing * Customizing /var/lib directory (#112) * Adding option to customize /var/lib directory * Adding option to customize /var/lib directory - Ozone * Solr related changes * Include Solr upgrade directory * Added option for including keytab generation script * Revert "Include option for Custom Keytab generation script" * Option for including keytab generation script (#114) * Added option for including keytab generation script * Renamed parameter to krb5_keytab_retrieval_script and added new line * added skip_cluster_deletion option (#116) * Removed need to fix ycloud DNS with hosts file (#117) * Fixes for Cloudera Flow Management 2.0.1 with CDP Base 7.1.x (#118) * minor fixes * workaround for OPSAPS-56076 * Minor fix to template condition for logdirs (#119) * Update getting-started.md (#124) Change definitions locations * Improved TLS CA subject building (#123) * made ca attriutes configurable * made host certifiate subject configurable * removed certificate defaults * fixed certificate subject templates * added certificate attribute escaping * added certificate defaults back * removed L from the certificate attributes * Documentation for packing playbook for airgapped environments (#127) * Added doc for creating self-contained playbooks (#29) * minor edits * Fixed license detection template (#126) * Oracle support - prereqs and teardown (#121) * added autotls without ca * added bigtop util call to second command * added oracle connector * added oracle port number to filter * fixed scm db check for managed database and corrected default scm db host * fixed oracle connector role * changed oracle jdbc path * added https maven repository * added untested command to teardown oracle users (and data) * srm missed off the acl list * made maven for oracle repo configurable * added oracle instantclient unarchive * added flag to unarchive instantclient flattened * added smm to key acls * udpated oracle teardown to match playbook v1 * added default for db_server * fixed oracle db teardown * added libaio to oracle connector tasks * fixed cms database delete * removed autotls bits * fixed temporary package name * Added skip_krb5_conf_distribution flag (#128) * Role mappings fix (#130) * added check for existing role mappings * fixed role mapping get * commented out role name check * removed role name check logic * added krb5.conf template location var with default (#129) * moved cluster deletion in cm and agent server deletion around in teardown * fixed when conditions for teardown * Nifi TLS (#137) * added tls and kerberos auth configs * added nifi to keystore acls * fixed NiFi TLS configs to match CFM2 * added nifi system users * Support for Data Contexts (SDX) and Virtual Private Clusters (#109) * add creation of data contexts on base clusters * create data context on base clusters by default * virtual private cluster templates * fixes following testing fixes following testing * camel to snake case * kerberos fixes * added default empty dicts for KTS configs * added documentation * separated data context creation from cluster creation * fixes for idempotence * GPG check (#134) * added var to configure gpgcheck * added skip_rdbms_repo_setup flag * added missing flag and removed enablerepo param * Secure ha (#131) * added overlay configs to the kms service deployer and tls configs * fixed bug with default kts template erasing all auto-configs * added secure-ha definition * added hive gateway to fix warning * fixed cluster.yml destination in secure-ha * removed definitions * separated config and deployment of kms * Configurable timeouts (#132) * made timeout and retries configurable * added vars to configure kms parcel poll max and durations * made poll vars naming consistent * made deploy client config timeout configurable * changed client config timeout default * changed poll_max_retries to poll_max_failed_retries * CM custom admin password (#136) * added calls to cloudera_manager/admin_password/{check,set} * added cloudera_manager/admin_password/check * removed dependency on cm_api in cloudera_manager/admin_password * added cloudera_manager/admin_password/check as a dependency of cm_api * added implementation for cloudera_manager/admin_password/set * changed to use consistent var for cm host and consistent validate certs bool * changed set_cloudera_manager_password.yml to ensure new password is detected * fix(oozie): add properties for tls required (#140) LGTM merging * added version flag for data context (#143) * separated oozie tls configs in 7.1.4 file (#144) * separated oozie tls configs in 7.1.4 file * yaml syntax to match other file * added kudu defaults to defaults.j2 (#142) * Update main.yml (#139) LGTM * fix(cdh): remove clusterspec for 5 or 6 (#138) * fix(cdh): remove clusterspec for 5 or 6 feat(cdh): cluster spec compatibility with cdh < 6.2.0 * fix(cdh): remove trailing comma * added wait for cluster stop in operations/delete_cluster * separated wait and get cluster commands * added data context deletion to cluster deletion operation * fixed typo * added cluster defaults to teardown * added 404 as a successful http status code for data context delete * correct success status for data context deletion * added configurable timeouts for cluster stop wait * changed supervisor stop from service stop to pid kill * fixed process directory find to match directories and clean up old processes * changed command to shell so that the process directory is actually cleaned up * added kudu directories to teardown * Role mappings fix (#151) * fixed role mapping get * added cdsw teardown * Fixed merge * fixed teardown to merge configs for compute clusters * fixed when condition for kts cluster teardown * added any errors fatal to teardown * added the CDSW var directory to teardown * added teardown_preserve_parcels * fixed parcels paths * fixed loop typo * added teardown_skip_daemon_package_deletion * updated teardown docs with cdsw and new flags * Support for external accounts like AWS (#154) * New Role for external accounts like aws * Fixed some typo's * Added support for altus and azure external accounts, as well multiple account support * Removed _ext in the naming of the variables * This commit does not contain secrets * Oracle docs (#153) * updated oracle docs * cleaned up oracle docs * Fixed a bug that broke the ansible role for Cloudera Manager (#160) * changed inventory verify to complete quicker (#155) * Fix cluster deletion and teardown (#152) * added wait for cluster stop in operations/delete_cluster * separated wait and get cluster commands * added data context deletion to cluster deletion operation * fixed typo * added cluster defaults to teardown * added 404 as a successful http status code for data context delete * correct success status for data context deletion * added configurable timeouts for cluster stop wait * changed supervisor stop from service stop to pid kill * fixed process directory find to match directories and clean up old processes * changed command to shell so that the process directory is actually cleaned up * added kudu directories to teardown * Repometa speedup (#157) * added run_once to repometa * added cache to repometa * Revert "added cache to repometa" This reverts commit c54709dec400ad7fc65ac51a8e8f508580608398. * Conditional users and keys (#148) * moved the tls acls into local_accounts * added conditionals to users acccounts * included correct vars file and cleaned up acl logicc * removed comment * removed user verifcation * added message about user creation failure * corrected verify_parcels.yml import following rename * added nifi and nifiregistry users * adding back non-acl users * added run_once to debug message * added flink user * KMS ranger repo fix (#150) * added empty playbook ready for kms fix * add a role to find the correct cluster and recreate the kms repo in ranger * changed keyadmin password default to match playbook * moved about roles and removed ranger requirement for kms fix * fixed boolean logic * changed refresh_ranger_kms_repo task name * TLS 2020 September (#133) * changed skip_tls_cert_distribution var name and added check for absense of tls in inventory * added check for manaual cert distribution when enabling tls on cm * added customizable keystore aliases * added bits to customize the cloudera manager opts * added logic to find and merge cacerts into a fat truststore * fixed /etc/default/cloudera-scm-server edits * fixed cm fat truststore * added check for cacerts import and option to skip * added check for truststore password * removed single quotes in cm truststore config and added failure for spaces * added missing group gen for setup_hdfs_encryption * added manual_tls_cert_distribution exception to agent tls config * added zookeeper check for truststore keystore extension * fixed the zookeeper truststore assertion * Agent wait for (#135) * added wait_for_heartbeat role * removed the 30 second wait in install_cloudera_manager and added HB wait to cluster installation * added configurable retry timeouts * added any_errors_fatal to heartbeat check * changed wait_for_heartbeat to use last heartbeat time * removed unused filter * removed from_iso filter * added default for when lastHeartbeat does not exist * added cloudera_manager_agent_lib_directory * added quotations around template expressions * replaced any reference of external definitions with examples/basic-7.1.x * added cluster definition example * replaced any reference of external definitions in VPC docs * replaced any reference of external definitions in getting started docs * fixed loop_var * added database-configuration.md * added note about database_type * changed cruise control config to work with OPSAPS-55800 change * removed mention of prepare_ycloud.yml from the docs * changed database configuration docs formatting * cleaned up VPC docs * fixed templating of cruise control workaround * Freeipa (#120) * added FreeIPA pieces * added freeipa.md * fixed when conditions in create_infrastructure.yml and added cloudera_manager to freeipa clients * disabled freeipa firewall * added KRB_AUTH_ENABLE config to cm kerberos config * added cert and key locations to freeipa prov * added default knox proxy user configs to CM * added p12 keystore generation * removed playbook certs from freeipa * added bits to allow freeipa to sign certs * added 'enumerate = True' to SSSD config * added option to disable automatic freeipa certificate signing * added automatic ldap settings for hue * changed the format of auth_providers to allow for more flexiability * added chmod to freeipa certificate signing command * fixed improper escaping in regex * updated freeipa docs * changed ca_server logic to handle intermediate key and certificate gen failure better * changed freeipa cert signing due to ZOOKEEPER-3832 * added defaults for kudu dirs * fixed freeipa signing command * split out freeipa * added verify for ca_server and FreeIPA * fixed formatting of FreeIPA ca_server assert * documented ldap for freeipa * corrected freeipa docs * widened check on ca_server freeipa collision * fixed docs typos * removed freeipa client install in prereqs * added systemd fix to prepare_ycloud.yml * added command to change systemd gid * added defaults for kerberos and tls to config overlay * fixed formatting of workaround * Updated refname (#173) * changed refname format * updated refName to match old behaviour * changed ref for infra solr to match explicit ref * stopped template replacing infra solr display name * changed order of service_type in template to fix bug * Added Copyright header and ASLv2 * Added ICLA and CCLA * Added sample definitions * Updated docs for external consumption * Add dfs_datanode_failed_volumes_tolerated=0 for when deploying with only one DN Data Dir * Updated to remove references to SOLR-1 and RANGER-1 * Merge in docs changes to common examples location * Fix broken links * Add license header to basic examples Co-authored-by: Dave Beech <[email protected]> Co-authored-by: Jim Halfpenny <[email protected]> Co-authored-by: krishna kalyan <[email protected]> Co-authored-by: Dima Fadeyev <[email protected]> Co-authored-by: cteoh <[email protected]> Co-authored-by: mwies <[email protected]> Co-authored-by: Denis Coady <[email protected]> Co-authored-by: wdyson <[email protected]> Co-authored-by: luciano <[email protected]> Co-authored-by: Chris Jacques <[email protected]> Co-authored-by: vudamala <[email protected]> Co-authored-by: vkarthikeyan <[email protected]> Co-authored-by: mokane <[email protected]> Co-authored-by: Francois <[email protected]>
1 parent 4889b61 commit 69e27fc

File tree

392 files changed

+12882
-2043
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

392 files changed

+12882
-2043
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,4 @@ test*
44
*.pyc
55
*.bak
66
.DS_Store
7+
venv

LICENSE

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
Apache License
2+
Version 2.0, January 2004
3+
http://www.apache.org/licenses/
4+
5+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6+
7+
1. Definitions.
8+
9+
"License" shall mean the terms and conditions for use, reproduction,
10+
and distribution as defined by Sections 1 through 9 of this document.
11+
12+
"Licensor" shall mean the copyright owner or entity authorized by
13+
the copyright owner that is granting the License.
14+
15+
"Legal Entity" shall mean the union of the acting entity and all
16+
other entities that control, are controlled by, or are under common
17+
control with that entity. For the purposes of this definition,
18+
"control" means (i) the power, direct or indirect, to cause the
19+
direction or management of such entity, whether by contract or
20+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
21+
outstanding shares, or (iii) beneficial ownership of such entity.
22+
23+
"You" (or "Your") shall mean an individual or Legal Entity
24+
exercising permissions granted by this License.
25+
26+
"Source" form shall mean the preferred form for making modifications,
27+
including but not limited to software source code, documentation
28+
source, and configuration files.
29+
30+
"Object" form shall mean any form resulting from mechanical
31+
transformation or translation of a Source form, including but
32+
not limited to compiled object code, generated documentation,
33+
and conversions to other media types.
34+
35+
"Work" shall mean the work of authorship, whether in Source or
36+
Object form, made available under the License, as indicated by a
37+
copyright notice that is included in or attached to the work
38+
(an example is provided in the Appendix below).
39+
40+
"Derivative Works" shall mean any work, whether in Source or Object
41+
form, that is based on (or derived from) the Work and for which the
42+
editorial revisions, annotations, elaborations, or other modifications
43+
represent, as a whole, an original work of authorship. For the purposes
44+
of this License, Derivative Works shall not include works that remain
45+
separable from, or merely link (or bind by name) to the interfaces of,
46+
the Work and Derivative Works thereof.
47+
48+
"Contribution" shall mean any work of authorship, including
49+
the original version of the Work and any modifications or additions
50+
to that Work or Derivative Works thereof, that is intentionally
51+
submitted to Licensor for inclusion in the Work by the copyright owner
52+
or by an individual or Legal Entity authorized to submit on behalf of
53+
the copyright owner. For the purposes of this definition, "submitted"
54+
means any form of electronic, verbal, or written communication sent
55+
to the Licensor or its representatives, including but not limited to
56+
communication on electronic mailing lists, source code control systems,
57+
and issue tracking systems that are managed by, or on behalf of, the
58+
Licensor for the purpose of discussing and improving the Work, but
59+
excluding communication that is conspicuously marked or otherwise
60+
designated in writing by the copyright owner as "Not a Contribution."
61+
62+
"Contributor" shall mean Licensor and any individual or Legal Entity
63+
on behalf of whom a Contribution has been received by Licensor and
64+
subsequently incorporated within the Work.
65+
66+
2. Grant of Copyright License. Subject to the terms and conditions of
67+
this License, each Contributor hereby grants to You a perpetual,
68+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69+
copyright license to reproduce, prepare Derivative Works of,
70+
publicly display, publicly perform, sublicense, and distribute the
71+
Work and such Derivative Works in Source or Object form.
72+
73+
3. Grant of Patent License. Subject to the terms and conditions of
74+
this License, each Contributor hereby grants to You a perpetual,
75+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76+
(except as stated in this section) patent license to make, have made,
77+
use, offer to sell, sell, import, and otherwise transfer the Work,
78+
where such license applies only to those patent claims licensable
79+
by such Contributor that are necessarily infringed by their
80+
Contribution(s) alone or by combination of their Contribution(s)
81+
with the Work to which such Contribution(s) was submitted. If You
82+
institute patent litigation against any entity (including a
83+
cross-claim or counterclaim in a lawsuit) alleging that the Work
84+
or a Contribution incorporated within the Work constitutes direct
85+
or contributory patent infringement, then any patent licenses
86+
granted to You under this License for that Work shall terminate
87+
as of the date such litigation is filed.
88+
89+
4. Redistribution. You may reproduce and distribute copies of the
90+
Work or Derivative Works thereof in any medium, with or without
91+
modifications, and in Source or Object form, provided that You
92+
meet the following conditions:
93+
94+
(a) You must give any other recipients of the Work or
95+
Derivative Works a copy of this License; and
96+
97+
(b) You must cause any modified files to carry prominent notices
98+
stating that You changed the files; and
99+
100+
(c) You must retain, in the Source form of any Derivative Works
101+
that You distribute, all copyright, patent, trademark, and
102+
attribution notices from the Source form of the Work,
103+
excluding those notices that do not pertain to any part of
104+
the Derivative Works; and
105+
106+
(d) If the Work includes a "NOTICE" text file as part of its
107+
distribution, then any Derivative Works that You distribute must
108+
include a readable copy of the attribution notices contained
109+
within such NOTICE file, excluding those notices that do not
110+
pertain to any part of the Derivative Works, in at least one
111+
of the following places: within a NOTICE text file distributed
112+
as part of the Derivative Works; within the Source form or
113+
documentation, if provided along with the Derivative Works; or,
114+
within a display generated by the Derivative Works, if and
115+
wherever such third-party notices normally appear. The contents
116+
of the NOTICE file are for informational purposes only and
117+
do not modify the License. You may add Your own attribution
118+
notices within Derivative Works that You distribute, alongside
119+
or as an addendum to the NOTICE text from the Work, provided
120+
that such additional attribution notices cannot be construed
121+
as modifying the License.
122+
123+
You may add Your own copyright statement to Your modifications and
124+
may provide additional or different license terms and conditions
125+
for use, reproduction, or distribution of Your modifications, or
126+
for any such Derivative Works as a whole, provided Your use,
127+
reproduction, and distribution of the Work otherwise complies with
128+
the conditions stated in this License.
129+
130+
5. Submission of Contributions. Unless You explicitly state otherwise,
131+
any Contribution intentionally submitted for inclusion in the Work
132+
by You to the Licensor shall be under the terms and conditions of
133+
this License, without any additional terms or conditions.
134+
Notwithstanding the above, nothing herein shall supersede or modify
135+
the terms of any separate license agreement you may have executed
136+
with Licensor regarding such Contributions.
137+
138+
6. Trademarks. This License does not grant permission to use the trade
139+
names, trademarks, service marks, or product names of the Licensor,
140+
except as required for reasonable and customary use in describing the
141+
origin of the Work and reproducing the content of the NOTICE file.
142+
143+
7. Disclaimer of Warranty. Unless required by applicable law or
144+
agreed to in writing, Licensor provides the Work (and each
145+
Contributor provides its Contributions) on an "AS IS" BASIS,
146+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147+
implied, including, without limitation, any warranties or conditions
148+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149+
PARTICULAR PURPOSE. You are solely responsible for determining the
150+
appropriateness of using or redistributing the Work and assume any
151+
risks associated with Your exercise of permissions under this License.
152+
153+
8. Limitation of Liability. In no event and under no legal theory,
154+
whether in tort (including negligence), contract, or otherwise,
155+
unless required by applicable law (such as deliberate and grossly
156+
negligent acts) or agreed to in writing, shall any Contributor be
157+
liable to You for damages, including any direct, indirect, special,
158+
incidental, or consequential damages of any character arising as a
159+
result of this License or out of the use or inability to use the
160+
Work (including but not limited to damages for loss of goodwill,
161+
work stoppage, computer failure or malfunction, or any and all
162+
other commercial damages or losses), even if such Contributor
163+
has been advised of the possibility of such damages.
164+
165+
9. Accepting Warranty or Additional Liability. While redistributing
166+
the Work or Derivative Works thereof, You may choose to offer,
167+
and charge a fee for, acceptance of support, warranty, indemnity,
168+
or other liability obligations and/or rights consistent with this
169+
License. However, in accepting such obligations, You may act only
170+
on Your own behalf and on Your sole responsibility, not on behalf
171+
of any other Contributor, and only if You agree to indemnify,
172+
defend, and hold each Contributor harmless for any liability
173+
incurred by, or claims asserted against, such Contributor by reason
174+
of your accepting any such warranty or additional liability.
175+
176+
END OF TERMS AND CONDITIONS
177+
178+
APPENDIX: How to apply the Apache License to your work.
179+
180+
To apply the Apache License to your work, attach the following
181+
boilerplate notice, with the fields enclosed by brackets "[]"
182+
replaced with your own identifying information. (Don't include
183+
the brackets!) The text should be enclosed in the appropriate
184+
comment syntax for the file format. We also recommend that a
185+
file or class name and description of purpose be included on the
186+
same "printed page" as the copyright notice for easier
187+
identification within third-party archives.
188+
189+
Copyright 2021 Cloudera, Inc. All rights reserved.
190+
191+
Licensed under the Apache License, Version 2.0 (the "License");
192+
you may not use this file except in compliance with the License.
193+
You may obtain a copy of the License at
194+
195+
http://www.apache.org/licenses/LICENSE-2.0
196+
197+
Unless required by applicable law or agreed to in writing, software
198+
distributed under the License is distributed on an "AS IS" BASIS,
199+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200+
See the License for the specific language governing permissions and
201+
limitations under the License.

README.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Ansible Playbooks for Cloudera Data Platform
2+
3+
## Requirements
4+
5+
- Python 2.x or 3.x
6+
- [Ansible](http://docs.ansible.com/ansible/intro_installation.html)
7+
- [JMESPath](https://jmespath.org/)
8+
9+
**Do not use Ansible 2.9.0**. This version has an [issue with templating](https://github.com/ansible/ansible/issues/64745) which causes the playbook execution to fail. Instead, use any 2.8.x version or a later 2.9.x version as these are not affected.
10+
11+
## Supported Platforms
12+
13+
### Cloudera Distributions
14+
15+
- Cloudera Manager / CDP Private Cloud Base 7.1.x
16+
- Cloudera Manager / CDP Private Cloud Base 7.0.3 (limited support)
17+
- Cloudera Manager / CDH 6.x
18+
- Cloudera Manager / CDH 5.x (limited support)
19+
20+
### Operating Systems
21+
22+
- Red Hat / CentOS 7.x
23+
- Ubuntu 18.04.04 LTS (Bionic Beaver)
24+
25+
Active development is focused on **CDP Private Cloud Base** (formerly CDP-DC) deployments and their respective platform compatibility matrices.
26+
27+
> While these playbooks can be used to deploy CDH 5.x and CDH 6.x environments, it is only possible to install a subset of their supported platform components (i.e JDK and database versions) using this tool.
28+
29+
## Getting Started
30+
31+
For help setting up the playbook, creating configs and deploying clusters, see the [Getting Started](docs/getting-started.md) guide.
32+
33+
## How-to Guides
34+
35+
For more detailed information, check the following guides:
36+
37+
* Deploying [secure clusters](docs/security.md)
38+
* Deploying [data contexts (SDX) and virtual private clusters](docs/how-to/virtual-private-clusters.md)
39+
40+
## Help!
41+
42+
Common issues and their solutions are documented on the [Troubleshooting](docs/troubleshooting.md) page. Check here first.
43+
44+
## How do I contribute code?
45+
You need to first sign and return an
46+
[ICLA](icla/Cloudera_ICLA_25APR2018.pdf)
47+
and
48+
[CCLA](icla/Cloudera_CCLA_25APR2018.pdf)
49+
before we can accept and redistribute your contribution. Once these are submitted you are
50+
free to start contributing to cloudera-playbook. Submit these to [email protected].
51+
52+
### Main steps
53+
* Fork the repo and create a topic branch
54+
* Push commits to your repo
55+
* Create a pull request!
56+
57+
### Find
58+
We use Github issues to track bugs for this project. Find an issue that you would like to
59+
work on (or file one if you have discovered a new issue!). If no-one is working on it,
60+
assign it to yourself only if you intend to work on it shortly.
61+
62+
### Fix
63+
64+
Please write a good, clear commit message, with a short, descriptive title and
65+
a message that is exactly long enough to explain what the problem was, and how it was
66+
fixed.
67+
68+
## Copyright
69+
70+
(C) Cloudera, Inc. 2021 All rights reserved.
71+
72+
## License
73+
[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)

create_freeipa.yml

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Copyright 2021 Cloudera, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
---
16+
17+
- name: Install FreeIPA server
18+
hosts: krb5_server
19+
become: yes
20+
roles:
21+
- role: freeipa.ansible_freeipa.ipaserver
22+
state: present
23+
ipaserver_realm: "{{ krb5_realm }}"
24+
ipaserver_domain: "{{ krb5_domain | default(krb5_realm | lower) }}"
25+
ipaserver_setup_firewalld: "no"
26+
when: "krb5_kdc_type == 'Red Hat IPA'"
27+
tags:
28+
- security
29+
- kerberos
30+
- tls
31+
32+
- name: Setup FreeIPA clients
33+
hosts: cloudera_manager:cluster
34+
become: yes
35+
roles:
36+
- role: freeipa.ansible_freeipa.ipaclient
37+
state: present
38+
ipaserver_realm: "{{ krb5_realm }}"
39+
ipaserver_domain: "{{ krb5_domain | default(krb5_realm | lower) }}"
40+
ipaclient_servers: "{{ groups['krb5_server'] }}"
41+
when: "krb5_kdc_type == 'Red Hat IPA' and 'krb5_server' in groups"
42+
tasks:
43+
- name: Set sssd to enumerate users and groups
44+
lineinfile:
45+
path: /etc/sssd/sssd.conf
46+
insertafter: "^\\[domain/.+\\]"
47+
regexp: "^enumerate"
48+
line: "enumerate = True"
49+
when: "krb5_kdc_type == 'Red Hat IPA' and 'krb5_server' in groups"
50+
notify:
51+
- restart sssd
52+
handlers:
53+
- name: restart sssd
54+
service:
55+
name: sssd
56+
state: restarted
57+
tags:
58+
- security
59+
- kerberos
60+
- tls
61+

0 commit comments

Comments
 (0)