Moar content coming soon!
"},{"location":"UPGRADE-17.0.html","title":"How to handle the terraform-aws-eks module upgrade","text":""},{"location":"UPGRADE-17.0.html#upgrade-module-to-v1700-for-managed-node-groups","title":"Upgrade module to v17.0.0 for Managed Node Groups","text":"In this release, we now decided to remove random_pet resources in Managed Node Groups (MNG). Those were used to recreate MNG if something changed. But they were causing a lot of issues. To upgrade the module without recreating your MNG, you will need to explicitly reuse their previous name and set them in your MNG name argument.
terraform apply with the module version v16.2.0~ terraform state show 'module.eks.module.node_groups.aws_eks_node_group.workers[\"example\"]' | grep node_group_name\nnode_group_name = \"test-eks-mwIwsvui-example-sincere-squid\"\nmodule \"eks\" {\n source = \"terraform-aws-modules/eks/aws\"\n version = \"17.0.0\"\n\n cluster_name = \"test-eks-mwIwsvui\"\n cluster_version = \"1.20\"\n # ...\n\n node_groups = {\n example = {\n name = \"test-eks-mwIwsvui-example-sincere-squid\"\n\n # ...\n }\n }\n # ...\n}\nterraform plan, you should see that only random_pets will be destroyedTerraform will perform the following actions:\n\n # module.eks.module.node_groups.random_pet.node_groups[\"example\"] will be destroyed\n - resource \"random_pet\" \"node_groups\" {\n - id = \"sincere-squid\" -> null\n - keepers = {\n - \"ami_type\" = \"AL2_x86_64\"\n - \"capacity_type\" = \"SPOT\"\n - \"disk_size\" = \"50\"\n - \"iam_role_arn\" = \"arn:aws:iam::123456789123:role/test-eks-mwIwsvui20210527220853611600000009\"\n - \"instance_types\" = \"t3.large\"\n - \"key_name\" = \"\"\n - \"node_group_name\" = \"test-eks-mwIwsvui-example\"\n - \"source_security_group_ids\" = \"\"\n - \"subnet_ids\" = \"subnet-xxxxxxxxxxxx|subnet-xxxxxxxxxxxx|subnet-xxxxxxxxxxxx\"\n } -> null\n - length = 2 -> null\n - separator = \"-\" -> null\n }\n\nPlan: 0 to add, 0 to change, 1 to destroy.\nterraform applyAfter the first apply, we recommend you to create a new node group and let the module use the node_group_name_prefix (by removing the name argument) to generate names and avoid collision during node groups re-creation if needed, because the lifecycle is create_before_destroy = true.
Please consult the examples directory for reference example configurations. If you find a bug, please open an issue with supporting configuration to reproduce.
Note: please see https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1744 where users have shared the steps/changes that have worked for their configurations to upgrade. Due to the numerous configuration possibilities, it is difficult to capture specific steps that will work for all; this has proven to be a useful thread to share collective information from the broader community regarding v18.x upgrades.
For most users, adding the following to your v17.x configuration will preserve the state of your cluster control plane when upgrading to v18.x:
prefix_separator = \"\"\niam_role_name = $CLUSTER_NAME\ncluster_security_group_name = $CLUSTER_NAME\ncluster_security_group_description = \"EKS cluster security group.\"\nThis configuration assumes that create_iam_role is set to true, which is the default value.
As the location of the Terraform state of the IAM role has been changed from 17.x to 18.x, you'll also have to move the state before running terraform apply by calling:
terraform state mv 'module.eks.aws_iam_role.cluster[0]' 'module.eks.aws_iam_role.this[0]'\nSee more information here
"},{"location":"UPGRADE-18.0.html#list-of-backwards-incompatible-changes","title":"List of backwards incompatible changes","text":"We strongly recommend that you do not use launch configurations. They do not provide full functionality for Amazon EC2 Auto Scaling or Amazon EC2. We provide information about launch configurations for customers who have not yet migrated from launch configurations to launch templates.aws_auth_configmap_yaml has been provided which renders the aws-auth configmap necessary to support at least the IAM roles used by the module (additional mapRoles/mapUsers definitions to be provided by users)local_file resources have been removed; users are able to use the awscli provided aws eks update-kubeconfig --name <cluster_name> to update their local kubeconfig as necessaryeks_managed_node_groups, was previously referred to as simply node group, node_groupsself_managed_node_groups, was previously referred to as worker group, worker_groupsfargate_profiles, remains unchanged in terms of naming and terminologyeks module as well as available for individual, standalone consumption if desired.node_groups sub-module is now named eks-managed-node-group and provisions a single AWS EKS Managed Node Group per sub-module definition (previous version utilized for_each to create 0 or more node groups)eks-managed-node-group sub-module over the previous node_groups module include:Variable and output changes below_user_data internal sub-module; the local userdata.sh.tpl has been removed entirelyfargate sub-module is now named fargate-profile and provisions a single AWS EKS Fargate Profile per sub-module definition (previous version utilized for_each to create 0 or more profiles)fargate-profile sub-module over the previous fargate module include:Variable and output changes beloweks_managed_node_group_defaults and self_managed_node_group_defaults, a fargate_profile_defaults has been provided to allow users to control the default configurations for the Fargate profiles createdself-managed-node-group has been created and provisions a single self managed node group (autoscaling group) per sub-module definitionself-managed-node-group sub-module over the previous node_groups variable include:terraform-aws-autoscaling module and the features it offers_user_data internal sub-module has been created to consolidate all user data configuration in one location which provides better support for testability (via the tests/user-data example). The new sub-module supports nearly all possible combinations including the ability to allow users to provide their own user data template which will be rendered by the module. See the tests/user-data example project for the full plethora of example configuration possibilities and more details on the logic of the design can be found in the modules/_user_data directory.prefix_separator to \"\".user_data project has been added to aid in demonstrating, testing, and validating the various methods of configuring user data with the _user_data sub-module as well as the root eks moduleaws_auth_cm.tplbottlerocket_user_data.tplfargate example has been renamed to fargate_profileirsa and instance_refresh examples have been merged into one example irsa_autoscale_refreshmanaged_node_groups example has been renamed to self_managed_node_grouptls_certificate data source that refers to the cluster OIDC issuer url. Thumbprint values should remain unchanged however0.0.0.0/0 egress rule has been removed, instead TCP/443 and TCP/10250 egress rules to the node group security group are used insteaduserdata.sh.tpl and is now named linux_user_data.tpluserdata_windows.tpl to windows_user_data.tplbottlerocket example has been removed in favor of demonstrating the use and configuration of Bottlerocket nodes via the respective eks_managed_node_group and self_managed_node_group exampleslaunch_template and launch_templates_with_managed_node_groups examples have been removed; only launch templates are now supported (default) and launch configuration support has been removedsecrets_encryption example has been removed; the functionality has been demonstrated in several of the new examples rendering this standalone example redundantkubeconfig.tpl template; kubeconfig management is no longer supported under this moduleRemoved variables:
cluster_create_timeout, cluster_update_timeout, and cluster_delete_timeout have been replaced with cluster_timeoutskubeconfig_namekubeconfig_output_pathkubeconfig_file_permissionkubeconfig_api_versionkubeconfig_aws_authenticator_commandkubeconfig_aws_authenticator_command_argskubeconfig_aws_authenticator_additional_argskubeconfig_aws_authenticator_env_variableswrite_kubeconfigdefault_platformmanage_aws_authaws_auth_additional_labelsmap_accountsmap_rolesmap_usersfargate_subnetsworker_groups_launch_templateworker_security_group_idworker_ami_name_filterworker_ami_name_filter_windowsworker_ami_owner_idworker_ami_owner_id_windowsworker_additional_security_group_idsworker_sg_ingress_from_portworkers_additional_policiesworker_create_security_groupworker_create_initial_lifecycle_hooksworker_create_cluster_primary_security_group_rulescluster_create_endpoint_private_access_sg_rulecluster_endpoint_private_access_cidrscluster_endpoint_private_access_sgmanage_worker_iam_resourcesworkers_role_nameattach_worker_cni_policyeks_oidc_root_ca_thumbprintcreate_fargate_pod_execution_rolefargate_pod_execution_role_namecluster_egress_cidrsworkers_egress_cidrswait_for_cluster_timeoutnode_groups)default_iam_role_arnworkers_group_defaultsworker_security_group_idnode_groups_defaultsnode_groupsebs_optimized_not_supportedfargate)create_eks and create_fargate_pod_execution_role have been replaced with simply createRenamed variables:
create_eks -> createsubnets -> subnet_idscluster_create_security_group -> create_cluster_security_groupcluster_log_retention_in_days -> cloudwatch_log_group_retention_in_dayscluster_log_kms_key_id -> cloudwatch_log_group_kms_key_idmanage_cluster_iam_resources -> create_iam_rolecluster_iam_role_name -> iam_role_namepermissions_boundary -> iam_role_permissions_boundaryiam_path -> iam_role_pathpre_userdata -> pre_bootstrap_user_dataadditional_userdata -> post_bootstrap_user_dataworker_groups -> self_managed_node_groupsworkers_group_defaults -> self_managed_node_group_defaultsnode_groups -> eks_managed_node_groupsnode_groups_defaults -> eks_managed_node_group_defaultsnode_groups)create_eks -> createworker_additional_security_group_ids -> vpc_security_group_idsfargate_pod_execution_role_name -> namecreate_fargate_pod_execution_role -> create_iam_rolesubnets -> subnet_idsiam_path -> iam_role_pathpermissions_boundary -> iam_role_permissions_boundaryAdded variables:
cluster_additional_security_group_ids added to allow users to add additional security groups to the cluster as neededcluster_security_group_namecluster_security_group_use_name_prefix added to allow users to use either the name as specified or default to using the name specified as a prefixcluster_security_group_descriptioncluster_security_group_additional_rulescluster_security_group_tagscreate_cloudwatch_log_group added in place of the logic that checked if any cluster log types were enabled to allow users to opt in as they see fitcreate_node_security_group added to create single security group that connects node groups and cluster in central locationnode_security_group_idnode_security_group_namenode_security_group_use_name_prefixnode_security_group_descriptionnode_security_group_additional_rulesnode_security_group_tagsiam_role_arniam_role_use_name_prefixiam_role_descriptioniam_role_additional_policiesiam_role_tagscluster_addonscluster_identity_providersfargate_profile_defaultsprefix_separator added to support legacy behavior of not having a prefix separatornode_groups)platformenable_bootstrap_user_datapre_bootstrap_user_datapost_bootstrap_user_databootstrap_extra_argsuser_data_template_pathcreate_launch_templatelaunch_template_namelaunch_template_use_name_prefixdescriptionebs_optimizedami_idkey_namelaunch_template_default_versionupdate_launch_template_default_versiondisable_api_terminationkernel_idram_disk_idblock_device_mappingscapacity_reservation_specificationcpu_optionscredit_specificationelastic_gpu_specificationselastic_inference_acceleratorenclave_optionsinstance_market_optionslicense_specificationsmetadata_optionsenable_monitoringnetwork_interfacesplacementmin_sizemax_sizedesired_sizeuse_name_prefixami_typeami_release_versioncapacity_typedisk_sizeforce_update_versioninstance_typeslabelscluster_versionlaunch_template_versionremote_accesstaintsupdate_configtimeoutscreate_security_groupsecurity_group_namesecurity_group_use_name_prefixsecurity_group_descriptionvpc_idsecurity_group_rulescluster_security_group_idsecurity_group_tagscreate_iam_roleiam_role_arniam_role_nameiam_role_use_name_prefixiam_role_pathiam_role_descriptioniam_role_permissions_boundaryiam_role_additional_policiesiam_role_tagsfargate)iam_role_arn (for if create_iam_role is false to bring your own externally created role)iam_role_nameiam_role_use_name_prefixiam_role_descriptioniam_role_additional_policiesiam_role_tagsselectorstimeoutsRemoved outputs:
cluster_versionkubeconfigkubeconfig_filenameworkers_asg_arnsworkers_asg_namesworkers_user_dataworkers_default_ami_idworkers_default_ami_id_windowsworkers_launch_template_idsworkers_launch_template_arnsworkers_launch_template_latest_versionsworker_security_group_idworker_iam_instance_profile_arnsworker_iam_instance_profile_namesworker_iam_role_nameworker_iam_role_arnfargate_profile_idsfargate_profile_arnsfargate_iam_role_namefargate_iam_role_arnnode_groupssecurity_group_rule_cluster_https_worker_ingressnode_groups)node_groupsaws_auth_rolesfargate)aws_auth_rolesRenamed outputs:
config_map_aws_auth -> aws_auth_configmap_yamlfargate)fargate_profile_ids -> fargate_profile_idfargate_profile_arns -> fargate_profile_arnAdded outputs:
cluster_platform_versioncluster_statuscluster_security_group_arncluster_security_group_idnode_security_group_arnnode_security_group_idcluster_iam_role_unique_idcluster_addonscluster_identity_providersfargate_profileseks_managed_node_groupsself_managed_node_groupsnode_groups)launch_template_idlaunch_template_arnlaunch_template_latest_versionnode_group_arnnode_group_idnode_group_resourcesnode_group_statussecurity_group_arnsecurity_group_idiam_role_nameiam_role_arniam_role_unique_idfargate)iam_role_unique_idfargate_profile_statusmodule \"eks\" {\n source = \"terraform-aws-modules/eks/aws\"\n version = \"~> 17.0\"\n\n cluster_name = local.name\n cluster_version = local.cluster_version\n cluster_endpoint_private_access = true\n cluster_endpoint_public_access = true\n\n vpc_id = module.vpc.vpc_id\n subnets = module.vpc.private_subnets\n\n # Managed Node Groups\n node_groups_defaults = {\n ami_type = \"AL2_x86_64\"\n disk_size = 50\n }\n\n node_groups = {\n node_group = {\n min_capacity = 1\n max_capacity = 10\n desired_capacity = 1\n\n instance_types = [\"t3.large\"]\n capacity_type = \"SPOT\"\n\n update_config = {\n max_unavailable_percentage = 50\n }\n\n k8s_labels = {\n Environment = \"test\"\n GithubRepo = \"terraform-aws-eks\"\n GithubOrg = \"terraform-aws-modules\"\n }\n\n taints = [\n {\n key = \"dedicated\"\n value = \"gpuGroup\"\n effect = \"NO_SCHEDULE\"\n }\n ]\n\n additional_tags = {\n ExtraTag = \"example\"\n }\n }\n }\n\n # Worker groups\n worker_additional_security_group_ids = [aws_security_group.additional.id]\n\n worker_groups_launch_template = [\n {\n name = \"worker-group\"\n override_instance_types = [\"m5.large\", \"m5a.large\", \"m5d.large\", \"m5ad.large\"]\n spot_instance_pools = 4\n asg_max_size = 5\n asg_desired_capacity = 2\n kubelet_extra_args = \"--node-labels=node.kubernetes.io/lifecycle=spot\"\n public_ip = true\n },\n ]\n\n # Fargate\n fargate_profiles = {\n default = {\n name = \"default\"\n selectors = [\n {\n namespace = \"kube-system\"\n labels = {\n k8s-app = \"kube-dns\"\n }\n },\n {\n namespace = \"default\"\n }\n ]\n\n tags = {\n Owner = \"test\"\n }\n\n timeouts = {\n create = \"20m\"\n delete = \"20m\"\n }\n }\n }\n\n tags = {\n Environment = \"test\"\n GithubRepo = \"terraform-aws-eks\"\n GithubOrg = \"terraform-aws-modules\"\n }\n}\nmodule \"cluster_after\" {\n source = \"terraform-aws-modules/eks/aws\"\n version = \"~> 18.0\"\n\n cluster_name = local.name\n cluster_version = local.cluster_version\n cluster_endpoint_private_access = true\n cluster_endpoint_public_access = true\n\n vpc_id = module.vpc.vpc_id\n subnet_ids = module.vpc.private_subnets\n\n eks_managed_node_group_defaults = {\n ami_type = \"AL2_x86_64\"\n disk_size = 50\n }\n\n eks_managed_node_groups = {\n node_group = {\n min_size = 1\n max_size = 10\n desired_size = 1\n\n instance_types = [\"t3.large\"]\n capacity_type = \"SPOT\"\n\n update_config = {\n max_unavailable_percentage = 50\n }\n\n labels = {\n Environment = \"test\"\n GithubRepo = \"terraform-aws-eks\"\n GithubOrg = \"terraform-aws-modules\"\n }\n\n taints = [\n {\n key = \"dedicated\"\n value = \"gpuGroup\"\n effect = \"NO_SCHEDULE\"\n }\n ]\n\n tags = {\n ExtraTag = \"example\"\n }\n }\n }\n\n self_managed_node_group_defaults = {\n vpc_security_group_ids = [aws_security_group.additional.id]\n }\n\n self_managed_node_groups = {\n worker_group = {\n name = \"worker-group\"\n\n min_size = 1\n max_size = 5\n desired_size = 2\n instance_type = \"m4.large\"\n\n bootstrap_extra_args = \"--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'\"\n\n block_device_mappings = {\n xvda = {\n device_name = \"/dev/xvda\"\n ebs = {\n delete_on_termination = true\n encrypted = false\n volume_size = 100\n volume_type = \"gp2\"\n }\n\n }\n }\n\n use_mixed_instances_policy = true\n mixed_instances_policy = {\n instances_distribution = {\n spot_instance_pools = 4\n }\n\n override = [\n { instance_type = \"m5.large\" },\n { instance_type = \"m5a.large\" },\n { instance_type = \"m5d.large\" },\n { instance_type = \"m5ad.large\" },\n ]\n }\n }\n }\n\n # Fargate\n fargate_profiles = {\n default = {\n name = \"default\"\n\n selectors = [\n {\n namespace = \"kube-system\"\n labels = {\n k8s-app = \"kube-dns\"\n }\n },\n {\n namespace = \"default\"\n }\n ]\n\n tags = {\n Owner = \"test\"\n }\n\n timeouts = {\n create = \"20m\"\n delete = \"20m\"\n }\n }\n }\n\n tags = {\n Environment = \"test\"\n GithubRepo = \"terraform-aws-eks\"\n GithubOrg = \"terraform-aws-modules\"\n }\n}\n module \"eks\" {\n source = \"terraform-aws-modules/eks/aws\"\n- version = \"~> 17.0\"\n+ version = \"~> 18.0\"\n\n cluster_name = local.name\n cluster_version = local.cluster_version\n cluster_endpoint_private_access = true\n cluster_endpoint_public_access = true\n\n vpc_id = module.vpc.vpc_id\n- subnets = module.vpc.private_subnets\n+ subnet_ids = module.vpc.private_subnets\n\n- # Managed Node Groups\n- node_groups_defaults = {\n+ eks_managed_node_group_defaults = {\n ami_type = \"AL2_x86_64\"\n disk_size = 50\n }\n\n- node_groups = {\n+ eks_managed_node_groups = {\n node_group = {\n- min_capacity = 1\n- max_capacity = 10\n- desired_capacity = 1\n+ min_size = 1\n+ max_size = 10\n+ desired_size = 1\n\n instance_types = [\"t3.large\"]\n capacity_type = \"SPOT\"\n\n update_config = {\n max_unavailable_percentage = 50\n }\n\n- k8s_labels = {\n+ labels = {\n Environment = \"test\"\n GithubRepo = \"terraform-aws-eks\"\n GithubOrg = \"terraform-aws-modules\"\n }\n\n taints = [\n {\n key = \"dedicated\"\n value = \"gpuGroup\"\n effect = \"NO_SCHEDULE\"\n }\n ]\n\n- additional_tags = {\n+ tags = {\n ExtraTag = \"example\"\n }\n }\n }\n\n- # Worker groups\n- worker_additional_security_group_ids = [aws_security_group.additional.id]\n-\n- worker_groups_launch_template = [\n- {\n- name = \"worker-group\"\n- override_instance_types = [\"m5.large\", \"m5a.large\", \"m5d.large\", \"m5ad.large\"]\n- spot_instance_pools = 4\n- asg_max_size = 5\n- asg_desired_capacity = 2\n- kubelet_extra_args = \"--node-labels=node.kubernetes.io/lifecycle=spot\"\n- public_ip = true\n- },\n- ]\n+ self_managed_node_group_defaults = {\n+ vpc_security_group_ids = [aws_security_group.additional.id]\n+ }\n+\n+ self_managed_node_groups = {\n+ worker_group = {\n+ name = \"worker-group\"\n+\n+ min_size = 1\n+ max_size = 5\n+ desired_size = 2\n+ instance_type = \"m4.large\"\n+\n+ bootstrap_extra_args = \"--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'\"\n+\n+ block_device_mappings = {\n+ xvda = {\n+ device_name = \"/dev/xvda\"\n+ ebs = {\n+ delete_on_termination = true\n+ encrypted = false\n+ volume_size = 100\n+ volume_type = \"gp2\"\n+ }\n+\n+ }\n+ }\n+\n+ use_mixed_instances_policy = true\n+ mixed_instances_policy = {\n+ instances_distribution = {\n+ spot_instance_pools = 4\n+ }\n+\n+ override = [\n+ { instance_type = \"m5.large\" },\n+ { instance_type = \"m5a.large\" },\n+ { instance_type = \"m5d.large\" },\n+ { instance_type = \"m5ad.large\" },\n+ ]\n+ }\n+ }\n+ }\n\n # Fargate\n fargate_profiles = {\n default = {\n name = \"default\"\n selectors = [\n {\n namespace = \"kube-system\"\n labels = {\n k8s-app = \"kube-dns\"\n }\n },\n {\n namespace = \"default\"\n }\n ]\n\n tags = {\n Owner = \"test\"\n }\n\n timeouts = {\n create = \"20m\"\n delete = \"20m\"\n }\n }\n }\n\n tags = {\n Environment = \"test\"\n GithubRepo = \"terraform-aws-eks\"\n GithubOrg = \"terraform-aws-modules\"\n }\n }\nresource \"aws_iam_role_policy_attachment\" \"default\" {\n role = module.eks.fargate_iam_role_name\n policy_arn = aws_iam_policy.default.arn\n}\n# Attach the policy to an \"example\" Fargate profile\nresource \"aws_iam_role_policy_attachment\" \"default\" {\n role = module.eks.fargate_profiles[\"example\"].iam_role_name\n policy_arn = aws_iam_policy.default.arn\n}\nOr:
# Attach the policy to all Fargate profiles\nresource \"aws_iam_role_policy_attachment\" \"default\" {\n for_each = module.eks.fargate_profiles\n\n role = each.value.iam_role_name\n policy_arn = aws_iam_policy.default.arn\n}\nPlease consult the examples directory for reference example configurations. If you find a bug, please open an issue with supporting configuration to reproduce.
cluster_id output used to output the name of the cluster. This is due to the fact that the cluster name is a unique constraint and therefore its set as the unique identifier within Terraform's state map. However, starting with local EKS clusters created on Outposts, there is now an attribute returned from the aws eks create-cluster API named id. The cluster_id has been updated to return this value which means that for current, standard EKS clusters created in the AWS cloud, no value will be returned (at the time of this writing) for cluster_id and only local EKS clusters on Outposts will return a value that looks like a UUID/GUID. Users should switch all instances of cluster_id to use cluster_name before upgrading to v19. Referencevar.iam_role_additional_policies (one for each of the following: cluster IAM role, EKS managed node group IAM role, self-managed node group IAM role, and Fargate Profile IAM role) accepted a list of strings. This worked well for policies that already existed but failed for policies being created at the same time as the cluster due to the well-known issue of unknown values used in a for_each loop. To rectify this issue in v19.x, two changes were made:var.iam_role_additional_policies was changed from type list(string) to type map(string) -> this is a breaking change. More information on managing this change can be found below, under Terraform State Movestry() with lookup(). More details on why can be found hereClusterName tag and the event rule name is now a prefix. This guarantees that users can have multiple instances of Karpenter with their respective event rules/SQS queue without name collisions, while also still being able to identify which queues and event rules belong to which cluster.node_security_group_enable_recommended_rules is set to true by default and may conflict with any custom ingress/egress rules. Please ensure that any duplicates from the node_security_group_additional_rules are removed before upgrading, or set node_security_group_enable_recommended_rules to false. Referencepreserve as well as most_recent on addons.preserve indicates if you want to preserve the created resources when deleting the EKS add-onmost_recent indicates if you want to use the most recent revision of the add-on or the default version (default)0.0.0.0/0/::/0cluster_security_group_additional_rules and node_security_group_additional_rules have been modified to use lookup() instead of try() to avoid the well-known issue of unknown values within a for_each loop0.0.0.0/0/::/00.0.0.0/0/::/0block_device_mappings previously required a map of maps but has since changed to an array of maps. Users can remove the outer key for each block device mapping and replace the outermost map {} with an array []. There are no state changes required for this change.create_kms_key previously defaulted to false and now defaults to true. Clusters created with this module now default to enabling secret encryption by default with a customer-managed KMS key created by this modulecluster_encryption_config previously used a type of list(any) and now uses a type of any -> users can simply remove the outer [...] brackets on v19.xcluster_encryption_config previously defaulted to [] and now defaults to {resources = [\"secrets\"]} to encrypt secrets by defaultcluster_endpoint_public_access previously defaulted to true and now defaults to false. Clusters created with this module now default to private-only access to the cluster endpointcluster_endpoint_private_access previously defaulted to false and now defaults to true\"OVERWRITE\" as the default value for resolve_conflicts to ease add-on upgrade management. Users can opt out of this by instead setting \"NONE\" as the value for resolve_conflictskms module used has been updated from v1.0.2 to v1.1.0 - no material changes other than updated to latestupdate_config has been updated to the recommended { max_unavailable_percentage = 33 }instance_refresh has been updated to the recommended: {\n strategy = \"Rolling\"\n preferences = {\n min_healthy_percentage = 66\n }\n}\naws_default_tags to avoid update conflicts; this is the responsibility of the provider and should be handled at the provider levelRemoved variables:
node_security_group_ntp_ipv4_cidr_block - default security group settings have an egress rule for ALL to 0.0.0.0/0/::/0
node_security_group_ntp_ipv6_cidr_block - default security group settings have an egress rule for ALL to 0.0.0.0/0/::/0create_security_groupsecurity_group_namesecurity_group_use_name_prefixsecurity_group_descriptionsecurity_group_rulessecurity_group_tagscluster_security_group_idvpc_idEKS managed node groups:
create_security_groupsecurity_group_namesecurity_group_use_name_prefixsecurity_group_descriptionsecurity_group_rulessecurity_group_tagscluster_security_group_idvpc_idRenamed variables:
N/A
Added variables:
provision_on_outpostfor Outposts support
outpost_config for Outposts supportcluster_addons_timeouts for setting a common set of timeouts for all addons (unless a specific value is provided within the addon configuration)service_ipv6_cidr for setting the IPv6 CIDR block for the Kubernetes service addressesnode_security_group_enable_recommended_rules for enabling recommended node security group rules for common access patterns
Self-managed node groups:
launch_template_id for use when using an existing/externally created launch template (Ref: https://github.com/terraform-aws-modules/terraform-aws-autoscaling/pull/204)maintenance_optionsprivate_dns_name_optionsinstance_requirementscontextdefault_instance_warmupforce_delete_warm_pooluse_custom_launch_template was added to better clarify how users can switch between a custom launch template or the default launch template provided by the EKS managed node group. Previously, to achieve this same functionality of using the default launch template, users needed to set create_launch_template = false and launch_template_name = \"\" which is not very intuitive.launch_template_id for use when using an existing/externally created launch template (Ref: https://github.com/terraform-aws-modules/terraform-aws-autoscaling/pull/204)maintenance_optionsprivate_dns_name_options -Removed outputs:
Self-managed node groups:
security_group_arnsecurity_group_idEKS managed node groups:
security_group_arnsecurity_group_idRenamed outputs:
cluster_id is not renamed but the value it returns is now different. For standard EKS clusters created in the AWS cloud, the value returned at the time of this writing is null/empty. For local EKS clusters created on Outposts, the value returned will look like a UUID/GUID. Users should switch all instances of cluster_id to use cluster_name before upgrading to v19. Reference
Added outputs:
cluster_name - The cluster_id currently set by the AWS provider is actually the cluster name, but in the future, this will change and there will be a distinction between the cluster_name and cluster_id. Reference
v19.x, please see below for both EKS managed node group(s) and self-managed node groups and remove the node group(s) security group prior to upgrading.Self-managed node groups on v18.x by default create a security group that does not specify any rules. In v19.x, this security group has been removed due to the predominant lack of usage (most users rely on the shared node security group). While still using version v18.x of your module definition, remove this security group from your node groups by setting create_security_group = false.
vpc_security_group_ids. Once this is in place, you can proceed with the original security group removal.instance_refresh parameter of Autoscaling groups to force nodes to re-deploy when removing the security group since changes to launch templates automatically trigger an instance refresh. An example configuration is provided below.self_managed_node_group_defaults or the individual self-managed node group definitions: create_security_group = false\ninstance_refresh = {\n strategy = \"Rolling\"\n preferences = {\n min_healthy_percentage = 66\n }\n}\naws-node-termination-handler while performing this update. Please refer to the irsa-autoscale-refresh example for usage. This will ensure that pods are safely evicted in a controlled manner to avoid service disruptions.instance_refresh configuration settingsEKS managed node groups on v18.x by default create a security group that does not specify any rules. In v19.x, this security group has been removed due to the predominant lack of usage (most users rely on the shared node security group). While still using version v18.x of your module definition, remove this security group from your node groups by setting create_security_group = false.
vpc_security_group_ids. Once this is in place, you can proceed with the original security group removal.update_config. No additional changes are required for removing the security group created by node groups (unlike self-managed node groups which should utilize the instance_refresh setting of Autoscaling groups).create_security_group = false has been set, you can apply the changes which will:update_config configuration settingsOnce the EKS managed node group has cycled, the security group will be deleted
Once the node group security group(s) have been removed, you can update your module definition to specify the v19.x version of the module
terraform init -upgrade=true to update your configuration and pull in the v19 changesv18.x to v19.x. You can utilize terraform plan as you go to help highlight any changes that you wish to make. See below for terraform state mv ... commands related to the use of iam_role_additional_policies. If you are not providing any values to these variables, you can skip this section.terraform plan output, you can apply the changes to sync your infrastructure with the updated module definition (or vice versa). module \"eks\" {\n source = \"terraform-aws-modules/eks/aws\"\n- version = \"~> 18.0\"\n+ version = \"~> 19.0\"\n\n cluster_name = local.name\n+ cluster_endpoint_public_access = true\n- cluster_endpoint_private_access = true # now the default\n\n cluster_addons = {\n- resolve_conflicts = \"OVERWRITE\" # now the default\n+ preserve = true\n+ most_recent = true\n\n+ timeouts = {\n+ create = \"25m\"\n+ delete = \"10m\"\n }\n kube-proxy = {}\n vpc-cni = {\n- resolve_conflicts = \"OVERWRITE\" # now the default\n }\n }\n\n # Encryption key\n create_kms_key = true\n- cluster_encryption_config = [{\n- resources = [\"secrets\"]\n- }]\n+ cluster_encryption_config = {\n+ resources = [\"secrets\"]\n+ }\n kms_key_deletion_window_in_days = 7\n enable_kms_key_rotation = true\n\n- iam_role_additional_policies = [aws_iam_policy.additional.arn]\n+ iam_role_additional_policies = {\n+ additional = aws_iam_policy.additional.arn\n+ }\n\n vpc_id = module.vpc.vpc_id\n subnet_ids = module.vpc.private_subnets\n control_plane_subnet_ids = module.vpc.intra_subnets\n\n # Extend node-to-node security group rules\n- node_security_group_ntp_ipv4_cidr_block = [\"169.254.169.123/32\"] # now the default\n node_security_group_additional_rules = {\n- ingress_self_ephemeral = {\n- description = \"Node to node ephemeral ports\"\n- protocol = \"tcp\"\n- from_port = 0\n- to_port = 0\n- type = \"ingress\"\n- self = true\n- }\n- egress_all = {\n- description = \"Node all egress\"\n- protocol = \"-1\"\n- from_port = 0\n- to_port = 0\n- type = \"egress\"\n- cidr_blocks = [\"0.0.0.0/0\"]\n- ipv6_cidr_blocks = [\"::/0\"]\n- }\n }\n\n # Self-Managed Node Group(s)\n self_managed_node_group_defaults = {\n vpc_security_group_ids = [aws_security_group.additional.id]\n- iam_role_additional_policies = [aws_iam_policy.additional.arn]\n+ iam_role_additional_policies = {\n+ additional = aws_iam_policy.additional.arn\n+ }\n }\n\n self_managed_node_groups = {\n spot = {\n instance_type = \"m5.large\"\n instance_market_options = {\n market_type = \"spot\"\n }\n\n pre_bootstrap_user_data = <<-EOT\n echo \"foo\"\n export FOO=bar\n EOT\n\n bootstrap_extra_args = \"--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'\"\n\n post_bootstrap_user_data = <<-EOT\n cd /tmp\n sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm\n sudo systemctl enable amazon-ssm-agent\n sudo systemctl start amazon-ssm-agent\n EOT\n\n- create_security_group = true\n- security_group_name = \"eks-managed-node-group-complete-example\"\n- security_group_use_name_prefix = false\n- security_group_description = \"EKS managed node group complete example security group\"\n- security_group_rules = {}\n- security_group_tags = {}\n }\n }\n\n # EKS Managed Node Group(s)\n eks_managed_node_group_defaults = {\n ami_type = \"AL2_x86_64\"\n instance_types = [\"m6i.large\", \"m5.large\", \"m5n.large\", \"m5zn.large\"]\n\n attach_cluster_primary_security_group = true\n vpc_security_group_ids = [aws_security_group.additional.id]\n- iam_role_additional_policies = [aws_iam_policy.additional.arn]\n+ iam_role_additional_policies = {\n+ additional = aws_iam_policy.additional.arn\n+ }\n }\n\n eks_managed_node_groups = {\n blue = {}\n green = {\n min_size = 1\n max_size = 10\n desired_size = 1\n\n instance_types = [\"t3.large\"]\n capacity_type = \"SPOT\"\n labels = {\n Environment = \"test\"\n GithubRepo = \"terraform-aws-eks\"\n GithubOrg = \"terraform-aws-modules\"\n }\n\n taints = {\n dedicated = {\n key = \"dedicated\"\n value = \"gpuGroup\"\n effect = \"NO_SCHEDULE\"\n }\n }\n\n update_config = {\n max_unavailable_percentage = 33 # or set `max_unavailable`\n }\n\n- create_security_group = true\n- security_group_name = \"eks-managed-node-group-complete-example\"\n- security_group_use_name_prefix = false\n- security_group_description = \"EKS managed node group complete example security group\"\n- security_group_rules = {}\n- security_group_tags = {}\n\n tags = {\n ExtraTag = \"example\"\n }\n }\n }\n\n # Fargate Profile(s)\n fargate_profile_defaults = {\n- iam_role_additional_policies = [aws_iam_policy.additional.arn]\n+ iam_role_additional_policies = {\n+ additional = aws_iam_policy.additional.arn\n+ }\n }\n\n fargate_profiles = {\n default = {\n name = \"default\"\n selectors = [\n {\n namespace = \"kube-system\"\n labels = {\n k8s-app = \"kube-dns\"\n }\n },\n {\n namespace = \"default\"\n }\n ]\n\n tags = {\n Owner = \"test\"\n }\n\n timeouts = {\n create = \"20m\"\n delete = \"20m\"\n }\n }\n }\n\n # OIDC Identity provider\n cluster_identity_providers = {\n cognito = {\n client_id = \"702vqsrjicklgb7c5b7b50i1gc\"\n issuer_url = \"https://cognito-idp.us-west-2.amazonaws.com/us-west-2_re1u6bpRA\"\n username_claim = \"email\"\n groups_claim = \"cognito:groups\"\n groups_prefix = \"gid:\"\n }\n }\n\n # aws-auth configmap\n manage_aws_auth_configmap = true\n\n aws_auth_node_iam_role_arns_non_windows = [\n module.eks_managed_node_group.iam_role_arn,\n module.self_managed_node_group.iam_role_arn,\n ]\n aws_auth_fargate_profile_pod_execution_role_arns = [\n module.fargate_profile.fargate_profile_pod_execution_role_arn\n ]\n\n aws_auth_roles = [\n {\n rolearn = \"arn:aws:iam::66666666666:role/role1\"\n username = \"role1\"\n groups = [\"system:masters\"]\n },\n ]\n\n aws_auth_users = [\n {\n userarn = \"arn:aws:iam::66666666666:user/user1\"\n username = \"user1\"\n groups = [\"system:masters\"]\n },\n {\n userarn = \"arn:aws:iam::66666666666:user/user2\"\n username = \"user2\"\n groups = [\"system:masters\"]\n },\n ]\n\n aws_auth_accounts = [\n \"777777777777\",\n \"888888888888\",\n ]\n\n tags = local.tags\n}\nThe following Terraform state move commands are optional but recommended if you are providing additional IAM policies that are to be attached to IAM roles created by this module (cluster IAM role, node group IAM role, Fargate profile IAM role). Because the resources affected are aws_iam_role_policy_attachment, in theory, you could get away with simply applying the configuration and letting Terraform detach and re-attach the policies. However, during this brief period of update, you could experience permission failures as the policy is detached and re-attached, and therefore the state move route is recommended.
Where \"<POLICY_ARN>\" is specified, this should be replaced with the full ARN of the policy, and \"<POLICY_MAP_KEY>\" should be replaced with the key used in the iam_role_additional_policies map for the associated policy. For example, if you have the followingv19.x configuration:
...\n # This is demonstrating the cluster IAM role additional policies\n iam_role_additional_policies = {\n additional = aws_iam_policy.additional.arn\n }\n ...\nThe associated state move command would look similar to (albeit with your correct policy ARN):
terraform state mv 'module.eks.aws_iam_role_policy_attachment.this[\"arn:aws:iam::111111111111:policy/ex-complete-additional\"]' 'module.eks.aws_iam_role_policy_attachment.additional[\"additional\"]'\nIf you are not providing any additional IAM policies, no actions are required.
"},{"location":"UPGRADE-19.0.html#cluster-iam-role","title":"Cluster IAM Role","text":"Repeat for each policy provided in iam_role_additional_policies:
terraform state mv 'module.eks.aws_iam_role_policy_attachment.this[\"<POLICY_ARN>\"]' 'module.eks.aws_iam_role_policy_attachment.additional[\"<POLICY_MAP_KEY>\"]'\nWhere \"<NODE_GROUP_KEY>\" is the key used in the eks_managed_node_groups map for the associated node group. Repeat for each policy provided in iam_role_additional_policies in either/or eks_managed_node_group_defaults or the individual node group definitions:
terraform state mv 'module.eks.module.eks_managed_node_group[\"<NODE_GROUP_KEY>\"].aws_iam_role_policy_attachment.this[\"<POLICY_ARN>\"]' 'module.eks.module.eks_managed_node_group[\"<NODE_GROUP_KEY>\"].aws_iam_role_policy_attachment.additional[\"<POLICY_MAP_KEY>\"]'\nWhere \"<NODE_GROUP_KEY>\" is the key used in the self_managed_node_groups map for the associated node group. Repeat for each policy provided in iam_role_additional_policies in either/or self_managed_node_group_defaults or the individual node group definitions:
terraform state mv 'module.eks.module.self_managed_node_group[\"<NODE_GROUP_KEY>\"].aws_iam_role_policy_attachment.this[\"<POLICY_ARN>\"]' 'module.eks.module.self_managed_node_group[\"<NODE_GROUP_KEY>\"].aws_iam_role_policy_attachment.additional[\"<POLICY_MAP_KEY>\"]'\nWhere \"<FARGATE_PROFILE_KEY>\" is the key used in the fargate_profiles map for the associated profile. Repeat for each policy provided in iam_role_additional_policies in either/or fargate_profile_defaults or the individual profile definitions:
terraform state mv 'module.eks.module.fargate_profile[\"<FARGATE_PROFILE_KEY>\"].aws_iam_role_policy_attachment.this[\"<POLICY_ARN>\"]' 'module.eks.module.fargate_profile[\"<FARGATE_PROFILE_KEY>\"].aws_iam_role_policy_attachment.additional[\"<POLICY_MAP_KEY>\"]'\nPlease consult the examples directory for reference example configurations. If you find a bug, please open an issue with supporting configuration to reproduce.
v5.34v1.3 to support Terraform state moved blocks as well as other advanced featuresresolve_conflicts argument within the cluster_addons configuration has been replaced with resolve_conflicts_on_create and resolve_conflicts_on_update now that resolve_conflicts is deprecatedpreserve argument of cluster_addonsis now set to true. This has shown to be useful for users deprovisioning clusters while avoiding the situation where the CNI is deleted too early and causes resources to be left orphaned resulting in conflicts.irsa naming convention has been removed, along with an update to the Karpenter controller IAM policy to align with Karpenter's v1beta1/v0.32 changes. Instead of referring to the role as irsa or pod_identity, its simply just an IAM role used by the Karpenter controller and there is support for use with either IRSA and/or Pod Identity (default) at this timeaws-auth ConfigMap resources have been moved to a standalone sub-module. This removes the Kubernetes provider requirement from the main module and allows for the aws-auth ConfigMap to be managed independently of the main module. This sub-module will be removed entirely in the next major release.API_AND_CONFIG_MAP. Support for CONFIG_MAP is no longer supported; instead you will need to use API_AND_CONFIG_MAP at minimumspot_interrupt updated to correct mis-spelling (was spot_interupt). This will cause the rule to be replacedTo give users advanced notice and provide some future direction for this module, these are the following changes we will be looking to make in the next major release of this module:
aws-auth sub-module will be removed entirely from the project. Since this sub-module is captured in the v20.x releases, users can continue using it even after the module moves forward with the next major version. The long term strategy and direction is cluster access entry and to rely only on the AWS Terraform provider.authentication_mode will change to API. Aligning with point 1 above, this is a one way change, but users are free to specify the value of their choosing in place of this default (when the change is made). This module will proceed with an EKS API first strategy.terraform-aws-autoscaling module. At minimum, it makes sense to replace most of functionality in the self-managed node group module with this external module, but its not yet clear if there is any benefit of using it in the EKS managed node group sub-module. The interface that users interact with will stay the same, the changes will be internal to the implementation and we will do everything we can to keep the disruption to a minimum.platform variable will be replaced and instead ami_type will become the standard across both self-managed node group(s) and EKS managed node group(s). As EKS expands its portfolio of supported operating systems, the ami_type is better suited to associate the correct user data format to the respective OS. The platform variable is a legacy artifact of self-managed node groups but not as descriptive as the ami_type, and therefore it will be removed in favor of ami_type.bootstrap_cluster_creator_admin_permissions setting on the control plane has been hardcoded to false since this operation is a one time operation only at cluster creation per the EKS API. Instead, users can enable/disable enable_cluster_creator_admin_permissions at any time to achieve the same functionality. This takes the identity that Terraform is using to make API calls and maps it into a cluster admin via an access entry. For users on existing clusters, you will need to remove the default cluster administrator that was created by EKS prior to the cluster access entry APIs - see the section Removing the default cluster administrator for more details.instance_maintenance_policy and have added max_healthy_percentage, scale_in_protected_instances, and standby_instances arguments to the instance_refresh.preferences blocksts:AssumeRole permissions by services, the use of dynamically looking up the DNS suffix has been replaced with the static value of amazonaws.com. This does not appear to change by partition and instead requires users to set this manually for non-commercial regions.kms_key_enable_default_policy has changed from false to true to align with the default behavior of the aws_kms_key resourcecreate_instance_profile has changed from true to false to align with the changes in Karpenter v0.32create_instance_profile default value has changed from true to false. Starting with Karpenter v0.32.0, Karpenter accepts an IAM role and creates the EC2 instance profile used by the nodescomplete example has been removed due to its redundancy with the other examplesRemoved variables:
cluster_iam_role_dns_suffix - replaced with a static string of amazonaws.com
manage_aws_auth_configmapcreate_aws_auth_configmapaws_auth_node_iam_role_arns_non_windowsaws_auth_node_iam_role_arns_windowsaws_auth_fargate_profile_pod_execution_role_arnaws_auth_rolesaws_auth_usersaws_auth_accounts
Karpenter
irsa_tag_keyirsa_tag_valuesirsa_subnet_account_idenable_karpenter_instance_profile_creationRenamed variables:
Karpenter
create_irsa -> create_iam_roleirsa_name -> iam_role_nameirsa_use_name_prefix -> iam_role_name_prefixirsa_path -> iam_role_pathirsa_description -> iam_role_descriptionirsa_max_session_duration -> iam_role_max_session_durationirsa_permissions_boundary_arn -> iam_role_permissions_boundary_arnirsa_tags -> iam_role_tagspolicies -> iam_role_policiesirsa_policy_name -> iam_policy_nameirsa_ssm_parameter_arns -> ami_id_ssm_parameter_arnscreate_iam_role -> create_node_iam_roleiam_role_additional_policies -> node_iam_role_additional_policiespolicies -> iam_role_policiesiam_role_arn -> node_iam_role_arniam_role_name -> node_iam_role_nameiam_role_name_prefix -> node_iam_role_name_prefixiam_role_path -> node_iam_role_pathiam_role_description -> node_iam_role_descriptioniam_role_max_session_duration -> node_iam_role_max_session_durationiam_role_permissions_boundary_arn -> node_iam_role_permissions_boundary_arniam_role_attach_cni_policy -> node_iam_role_attach_cni_policyiam_role_additional_policies -> node_iam_role_additional_policiesiam_role_tags -> node_iam_role_tagsAdded variables:
create_access_entry
enable_cluster_creator_admin_permissionsauthentication_modeaccess_entriescloudwatch_log_group_class
Karpenter
iam_policy_nameiam_policy_use_name_prefixiam_policy_descriptioniam_policy_pathenable_irsacreate_access_entryaccess_entry_typeSelf-managed node group
instance_maintenance_policycreate_access_entryiam_role_arnRemoved outputs:
aws_auth_configmap_yaml
Renamed outputs:
Karpenter
irsa_name -> iam_role_nameirsa_arn -> iam_role_arnirsa_unique_id -> iam_role_unique_idrole_name -> node_iam_role_namerole_arn -> node_iam_role_arnrole_unique_id -> node_iam_role_unique_idAdded outputs:
access_entries
Karpenter
node_access_entry_arnSelf-managed node group
access_entry_arn module \"eks\" {\n source = \"terraform-aws-modules/eks/aws\"\n- version = \"~> 19.21\"\n+ version = \"~> 20.0\"\n\n# If you want to maintain the current default behavior of v19.x\n+ kms_key_enable_default_policy = false\n\n- manage_aws_auth_configmap = true\n\n- aws_auth_roles = [\n- {\n- rolearn = \"arn:aws:iam::66666666666:role/role1\"\n- username = \"role1\"\n- groups = [\"custom-role-group\"]\n- },\n- ]\n\n- aws_auth_users = [\n- {\n- userarn = \"arn:aws:iam::66666666666:user/user1\"\n- username = \"user1\"\n- groups = [\"custom-users-group\"]\n- },\n- ]\n}\n\n+ module \"eks_aws_auth\" {\n+ source = \"terraform-aws-modules/eks/aws//modules/aws-auth\"\n+ version = \"~> 20.0\"\n\n+ manage_aws_auth_configmap = true\n\n+ aws_auth_roles = [\n+ {\n+ rolearn = \"arn:aws:iam::66666666666:role/role1\"\n+ username = \"role1\"\n+ groups = [\"custom-role-group\"]\n+ },\n+ ]\n\n+ aws_auth_users = [\n+ {\n+ userarn = \"arn:aws:iam::66666666666:user/user1\"\n+ username = \"user1\"\n+ groups = [\"custom-users-group\"]\n+ },\n+ ]\n+ }\n module \"eks_karpenter\" {\n source = \"terraform-aws-modules/eks/aws//modules/karpenter\"\n- version = \"~> 19.21\"\n+ version = \"~> 20.0\"\n\n# If you wish to maintain the current default behavior of v19.x\n+ enable_irsa = true\n+ create_instance_profile = true\n\n# To avoid any resource re-creation\n+ iam_role_name = \"KarpenterIRSA-${module.eks.cluster_name}\"\n+ iam_role_description = \"Karpenter IAM role for service account\"\n+ iam_policy_name = \"KarpenterIRSA-${module.eks.cluster_name}\"\n+ iam_policy_description = \"Karpenter IAM role for service account\"\n}\nChanging the authentication_mode is a one-way decision. See announcement blog for further details:
Switching authentication modes on an existing cluster is a one-way operation. You can switch from CONFIG_MAP to API_AND_CONFIG_MAP. You can then switch from API_AND_CONFIG_MAP to API. You cannot revert these operations in the opposite direction. Meaning you cannot switch back to CONFIG_MAP or API_AND_CONFIG_MAP from API.
[!IMPORTANT] If migrating to cluster access entries and you will NOT have any entries that remain in the aws-auth ConfigMap, you do not need to remove the configmap from the statefile. You can simply follow the migration guide and once access entries have been created, you can let Terraform remove/delete the aws-auth ConfigMap.
If you WILL have entries that remain in the aws-auth ConfigMap, then you will need to remove the ConfigMap resources from the statefile to avoid any disruptions. When you add the new aws-auth sub-module and apply the changes, the sub-module will upsert the ConfigMap on the cluster. Provided the necessary entries are defined in that sub-module's definition, it will \"re-adopt\" the ConfigMap under Terraform's control.
When using authentication_mode = \"API_AND_CONFIG_MAP\" and there are entries that will remain in the configmap (entries that cannot be replaced by cluster access entry), you will first need to update the authentication_mode on the cluster to \"API_AND_CONFIG_MAP\". To help make this upgrade process easier, a copy of the changes defined in the v20.0.0 PR have been captured here but with the aws-auth components still provided in the module. This means you get the equivalent of the v20.0.0 module, but it still includes support for the aws-auth configmap. You can follow the provided README on that interim migration module for the order of execution and return here once the authentication_mode has been updated to \"API_AND_CONFIG_MAP\". Note - EKS automatically adds access entries for the roles used by EKS managed node groups and Fargate profiles; users do not need to do anything additional for these roles.
Once the authentication_mode has been updated, next you will need to remove the configmap from the statefile to avoid any disruptions:
[!NOTE] This is only required if there are entries that will remain in the aws-auth ConfigMap after migrating. Otherwise, you can skip this step and let Terraform destroy the ConfigMap.
terraform state rm 'module.eks.kubernetes_config_map_v1_data.aws_auth[0]'\nterraform state rm 'module.eks.kubernetes_config_map.aws_auth[0]' # include if Terraform created the original configmap\nIf you are using Terraform v1.7+, you can utilize the remove to facilitate both the removal of the configmap through code. You can create a fork/clone of the provided migration module and add the remove blocks and apply those changes before proceeding. We do not want to force users onto the bleeding edge with this module, so we have not included remove support at this time.
Once the configmap has been removed from the statefile, you can add the new aws-auth sub-module and copy the relevant definitions from the EKS module over to the new aws-auth sub-module definition (see before after diff above). When you apply the changes with the new sub-module, the configmap in the cluster will get updated with the contents provided in the sub-module definition, so please be sure all of the necessary entries are added before applying the changes. In the before/example above - the configmap would remove any entries for roles used by node groups and/or Fargate Profiles, but maintain the custom entries for users and roles passed into the module definition.
In order to switch to API only using cluster access entry, you first need to update the authentication_mode on the cluster to API_AND_CONFIG_MAP without modifying the aws-auth configmap. To help make this upgrade process easier, a copy of the changes defined in the v20.0.0 PR have been captured here but with the aws-auth components still provided in the module. This means you get the equivalent of the v20.0.0 module, but it still includes support for the aws-auth configmap. You can follow the provided README on that interim migration module for the order of execution and return here once the authentication_mode has been updated to \"API_AND_CONFIG_MAP\". Note - EKS automatically adds access entries for the roles used by EKS managed node groups and Fargate profiles; users do not need to do anything additional for these roles.
Once the authentication_mode has been updated, you can update the authentication_mode on the cluster to API and remove the aws-auth configmap components.
If you have any questions regarding this upgrade process, please consult the examples directory: If you find a bug, please open an issue with supporting configuration to reproduce.
v1.5.7 is now minimum supported versionv6.0.0 is now minimum supported versionv4.0.0 is now minimum supported versionaws-auth sub-module has been removed. Users who wish to utilize its functionality can continue to do so by specifying a v20.x version, or ~> v20.0 version constraint in their module source.bootstrap_self_managed_addons is now hardcoded to false. This is a legacy setting and instead users should utilize the EKS addons API, which is what this module does by default. In conjunction with this change, the bootstrap_self_managed_addons is now ignored by the module to aid in upgrading without disruption (otherwise it would require cluster re-creation).enable_efa_support or creating placement groups within a node group, users must now specify the correct subnet_ids; the module no longer tries to automatically select a suitable subnet.ami_type now defaults to AL2023_x86_64_STANDARDenable_monitoring is now set to false by defaultenable_efa_only is now set to true by defaultuse_latest_ami_release_version is now set to true by defaultami_type now defaults to AL2023_x86_64_STANDARDenable_monitoring is now set to false by defaultenable_efa_only is now set to true by defaultv1 have been removed (i.e. v0.33); the v1 policy is now used by defaultcreate_pod_identity_association is now set to true by defaultaddons.resolve_conflicts_on_create is now set to \"NONE\" by default (was \"OVERWRITE\").addons.most_recent is now set to true by default (was false).cluster_identity_providers.issuer_url is now required to be set by users; the prior incorrect default has been removed. See https://github.com/terraform-aws-modules/terraform-aws-eks/pull/3055 and https://github.com/kubernetes/kubernetes/pull/123561 for more details.oidc-eks endpoint instead of oidc.eks. This is to align with https://github.com/aws/containers-roadmap/issues/2038#issuecomment-2278450601region parameter to specify the AWS region for the resources created if different from the provider region.enable_efa_support both at the cluster level (to add the appropriate security group rules to the shared node security group) as well as the node group level. However, its not always desirable to have these rules across ALL node groups when they are really only required on the node group where EFA is utilized. And similarly for other use cases, users can create custom rules for a specific node group instead of apply across ALL node groups.object types in place of the previously used any type.v4.0 to support the same version requirements as well as the new region argument.Removed variables:
enable_efa_support - users only need to set this within the node group configuration, as the module no longer manages EFA support at the cluster level.enable_security_groups_for_pods - users can instead attach the arn:aws:iam::aws:policy/AmazonEKSVPCResourceController policy via iam_role_additional_policies if using security groups for pods.eks-managed-node-group sub-modulecluster_service_ipv4_cidr - users should use cluster_service_cidr instead (for either IPv4 or IPv6).elastic_gpu_specificationselastic_inference_acceleratorplatform - this is superseded by ami_typeplacement_group_strategy - set to cluster by the moduleplacement_group_az - users will need to specify the correct subnet in subnet_idscreate_scheduleschedulesself-managed-node-group sub-moduleelastic_gpu_specificationselastic_inference_acceleratorplatform - this is superseded by ami_typecreate_scheduleschedulesplacement_group_az - users will need to specify the correct subnet in subnet_idshibernation_options - not valid in EKSmin_elb_capacity - not valid in EKSwait_for_elb_capacity - not valid in EKSwait_for_capacity_timeout - not valid in EKSdefault_cooldown - not valid in EKStarget_group_arns - not valid in EKSservice_linked_role_arn - not valid in EKSwarm_pool - not valid in EKSfargate-profile sub-modulekarpenter sub-moduleenable_v1_permissions - v1 permissions are now the defaultenable_irsairsa_oidc_provider_arnirsa_namespace_service_accountsirsa_assume_role_condition_testRenamed variables:
cluster_* have been stripped of the prefix to better match the underlying API:cluster_name -> namecluster_version -> kubernetes_versioncluster_enabled_log_types -> enabled_log_typescluster_force_update_version -> force_update_versioncluster_compute_config -> compute_configcluster_upgrade_policy -> upgrade_policycluster_remote_network_config -> remote_network_configcluster_zonal_shift_config -> zonal_shift_configcluster_additional_security_group_ids -> additional_security_group_idscluster_endpoint_private_access -> endpoint_private_accesscluster_endpoint_public_access -> endpoint_public_accesscluster_endpoint_public_access_cidrs -> endpoint_public_access_cidrscluster_ip_family -> ip_familycluster_service_ipv4_cidr -> service_ipv4_cidrcluster_service_ipv6_cidr -> service_ipv6_cidrcluster_encryption_config -> encryption_configcreate_cluster_primary_security_group_tags -> create_primary_security_group_tagscluster_timeouts -> timeoutscreate_cluster_security_group -> create_security_groupcluster_security_group_id -> security_group_idcluster_security_group_name -> security_group_namecluster_security_group_use_name_prefix -> security_group_use_name_prefixcluster_security_group_description -> security_group_descriptioncluster_security_group_additional_rules -> security_group_additional_rulescluster_security_group_tags -> security_group_tagscluster_encryption_policy_use_name_prefix -> encryption_policy_use_name_prefixcluster_encryption_policy_name -> encryption_policy_namecluster_encryption_policy_description -> encryption_policy_descriptioncluster_encryption_policy_path -> encryption_policy_pathcluster_encryption_policy_tags -> encryption_policy_tagscluster_addons -> addonscluster_addons_timeouts -> addons_timeoutscluster_identity_providers -> identity_providerseks-managed-node-group sub-modulecluster_version -> kubernetes_versionself-managed-node-group sub-modulecluster_version -> kubernetes_versiondelete_timeout -> timeoutsfargate-profile sub-modulekarpenter sub-moduleAdded variables:
regioneks-managed-node-group sub-moduleregionpartition - added to reduce number of GET requests from data sources when possibleaccount_id - added to reduce number of GET requests from data sources when possiblecreate_security_groupsecurity_group_namesecurity_group_use_name_prefixsecurity_group_descriptionsecurity_group_ingress_rulessecurity_group_egress_rulessecurity_group_tagsself-managed-node-group sub-moduleregionpartition - added to reduce number of GET requests from data sources when possibleaccount_id - added to reduce number of GET requests from data sources when possiblecreate_security_groupsecurity_group_namesecurity_group_use_name_prefixsecurity_group_descriptionsecurity_group_ingress_rulessecurity_group_egress_rulessecurity_group_tagsfargate-profile sub-moduleregionpartition - added to reduce number of GET requests from data sources when possibleaccount_id - added to reduce number of GET requests from data sources when possiblekarpenter sub-moduleregionRemoved outputs:
eks-managed-node-group sub-moduleplatform - this is superseded by ami_typeautoscaling_group_schedule_arnsself-managed-node-group sub-moduleplatform - this is superseded by ami_typeautoscaling_group_schedule_arnsfargate-profile sub-modulekarpenter sub-moduleRenamed outputs:
eks-managed-node-group sub-moduleself-managed-node-group sub-modulefargate-profile sub-modulekarpenter sub-moduleAdded outputs:
eks-managed-node-group sub-modulesecurity_group_arnsecurity_group_idself-managed-node-group sub-modulesecurity_group_arnsecurity_group_idfargate-profile sub-modulekarpenter sub-modulemodule \"eks\" {\n source = \"terraform-aws-modules/eks/aws\"\n version = \"~> 20.0\"\n\n # Truncated for brevity ...\n # Renamed variables are not shown here, please refer to the full list above.\n\n enable_efa_support = true\n\n eks_managed_node_group_defaults = {\n iam_role_additional_policies = {\n AmazonSSMManagedInstanceCore = \"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore\"\n }\n }\n\n eks_managed_node_groups = {\n efa = {\n ami_type = \"AL2023_x86_64_NVIDIA\"\n instance_types = [\"p5e.48xlarge\"]\n\n enable_efa_support = true\n enable_efa_only = true\n }\n }\n\n self_managed_node_groups = {\n example = {\n use_mixed_instances_policy = true\n mixed_instances_policy = {\n instances_distribution = {\n on_demand_base_capacity = 0\n on_demand_percentage_above_base_capacity = 0\n on_demand_allocation_strategy = \"lowest-price\"\n spot_allocation_strategy = \"price-capacity-optimized\"\n }\n\n # ASG configuration\n override = [\n {\n instance_requirements = {\n cpu_manufacturers = [\"intel\"]\n instance_generations = [\"current\", \"previous\"]\n spot_max_price_percentage_over_lowest_price = 100\n\n vcpu_count = {\n min = 1\n }\n\n allowed_instance_types = [\"t*\", \"m*\"]\n }\n }\n ]\n }\n }\n }\n}\nmodule \"eks\" {\n source = \"terraform-aws-modules/eks/aws\"\n version = \"~> 21.0\"\n\n # Truncated for brevity ...\n # Renamed variables are not shown here, please refer to the full list above.\n\n eks_managed_node_groups = {\n efa = {\n ami_type = \"AL2023_x86_64_NVIDIA\"\n instance_types = [\"p5e.48xlarge\"]\n\n iam_role_additional_policies = {\n AmazonSSMManagedInstanceCore = \"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore\"\n }\n\n enable_efa_support = true\n\n subnet_ids = element(module.vpc.private_subnets, 0)\n }\n }\n\n self_managed_node_groups = {\n example = {\n use_mixed_instances_policy = true\n mixed_instances_policy = {\n instances_distribution = {\n on_demand_base_capacity = 0\n on_demand_percentage_above_base_capacity = 0\n on_demand_allocation_strategy = \"lowest-price\"\n spot_allocation_strategy = \"price-capacity-optimized\"\n }\n\n # ASG configuration\n # Need to wrap in `launch_template` now\n launch_template = {\n override = [\n {\n instance_requirements = {\n cpu_manufacturers = [\"intel\"]\n instance_generations = [\"current\", \"previous\"]\n spot_max_price_percentage_over_lowest_price = 100\n\n vcpu_count = {\n min = 1\n }\n\n allowed_instance_types = [\"t*\", \"m*\"]\n }\n }\n ]\n }\n }\n }\n }\n}\nNo state changes required.
"},{"location":"compute_resources.html","title":"Compute Resources","text":""},{"location":"compute_resources.html#table-of-contents","title":"Table of Contents","text":"\u2139\ufe0f Only the pertinent attributes are shown below for brevity
"},{"location":"compute_resources.html#eks-managed-node-groups","title":"EKS Managed Node Groups","text":"Refer to the EKS Managed Node Group documentation documentation for service related details.
use_custom_launch_template to false: eks_managed_node_groups = {\n default = {\n use_custom_launch_template = false\n }\n }\n eks_managed_node_groups = {\n bottlerocket_default = {\n use_custom_launch_template = false\n\n ami_type = \"BOTTLEROCKET_x86_64\"\n }\n }\n eks_managed_node_groups = {\n bottlerocket_prepend_userdata = {\n ami_type = \"BOTTLEROCKET_x86_64\"\n\n bootstrap_extra_args = <<-EOT\n # extra args added\n [settings.kernel]\n lockdown = \"integrity\"\n EOT\n }\n }\n eks_managed_node_groups = {\n custom_ami = {\n ami_id = \"ami-0caf35bc73450c396\"\n ami_type = \"AL2023_x86_64_STANDARD\"\n\n # By default, EKS managed node groups will not append bootstrap script;\n # this adds it back in using the default template provided by the module\n # Note: this assumes the AMI provided is an EKS optimized AMI derivative\n enable_bootstrap_user_data = true\n\n cloudinit_pre_nodeadm = [{\n content = <<-EOT\n ---\n apiVersion: node.eks.aws/v1alpha1\n kind: NodeConfig\n spec:\n kubelet:\n config:\n shutdownGracePeriod: 30s\n EOT\n content_type = \"application/node.eks.aws\"\n }]\n\n # This is only possible when `ami_id` is specified, indicating a custom AMI\n cloudinit_post_nodeadm = [{\n content = <<-EOT\n echo \"All done\"\n EOT\n content_type = \"text/x-shellscript; charset=\\\"us-ascii\\\"\"\n }]\n }\n }\n eks_managed_node_groups = {\n bottlerocket_custom_ami = {\n ami_id = \"ami-0ff61e0bcfc81dc94\"\n ami_type = \"BOTTLEROCKET_x86_64\"\n\n # use module user data template to bootstrap\n enable_bootstrap_user_data = true\n # this will get added to the template\n bootstrap_extra_args = <<-EOT\n # extra args added\n [settings.kernel]\n lockdown = \"integrity\"\n\n [settings.kubernetes.node-labels]\n \"label1\" = \"foo\"\n \"label2\" = \"bar\"\n\n [settings.kubernetes.node-taints]\n \"dedicated\" = \"experimental:PreferNoSchedule\"\n \"special\" = \"true:NoSchedule\"\n EOT\n }\n }\nSee the examples/eks-managed-node-group/ example for a working example of various configurations.
Refer to the Self Managed Node Group documentation documentation for service related details.
self-managed-node-group uses the latest AWS EKS Optimized AMI (Linux) for the given Kubernetes version by default: kubernetes_version = \"1.33\"\n\n # This self managed node group will use the latest AWS EKS Optimized AMI for Kubernetes 1.33\n self_managed_node_groups = {\n default = {}\n }\nami_type as one of the respective \"BOTTLEROCKET_*\" types and supply a Bottlerocket OS AMI: kubernetes_version = \"1.33\"\n\n self_managed_node_groups = {\n bottlerocket = {\n ami_id = data.aws_ami.bottlerocket_ami.id\n ami_type = \"BOTTLEROCKET_x86_64\"\n }\n }\nSee the examples/self-managed-node-group/ example for a working example of various configurations.
Fargate profiles are straightforward to use and therefore no further details are provided here. See the tests/fargate-profile/ tests for a working example of various configurations.
disk_size or remote_access does not make any changesexpect exactly one securityGroup tagged with kubernetes.io/cluster/<NAME> ...desired_size is modified?disk_size or remote_access does not make any changes","text":"disk_size, and remote_access can only be set when using the EKS managed node group default launch template. This module defaults to providing a custom launch template to allow for custom security groups, tag propagation, etc. If you wish to forgo the custom launch template route, you can set use_custom_launch_template = false and then you can set disk_size and remote_access.
expect exactly one securityGroup tagged with kubernetes.io/cluster/<CLUSTER_NAME> ...","text":"\u26a0\ufe0f <CLUSTER_NAME> would be the name of your cluster
By default, EKS creates a cluster primary security group that is created outside of the module and the EKS service adds the tag { \"kubernetes.io/cluster/<CLUSTER_NAME>\" = \"owned\" }. This on its own does not cause any conflicts for addons such as the AWS Load Balancer Controller until users decide to attach both the cluster primary security group and the shared node security group created by the module (by setting attach_cluster_primary_security_group = true). The issue is not with having multiple security groups in your account with this tag key:value combination, but having multiple security groups with this tag key:value combination attached to nodes in the same cluster. There are a few ways to resolve this depending on your use case/intentions:
create_node_security_group = false # default is true\n\n eks_managed_node_group = {\n example = {\n attach_cluster_primary_security_group = true # default is false\n }\n }\n # Or for self-managed\n self_managed_node_group = {\n example = {\n attach_cluster_primary_security_group = true # default is false\n }\n }\n eks_managed_node_group = {\n example = {\n attach_cluster_primary_security_group = true # default is false\n }\n }\n # Or for self-managed\n self_managed_node_group = {\n example = {\n attach_cluster_primary_security_group = true # default is false\n }\n }\nIn theory, if you are attaching the cluster primary security group, you shouldn't need to use the shared node security group created by the module. However, this is left up to users to decide for their requirements and use case.
If you choose to use Custom Networking, make sure to only attach the security groups matching your choice above in your ENIConfig resources. This will ensure you avoid redundant tags.
"},{"location":"faq.html#why-are-nodes-not-being-registered","title":"Why are nodes not being registered?","text":"Nodes not being able to register with the EKS control plane is generally due to networking mis-configurations.
If you require a public endpoint, setting up both (public and private) and restricting the public endpoint via setting cluster_endpoint_public_access_cidrs is recommended. More info regarding communication with an endpoint is available here.
Nodes need to be able to contact the EKS cluster endpoint. By default, the module only creates a public endpoint. To access the endpoint, the nodes need outgoing internet access:
Nodes in private subnets: via a NAT gateway or instance along with the appropriate routing rules
Important: If you apply only the public endpoint and configure the cluster_endpoint_public_access_cidrs to restrict access, know that EKS nodes will also use the public endpoint and you must allow access to the endpoint. If not, then your nodes will fail to work correctly.
The private endpoint can also be enabled by setting cluster_endpoint_private_access = true. Ensure that VPC DNS resolution and hostnames are also enabled for your VPC when the private endpoint is enabled.
Nodes need to be able to connect to other AWS services to function (download container images, make API calls to assume roles, etc.). If for some reason you cannot enable public internet access for nodes you can add VPC endpoints to the relevant services: EC2 API, ECR API, ECR DKR and S3.
desired_size is modified?","text":"The module is configured to ignore this value. Unfortunately, Terraform does not support variables within the lifecycle block. The setting is ignored to allow autoscaling via controllers such as cluster autoscaler or Karpenter to work properly and without interference by Terraform. Changing the desired count must be handled outside of Terraform once the node group is created.
:info: See this for a workaround to this limitation.
"},{"location":"faq.html#how-do-i-access-compute-resource-attributes","title":"How do I access compute resource attributes?","text":"Examples of accessing the attributes of the compute resource(s) created by the root module are shown below. Note - the assumption is that your cluster module definition is named eks as in module \"eks\" { ... }:
eks_managed_role_arns = [for group in module.eks_managed_node_group : group.iam_role_arn]\nself_managed_role_arns = [for group in module.self_managed_node_group : group.iam_role_arn]\nfargate_profile_pod_execution_role_arns = [for group in module.fargate_profile : group.fargate_profile_pod_execution_role_arn]\nThe available EKS add-ons can be found here. You can also retrieve the available addons from the API using:
aws eks describe-addon-versions --query 'addons[*].addonName'\n[!NOTE] The available configuration values will vary between add-on versions, typically more configuration values will be added in later versions as functionality is enabled by EKS.
You can retrieve the configuration value schema for a given addon using the following command:
aws eks describe-addon-configuration --addon-name <value> --addon-version <value> --query 'configurationSchema' --output text | jq\nFor example:
aws eks describe-addon-configuration --addon-name coredns --addon-version v1.11.1-eksbuild.8 --query 'configurationSchema' --output text | jq\nReturns (at the time of writing):
{\n \"$ref\": \"#/definitions/Coredns\",\n \"$schema\": \"http://json-schema.org/draft-06/schema#\",\n \"definitions\": {\n \"Coredns\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"affinity\": {\n \"default\": {\n \"affinity\": {\n \"nodeAffinity\": {\n \"requiredDuringSchedulingIgnoredDuringExecution\": {\n \"nodeSelectorTerms\": [\n {\n \"matchExpressions\": [\n {\n \"key\": \"kubernetes.io/os\",\n \"operator\": \"In\",\n \"values\": [\n \"linux\"\n ]\n },\n {\n \"key\": \"kubernetes.io/arch\",\n \"operator\": \"In\",\n \"values\": [\n \"amd64\",\n \"arm64\"\n ]\n }\n ]\n }\n ]\n }\n },\n \"podAntiAffinity\": {\n \"preferredDuringSchedulingIgnoredDuringExecution\": [\n {\n \"podAffinityTerm\": {\n \"labelSelector\": {\n \"matchExpressions\": [\n {\n \"key\": \"k8s-app\",\n \"operator\": \"In\",\n \"values\": [\n \"kube-dns\"\n ]\n }\n ]\n },\n \"topologyKey\": \"kubernetes.io/hostname\"\n },\n \"weight\": 100\n }\n ]\n }\n }\n },\n \"description\": \"Affinity of the coredns pods\",\n \"type\": [\n \"object\",\n \"null\"\n ]\n },\n \"computeType\": {\n \"type\": \"string\"\n },\n \"corefile\": {\n \"description\": \"Entire corefile contents to use with installation\",\n \"type\": \"string\"\n },\n \"nodeSelector\": {\n \"additionalProperties\": {\n \"type\": \"string\"\n },\n \"type\": \"object\"\n },\n \"podAnnotations\": {\n \"properties\": {},\n \"title\": \"The podAnnotations Schema\",\n \"type\": \"object\"\n },\n \"podDisruptionBudget\": {\n \"description\": \"podDisruptionBudget configurations\",\n \"enabled\": {\n \"default\": true,\n \"description\": \"the option to enable managed PDB\",\n \"type\": \"boolean\"\n },\n \"maxUnavailable\": {\n \"anyOf\": [\n {\n \"pattern\": \".*%$\",\n \"type\": \"string\"\n },\n {\n \"type\": \"integer\"\n }\n ],\n \"default\": 1,\n \"description\": \"minAvailable value for managed PDB, can be either string or integer; if it's string, should end with %\"\n },\n \"minAvailable\": {\n \"anyOf\": [\n {\n \"pattern\": \".*%$\",\n \"type\": \"string\"\n },\n {\n \"type\": \"integer\"\n }\n ],\n \"description\": \"maxUnavailable value for managed PDB, can be either string or integer; if it's string, should end with %\"\n },\n \"type\": \"object\"\n },\n \"podLabels\": {\n \"properties\": {},\n \"title\": \"The podLabels Schema\",\n \"type\": \"object\"\n },\n \"replicaCount\": {\n \"type\": \"integer\"\n },\n \"resources\": {\n \"$ref\": \"#/definitions/Resources\"\n },\n \"tolerations\": {\n \"default\": [\n {\n \"key\": \"CriticalAddonsOnly\",\n \"operator\": \"Exists\"\n },\n {\n \"effect\": \"NoSchedule\",\n \"key\": \"node-role.kubernetes.io/control-plane\"\n }\n ],\n \"description\": \"Tolerations of the coredns pod\",\n \"items\": {\n \"type\": \"object\"\n },\n \"type\": \"array\"\n },\n \"topologySpreadConstraints\": {\n \"description\": \"The coredns pod topology spread constraints\",\n \"type\": \"array\"\n }\n },\n \"title\": \"Coredns\",\n \"type\": \"object\"\n },\n \"Limits\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"cpu\": {\n \"type\": \"string\"\n },\n \"memory\": {\n \"type\": \"string\"\n }\n },\n \"title\": \"Limits\",\n \"type\": \"object\"\n },\n \"Resources\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"limits\": {\n \"$ref\": \"#/definitions/Limits\"\n },\n \"requests\": {\n \"$ref\": \"#/definitions/Limits\"\n }\n },\n \"title\": \"Resources\",\n \"type\": \"object\"\n }\n }\n}\nIn order to run the documentation site locally, you will need to have the following installed locally:
pip install ...)mkdocs-materialmkdocs-include-markdown-pluginmkdocs-awesome-pages-pluginTo run the documentation site locally, run the following command from the root of the repository:
mkdocs serve\nOpening the documentation at the link posted in the terminal output (i.e. - http://127.0.0.1:8000/terraform-aws-eks/)
"},{"location":"network_connectivity.html","title":"Network Connectivity","text":""},{"location":"network_connectivity.html#cluster-endpoint","title":"Cluster Endpoint","text":""},{"location":"network_connectivity.html#public-endpoint-w-restricted-cidrs","title":"Public Endpoint w/ Restricted CIDRs","text":"When restricting the clusters public endpoint to only the CIDRs specified by users, it is recommended that you also enable the private endpoint, or ensure that the CIDR blocks that you specify include the addresses that nodes and Fargate pods (if you use them) access the public endpoint from.
Please refer to the AWS documentation for further information
"},{"location":"network_connectivity.html#security-groups","title":"Security Groups","text":"cluster_additional_security_group_ids variableLastly, users are able to opt in to attaching the primary security group automatically created by the EKS service by setting attach_cluster_primary_security_group = true from the root module for the respective node group (or set it within the node group defaults). This security group is not managed by the module; it is created by the EKS service. It permits all traffic within the domain of the security group as well as all egress traffic to the internet.
Node Group Security Group(s)
vpc_security_group_ids variableSee the example snippet below which adds additional security group rules to the cluster security group as well as the shared node security group (for node-to-node access). Users can use this extensibility to open up network access as they see fit using the security groups provided by the module:
...\n # Extend cluster security group rules\n security_group_additional_rules = {\n egress_nodes_ephemeral_ports_tcp = {\n description = \"To node 1025-65535\"\n protocol = \"tcp\"\n from_port = 1025\n to_port = 65535\n type = \"egress\"\n source_node_security_group = true\n }\n }\n\n # Extend node-to-node security group rules\n node_security_group_additional_rules = {\n ingress_self_all = {\n description = \"Node to node all ports/protocols\"\n protocol = \"-1\"\n from_port = 0\n to_port = 0\n type = \"ingress\"\n self = true\n }\n egress_all = {\n description = \"Node all egress\"\n protocol = \"-1\"\n from_port = 0\n to_port = 0\n type = \"egress\"\n cidr_blocks = [\"0.0.0.0/0\"]\n ipv6_cidr_blocks = [\"::/0\"]\n }\n }\n ...\n"},{"location":"user_data.html","title":"User Data & Bootstrapping","text":"
Users can see the various methods of using and providing user data through the user data tests as well more detailed information on the design and possible configurations via the user data module itself
"},{"location":"user_data.html#summary","title":"Summary","text":"ami_id, the service no longers supplies user data to bootstrap nodes; users can enable enable_bootstrap_user_data and use the module provided user data template, or provide their own user data templateBOTTLEROCKET_*, user data must be in TOML formatWINDOWS_*, user data must be in powershell/PS1 script formatAL2_* AMI types -> the user data template (bash/shell script) provided by the module is used as the default; users are able to provide their own user data templateAL2023_* AMI types -> the user data template (MIME multipart format) provided by the module is used as the default; users are able to provide their own user data templateBOTTLEROCKET_* AMI types -> the user data template (TOML file) provided by the module is used as the default; users are able to provide their own user data templateWINDOWS_* AMI types -> the user data template (powershell/PS1 script) provided by the module is used as the default; users are able to provide their own user data templateThe templates provided by the module can be found under the templates directory
"},{"location":"user_data.html#eks-managed-node-group","title":"EKS Managed Node Group","text":"When using an EKS managed node group, users have 2 primary routes for interacting with the bootstrap user data:
If a value for ami_id is not provided, users can supply additional user data that is pre-pended before the EKS Managed Node Group bootstrap user data. You can read more about this process from the AWS supplied documentation
Users can use the following variables to facilitate this process:
For AL2_*, BOTTLEROCKET_*, and WINDOWS_*:
pre_bootstrap_user_data = \"...\"\nFor AL2023_*
cloudinit_pre_nodeadm = [{\n content = <<-EOT\n ---\n apiVersion: node.eks.aws/v1alpha1\n kind: NodeConfig\n spec:\n ...\n EOT\n content_type = \"application/node.eks.aws\"\n}]\nIf a custom AMI is used, then per the AWS documentation, users will need to supply the necessary user data to bootstrap and register nodes with the cluster when launched. There are two routes to facilitate this bootstrapping process:
enable_bootstrap_user_data = true # to opt in to using the module supplied bootstrap user data template\npre_bootstrap_user_data = \"...\"\nbootstrap_extra_args = \"...\"\npost_bootstrap_user_data = \"...\"\ntemplatefile() for the respective AMI type are available for use in the supplied template, otherwise users will need to pre-render/pre-populate the template before supplying the final template to the module for rendering as user data.user_data_template_path = \"./your/user_data.sh\" # user supplied bootstrap user data template\npre_bootstrap_user_data = \"...\"\nbootstrap_extra_args = \"...\"\npost_bootstrap_user_data = \"...\"\npre_bootstrap_user_data and post_bootstrap_user_data are not valid since the bottlerocket OS handles when various settings are applied. If you wish to supply additional configuration settings when using bottlerocket, supply them via the bootstrap_extra_args variable. For the AL2_* AMI types, bootstrap_extra_args are settings that will be supplied to the AWS EKS Optimized AMI bootstrap script such as kubelet extra args, etc. See the bottlerocket GitHub repository documentation for more details on what settings can be supplied via the bootstrap_extra_args variable."},{"location":"user_data.html#self-managed-node-group","title":"Self Managed Node Group","text":"Self managed node groups require users to provide the necessary bootstrap user data. Users can elect to use the user data template provided by the module for their respective AMI type or provide their own user data template for rendering by the module.
enable_bootstrap_user_data = true # to opt in to using the module supplied bootstrap user data template\npre_bootstrap_user_data = \"...\"\nbootstrap_extra_args = \"...\"\npost_bootstrap_user_data = \"...\"\ntemplatefile() for the respective AMI type are available for use in the supplied template, otherwise users will need to pre-render/pre-populate the template before supplying the final template to the module for rendering as user data.user_data_template_path = \"./your/user_data.sh\" # user supplied bootstrap user data template\npre_bootstrap_user_data = \"...\"\nbootstrap_extra_args = \"...\"\npost_bootstrap_user_data = \"...\"\nThe rough flow of logic that is encapsulated within the _user_data module can be represented by the following diagram to better highlight the various manners in which user data can be populated.
"}]}