Notes: Setting up an EKS with Spot Instances using Terraform

5 minute read

The objective of this post is not to get a fully running AWS EKS cluster running with spot instances, but rather the key “pain points” I run into when trying to spin up this infrastructure using Terraform.

Overview

I am building:

  • 1 x AWS EKS cluster
  • using the AWS official ‘eks’ module
  • Spot Instances
  • 2 x Worker Groups (nodes)
  • Autoscaling Groups
  • using Terraform 0.12

Module: AWS EKS

Using the official EKS terraform module by AWS, looks like this:

module "eks" {
  source       = "terraform-aws-modules/eks/aws"
  version      = "12.2.0"
  cluster_name = local.cluster_name
  subnets      = data.terraform_remote_state.vpc.outputs.private_subnets

  tags = {
    Environment = "prod"
  }

  vpc_id = data.terraform_remote_state.vpc.outputs.vpc_id

  worker_groups = [
    {
      name  = "worker-group-1"
      instance_type = local.instance_type
      spot_price = local.spot_price
      asg_desired_capacity = local.asg_desired_capacity
      asg_max_size = local.asg_max_size
      asg_min_size = local.asg_min_size
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
      additional_userdata = "worker group config"
      tags = [{
          key = "worker-group-tag"
          value = "worker-group-1"
          propagate_at_launch = true
      }]
    },
    {
      name  = "worker-group-2"
      instance_type = local.instance_type
      spot_price = local.spot_price
      asg_desired_capacity = local.asg_desired_capacity
      asg_max_size = local.asg_max_size
      asg_min_size = local.asg_min_size
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_two.id]
      additional_userdata = "worker group config"
      tags = [{
          key = "worker-group-tag"
          value = "worker-group-2"
          propagate_at_launch = true
      }]
    },
  ]
}

Breaking down what’s going on here:

locals

I’m using a locals block for cluster specific variables

locals{
  cluster_name = "prod-eks-cluster"
  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  asg_desired_capacity = 1
  asg_max_size = 3
  asg_min_size = 1
  instance_type = "m4.large"
  spot_price = "0.20"
}

terraform_remote_state

I’m using terraform_remote_state data source to import the state-file of the VPC I created in another folder

data "terraform_remote_state" "vpc" {
  backend = "s3"

  config = {
    bucket = "tfstates3"
    key    = "prod/network/terraform.tfstate"
    region  = var.region
  }
}

These are the references to the vpc subnets and id

subnets      = data.terraform_remote_state.vpc.outputs.private_subnets
vpc_id = data.terraform_remote_state.vpc.outputs.vpc_id

Since v0.12 the .outputs. part is what calls whatever is defined in the modules outputs.tf file for the resources.

See more details on this below.

kubernetes provider

make sure you have these 3 resources:

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
  version                = "~> 1.11"
}

otherwise you’ll see this error near end of EKS setup

...
module.eks.aws_autoscaling_group.workers[1]: Creation complete after 1m50s [id=prod-eks-cluster-worker-group-220200917111118233900000009]

Error: Post "https://prod-bc5a71e2.hcp.australiaeast.azmk8s.io:443/api/v1/namespaces/kube-system/configmaps": dial tcp: lookup prod-bc5a71e2.hcp.australiaeast.azmk8s.io on 127.0.0.53:53: no such host

  on .terraform/modules/eks/aws_auth.tf line 64, in resource "kubernetes_config_map" "aws_auth":
  64: resource "kubernetes_config_map" "aws_auth" {
...

worker_groups

Self-explanatory, you can define however many “worker groups” you want (I think). In here you will define the autoscaling min, max, desired instances, and instance type (m4.large is the smallest type you can use with errors), and spot instance bid price (0.20).

  worker_groups = [
    {
      name  = "worker-group-1"
      instance_type = local.instance_type
      spot_price = local.spot_price
      asg_desired_capacity = local.asg_desired_capacity
      asg_max_size = local.asg_max_size
      asg_min_size = local.asg_min_size
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
      additional_userdata = "worker group config"
      tags = [{
          key = "worker-group-tag"
          value = "worker-group-1"
          propagate_at_launch = true
      }]
    },

The security groups referenced in additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id] (list), is in another security.tf and basically sets up all nodes to open port 22 from specific cidr_blocks.

Terraform: Remote State Files

If you have the “best practice” setup of having each component/section of your infrastructure layout in separate folders e.g. eks in one folder, vpc in another – and they have their own state files, which means they can’t just reference each other.

The solution is using terraform_remote_state data source.

In your VPC module, your remote state file key looks like this:

terraform {
  backend "s3" {
    key       = "prod/vpc/terraform.tfstate"
  }
}

to use this in another folder which builds your EKS infrastructure, you need this reference:

data "terraform_remote_state" "vpc" {
  backend = "s3"

  config = {
    bucket = "tfstates3"
    key    = "prod/vpc/terraform.tfstate" # references the VPC statefile 'key'
    region = "us-east-2"
  }
}

and now you can call the VPC modules outputs like this:

module "eks" {
  source       = "terraform-aws-modules/eks/aws"
  version      = "12.2.0"
  cluster_name = local.cluster_name
  subnets      = data.terraform_remote_state.vpc.outputs.private_subnets

  tags = {
    Environment = "prod"
  }

  vpc_id = data.terraform_remote_state.vpc.outputs.vpc_id

your VPC outputs.tf file needs to have the corresponding outputs e.g.

output "vpc_id" {
  description = "The ID of the VPC"
  value       = module.vpc.vpc_id
}

output "private_subnets" {
  description = "List of IDs of private subnets"
  value       = module.vpc.private_subnets
}

CLI: aws-cli and kubectl

kubectl needs your aws-cli to be able to find the same aws user you used to create the EKS cluster i.e. either your ‘default’ ~/.aws/credentials profile is the terraform one you used as environment variable credentials to run terraform apply OR you just have a default [default] block that has the same credentials.

This is the error you see when aws can’t find the terraform aws creds

from your terraform outout…

Error: Error running command 'aws eks --region us-west-2 update-kubeconfig --name prod-eks-cluster': exit status 255. Output: 
An error occurred (ResourceNotFoundException) when calling the DescribeCluster operation: No cluster found for name: prod-eks-cluster.

to trying to update your local kubeconfig file via aws command

$ aws eks --region us-west-2 update-kubeconfig --name prod-eks-cluster
Unable to locate credentials. You can configure credentials by running "aws configure".

to running kubectl with --kubeconfig on the kubeconfig file that was outputted by terraform to make a call to the EKS cluster

$ kubectl --kubeconfig ./kubeconfig_prod-eks-cluster get nodes
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
  For verbose messaging see aws.Config.CredentialsChainVerboseErrors

The solution was to add the [default] block to ~/.aws/credentials and voila:

kubectl --kubeconfig ./kubeconfig_prod-eks-cluster get nodes
NAME                                       STATUS   ROLES    AGE     VERSION
ip-10-0-1-130.us-east-2.compute.internal   Ready    <none>   4m56s   v1.16.13-eks-2ba888
ip-10-0-3-180.us-east-2.compute.internal   Ready    <none>   4m53s   v1.16.13-eks-2ba888

and configure kubectl

 aws eks --region us-east-2 update-kubeconfig --name prod-eks-cluster
Added new context arn:aws:eks:us-east-2:000000000000:cluster/prod-eks-cluster to /home/user/.kube/config

run it

kubectl get nodes
NAME                                       STATUS   ROLES    AGE   VERSION
ip-10-0-1-130.us-east-2.compute.internal   Ready    <none>   32m   v1.16.13-eks-2ba888
ip-10-0-3-180.us-east-2.compute.internal   Ready    <none>   32m   v1.16.13-eks-2ba888

IAM: minimum permissions

When you create a user for terraform to be able to create your EKS infrastructure, the bare minimum permissions you need to assign, as a policy, to your user is the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "autoscaling:AttachInstances",
                "autoscaling:CreateAutoScalingGroup",
                "autoscaling:CreateLaunchConfiguration",
                "autoscaling:CreateOrUpdateTags",
                "autoscaling:DeleteAutoScalingGroup",
                "autoscaling:DeleteLaunchConfiguration",
                "autoscaling:DeleteTags",
                "autoscaling:Describe*",
                "autoscaling:DetachInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:UpdateAutoScalingGroup",
                "autoscaling:SuspendProcesses",
                "ec2:AllocateAddress",
                "ec2:AssignPrivateIpAddresses",
                "ec2:Associate*",
                "ec2:AttachInternetGateway",
                "ec2:AttachNetworkInterface",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateDefaultSubnet",
                "ec2:CreateDhcpOptions",
                "ec2:CreateEgressOnlyInternetGateway",
                "ec2:CreateInternetGateway",
                "ec2:CreateNatGateway",
                "ec2:CreateNetworkInterface",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSubnet",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateVpc",
                "ec2:DeleteDhcpOptions",
                "ec2:DeleteEgressOnlyInternetGateway",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteNatGateway",
                "ec2:DeleteNetworkInterface",
                "ec2:DeleteRoute",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSubnet",
                "ec2:DeleteTags",
                "ec2:DeleteVolume",
                "ec2:DeleteVpc",
                "ec2:DeleteVpnGateway",
                "ec2:Describe*",
                "ec2:DetachInternetGateway",
                "ec2:DetachNetworkInterface",
                "ec2:DetachVolume",
                "ec2:Disassociate*",
                "ec2:ModifySubnetAttribute",
                "ec2:ModifyVpcAttribute",
                "ec2:ModifyVpcEndpoint",
                "ec2:ReleaseAddress",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:UpdateSecurityGroupRuleDescriptionsEgress",
                "ec2:UpdateSecurityGroupRuleDescriptionsIngress",
                "ec2:CreateLaunchTemplate",
                "ec2:CreateLaunchTemplateVersion",
                "ec2:DeleteLaunchTemplate",
                "ec2:DeleteLaunchTemplateVersions",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:GetLaunchTemplateData",
                "ec2:ModifyLaunchTemplate",
                "ec2:RunInstances",
                "eks:CreateCluster",
                "eks:DeleteCluster",
                "eks:DescribeCluster",
                "eks:ListClusters",
                "eks:UpdateClusterConfig",
                "eks:UpdateClusterVersion",
                "eks:DescribeUpdate",
                "eks:TagResource",
                "eks:UntagResource",
                "eks:ListTagsForResource",
                "eks:CreateFargateProfile",
                "eks:DeleteFargateProfile",
                "eks:DescribeFargateProfile",
                "eks:ListFargateProfiles",
                "eks:CreateNodegroup",
                "eks:DeleteNodegroup",
                "eks:DescribeNodegroup",
                "eks:ListNodegroups",
                "eks:UpdateNodegroupConfig",
                "eks:UpdateNodegroupVersion",
                "iam:AddRoleToInstanceProfile",
                "iam:AttachRolePolicy",
                "iam:CreateInstanceProfile",
                "iam:CreateOpenIDConnectProvider",
                "iam:CreateServiceLinkedRole",
                "iam:CreatePolicy",
                "iam:CreatePolicyVersion",
                "iam:CreateRole",
                "iam:DeleteInstanceProfile",
                "iam:DeleteOpenIDConnectProvider",
                "iam:DeletePolicy",
                "iam:DeleteRole",
                "iam:DeleteRolePolicy",
                "iam:DeleteServiceLinkedRole",
                "iam:DetachRolePolicy",
                "iam:GetInstanceProfile",
                "iam:GetOpenIDConnectProvider",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "iam:List*",
                "iam:PassRole",
                "iam:PutRolePolicy",
                "iam:RemoveRoleFromInstanceProfile",
                "iam:TagRole",
                "iam:UntagRole",
                "iam:UpdateAssumeRolePolicy",
                // Following permissions are needed if cluster_enabled_log_types is enabled
                "logs:CreateLogGroup",
                "logs:DescribeLogGroups",
                "logs:DeleteLogGroup",
                "logs:ListTagsLogGroup",
                "logs:PutRetentionPolicy",
                // Following permissions for working with secrets_encryption example
                "kms:CreateGrant",
                "kms:CreateKey",
                "kms:DescribeKey",
                "kms:GetKeyPolicy",
                "kms:GetKeyRotationStatus",
                "kms:ListResourceTags",
                "kms:ScheduleKeyDeletion"
            ],
            "Resource": "*"
        }
    ]
}

As long as your AWS user has this policy attached, it will be able to create all the resources EKS requires.

That’s it for now, I’ll update this if I come across any more pain points.

References