How To Register Ecs Container Instance Not Default

Amazon Elastic Container Service ( Amazon ECS ) is a container direction service that makes it easy to run, stop, and manage Docker containers on a cluster, letting you launch and stop container-based applications with simple API calls.

To use Amazon ECS y'all need a cluster that tin can be created within a new or an existing VPC. And so, after a cluster is up and running, the task definitions and services created volition specify which Docker images encounter the clusters. The container images are stored in and pulled from container registries, which may exist inside the AWS infrastructure (Amazon ECR) or exterior.

The services and tasks launched can use ane of the following Amazon ECS launch types:

AWS Fargate : The Fargate launch type allows running containerized applications without the need for provisioning or managing the backend infrastructure (Amazon EC2 instances). Just registering the task definition volition make AWS Fargate launch the container.
Amazon EC2 : The EC2 launch type allows running containerized applications on a cluster of custom Amazon EC2 instances.

In this commodity, we will show how to run Docker images of Denodo on an Amazon ECS cluster using EC2 example, for images stored in Amazon ECR.

AWS Command Line Interface

The AWS Command Line Interface (AWS CLI) is an open up-source tool that tin can be used for administering the AWS resources and we will employ it extensively in the article, we will assume that it is installed and up to date as a prerequisite for executing the commands presented below.

In addition to using the AWS CLI to access Amazon ECS resources, you lot can utilize the Amazon ECS CLI , which provides high-level commands to simplify creating, updating, and monitoring clusters and tasks from a local development environment using Docker Etch.

Although it is also possible to launch our first cluster using the Amazon ECS first-run wizard, in this article nosotros have chosen to use the AWS CLI because ordinarily, the commands are self-explanatory and also considering it is possible to use them and test them easily in any custom AWS deployment.

Amazon ECS using EC2

The Amazon ECS EC2 launch type allows you to run your containerized applications on a cluster of Amazon EC2 instances that y'all manage.

List of Prerequisites

T his article assumes that in that location is an environment already configured with the tools required to deploy a Docker image of Denodo in AWS, in other cases, it might be needed to install or configure something else before standing with the next department. In summary, you lot need:

A valid AWS account with enough privileges to build all the AWS elements involved in the deployment of the Denodo prototype.
An IAM entity , to avoid accessing AWS direct with the credentials of our AWS business relationship. With the AWS Identity and Access Management (IAM) service, we tin accept restricted permissions and policies over the AWS resources. In This instance, nosotros need to set IAM permissions to allow for Amazon EC2 access. For more information about IAM permissions for Amazon EC2, see IAM policies for Amazon EC2 in the Amazon EC2 User Guide for Linux Instances .
To execute tasks, y'all will need a Job Role, choose an IAM part that provides permissions for containers in your chore to brand calls to AWS APIs on your behalf. For tasks that use the EC2 launch blazon, these permissions are ordinarily granted by the Amazon ECS Container Instance IAM part. For more information, see Amazon ECS Container Instance IAM Role .
A Virtual Private Cloud (Amazon VPC) with public and private subnets (a listing of subnet ids separated by commas - subnetIdList ) for our clusters and a defended security group ( securityGroupId ). We want to betrayal the Denodo Server to the internet so in this guide we will be using a public subnet and assign a public IP to the ECS chore, but you could utilise a private subnet configured with a NAT waterway with an rubberband IP accost.
Y'all must specify an Amazon Machine Image (AMI) when yous launch an EC2 instance and annotation the AMI ID . The AMI provides the information required to launch an instance. For more information, come across the department Launch an EC2 Instance with the Amazon ECS AMI explained below in this commodity.
The latest version of the AWS CLI , because although the AWS Management Console tin be used to manage Amazon ECS, the AWS CLI allows you to build scripts that tin can automate common direction tasks in Amazon ECS.
A locally accessible Docker image of Denodo . If you don't have one you lot can bank check how to create your ain Docker images for Denodo Platform Containers

AWS Permissions

An IAM entity (user, group, or function) must be set up with the privileges required by the AWS CLI for running the statements in this guide.

In this article, we have configured an IAM Part ( roleARN ) to apply ECS and Fargate/EC2 services.

This IAM part must be able to list, push, and pull images in the ECR repository, which is granted with the post-obit AWS managed policy:

AmazonEC2ContainerRegistryPowerUser .

To grant someone permission to create an Amazon ECR repository with the Amazon ECR CreateRepository API operation, nosotros must include the following action in their policy:

ecr:CreateRepository

This IAM role must be able to list, push, and pull images in the ECR repository, which is granted with the following AWS managed policy:

ecsTaskExecutionRole

In improver, although this works automatically with the cluster creation, notice that Amazon ECS uses the service-linked role named AWSServiceRoleForECS to enable Amazon ECS to call AWS APIs on your behalf.

Configure the AWS CLI

To starting time with the AWS configuration, open a new console and execute the below AWS commands that volition do the bones configuration of the AWS CLI.

Configure the AWS access keys for the AWS account by using the following command:

When entering this command, the AWS CLI prompts for the following information:

AWS Access Key ID
AWS Secret Access Key
Default region proper noun
Default output format

To read more about the AWS CLI configuration bank check the AWS CLI user guide .

At this point, you lot demand to make up one's mind if you lot volition configure the default AWS CLI profile or if you want to create a custom profile. If you lot create a custom profile some of the commands in the residual of the guide will require an additional parameter to specify the profile to use ( --profile profilename ) so for simplicity, we volition just configure the default contour in this guide. You can read more about named profiles in the AWS Command Line Interface documentation.

Use these commands to configure the IAM function in the default profile:

$ aws configure set role_arn <roleARN>

$ aws configure set source_profile default

The role_arn parameter specifies the Amazon Resource Name (ARN) of an IAM role that yous want to utilise to perform operations with the default contour. Information technology will accept this format: arn:aws:iam::<accountNumber>:part/<roleName>

The source_profile parameter identifies what profile to use to observe credentials that take permission to assume the IAM role.

Create the Container Registry

Create a new Elastic Container Registry by providing a custom proper noun for the repository:

$ aws ecr create-repository --repository-name <denodo-ecr>

Check that the denodo-ecr repository has been created successfully and obtain its repositoryUri from the output of the create-repository statement. The field repositoryUri volition exist used afterward to cosign the local Docker client.

To obtain the repositoryUri value the following command tin can be used:

$ aws ecr describe-repositories --repository-proper name <denodo-ecr> --query "repositories[].repositoryUri"

Cosign the Docker client to the new registry to push the Denodo paradigm to the repository using the repositoryUri :

$ aws ecr get-login-countersign | docker login --username AWS --countersign-stdin <repositoryUri>

Finally, tag the Denodo image and button information technology to the Elastic Container Registry

$ docker tag denodo-platform:latest <repositoryUri>

$ docker push <repositoryUri>

$ aws ecr list-images --repository-proper noun <denodo-ecr>

Create the ECS Cluster

To deploy the Denodo image on ECS it is necessary to create a cluster . Past default, the AWS accounts take a default cluster. The do good of using this default cluster is that you lot don't have to specify the --cluster <cluster-proper noun> pick in the subsequent commands. But if you do create your own cluster, then yous must specify --cluster <cluster-name> for each control that you intend to employ with that cluster.

$ aws ecs create-cluster --cluster-name <denodo-ecs-cluster>

Information technology can take some time for the cluster to exist available, but it is possible to check its status with the following command:

$ aws ecs describe-clusters --cluster <denodo-ecs-cluster>

Add together EC2 Instances to the ECS Cluster

You must annals an Amazon ECS container case in your cluster before yous can run tasks on information technology, for this the ECS Container case must be running the Amazon ECS container agent. When yous run tasks using the EC2 launch blazon , your tasks are placed on your agile container instances to register into ane of your clusters.

You can utilise the AWS CLI to launch Amazon EC2 instances. See Launching, list, and terminating Amazon EC2 instances for more than information.

F irst, you have to cull an Amazon Machine Image (AMI) . If y'all are using an Amazon ECS-optimized AMI, the agent is already installed, but if you want to apply a different operating system you tin can as well install the ECS agent. Yous tin employ the describe-images command to listing merely the Linux AMIs that meet your needs. Nosotros will use the --owners parameter to brandish public AMIs endemic by Amazon.

$ aws ec2 describe-images --owners cocky amazon

To launch an Amazon EC2 instance using the AMI you selected, use the run-instances command. We volition launch the instance into our own virtual individual cloud (VPC) with this command:

$ aws ec2 run-instances \

--image-id < amiId > \

--example-blazon < t2.small > \

--subnet-id < subnetId > \

--security-group-ids < securityGroupId > \

--iam-instance-contour Name=< ecsInstanceRole > \

--user-data file://< userdata.txt >

Image-id : AMI Id of the AMI called. We will use <ami-0e4035ae3f70c400f> for this tutorial . For more information, run across Finding a Suitable AMI in the Amazon EC2 User Guide.
Instance-type : Determines the hardware of the host computer used for your example. Nosotros have selected t2.small since it is a low-toll full general purpose instance type. For more information meet Amazon EC2 Instance Types .
Subnet-id : The ID of the subnet to launch the instance into.
Security-group-ids: The id of our Security Group .
Iam-instance-profile: EC2 case profiles let you to attach an IAM role to your EC2 instance. This allows whatever application running on the instance to access certain resources defined in the part policies, in our example our instance volition need to admission the cluster. See more information nigh this in Managing instance profiles (AWS CLI) .
User-data : When you launch an instance in Amazon EC2, you have the option of passing user data to the instance. User data beat scripts are the easiest fashion to send instructions to an instance at launch. In our case, we will utilise this parameter to send to our instance the name of the cluster where we desire it to exist available, otherwise, the ECS amanuensis will register the instance on the default ECS cluster. The steps to configure the EC2 example to apply our custom ECS cluster are these:

Firstly create a userdata.txt beat script which uses echo to re-create the cluster name variable to the configuration file ecs.config . The ecs agent expects the cluster name within this file, if this value is undefined, so the default cluster is assumed) :

#!/bin/fustigate

echo ECS_CLUSTER=< denodo-ecs-cluster > >> /etc/ecs/ecs.config

Laissez passer the script file to the --user-data parameter. With run-instances , the AWS CLI performs base64 encoding of the user data for you lot:

--user-information file://< userdata.txt >

For more information about instance user data, see User data with AWS CLI .

Initially, your instance appears in the awaiting status , but changes to the running state after a few minutes and it will be ready for employ. You can check the status of your instance using describe-instances:

$ aws ec2 draw-instances \

--filters Proper name=case-type,Values=< t2.micro > Proper name=availability-zone,Values=< myregion >

Within a few minutes of launching your container instance, the Amazon ECS agent registers the instance with your cluster. You can listing the container instances in a cluster by running the post-obit control:

$ aws ecs list-container-instances --cluster < denodo-ecs-cluster >

Output :

{

"containerInstanceArns": [

"arn:aws:ecs:< region >:aws_account_id:container-instance/ container_instance_Id "

]

}

After you have the ARN or ID of a container instance, you can use the following command to get valuable information on the case, such every bit InstanceId, remaining and registered CPU, and memory resources .

$ aws ecs describe-container-instances \

--cluster < denodo-ecs-cluster > \

--container-instances < container_instance_Id >

Register an ECS Task

When the Amazon ECS container agent registers an instance into your cluster, the container example reports its status as ACTIVE and its amanuensis connexion status every bit Truthful. This container example can accept run task requests.

This example uses a unproblematic task definition named denodo-job-definition that makes use of the Denodo container image hosted on the new repository denodo-ecr .

Also, notice that the Denodo Platform requires a valid license to run, which can exist obtained from a Denodo License Manager server or in some scenarios, like evaluation or Denodo Express, equally a standalone license file:

If yous intend to utilize a license managed by the License Manager y'all need an appropriate SolutionManager.properties configuration file that points to the License Manager server.

Alternatively, yous can utilise a Denodo standalone license, if available.

In both scenarios, the Denodo server will expect a file in the <DENODO_HOME>/conf directory, either SolutionManager.properties or denodo.lic . The files could be added to the image, but in society to make this configuration dynamic, we will load the files from the task definition.

Hence, the solution proposed is based on the ability of the task definitions to specify environment variables, which tin be used later to create files. Therefore, we can define an environment variable, for example with the content of the SolutionManager.properties in base64 (which is more than convenient than working with real text in the chore definition) and materialize that variable value with the post-obit command before starting the server:

echo $SOLUTION_MANAGER_CONF_B64 | base64 -d > /opt/denodo/conf/SolutionManager.properties;

Also, the task definition tin can exist specified in a JSON file, which improves the readability and the management of the tasks because these definitions can exist very large. Below yous will find 2 examples of very simple task definitions , one per each blazon of license scenario.

Solution Director

If there is a Denodo License Manager server running and attainable by the cluster then its connexion details tin be configured in the Solution Manager Configuration file.

As explained to a higher place, the chore definition includes the content of the SolutionManager.backdrop configuration file encoded using base64 . It is necessary to replace the SOLUTION_MANAGER_CONF_B64 placeholder in the task definition with the base64 encoding of the file. Note that you lot can get the base64 value with a command available in your OS or using an online service . So for instance, if you lot have the following SolutionManager.properties file:

# License Director Configuration

com.denodo.license.host=solution-director

com.denodo.license.port=10091

com.denodo.license.maxRetries=iii

com.denodo.license.millisecondsBetweenRetries=5000

SolutionManager.properties

The equivalent base64 encoding would exist:

IyBMaWNlbnNlIE1hbmFnZXIgQ29uZmlndXJhdGlvbgpjb20uZGVub2RvLmxpY2Vuc2Uua

G9zdD1zb2x1dGlvbi1tYW5hZ2VyCmNvbS5kZW5vZG8ubGljZW5zZS5wb3J0PTEwMDkxCm

NvbS5kZW5vZG8ubGljZW5zZS5tYXhSZXRyaWVzPTMKY29tLmRlbm9kby5saWNlbnNlLm1

pbGxpc2Vjb25kc0JldHdlZW5SZXRyaWVzPTUwMDAK

SolutionManager.properties in base64

At this signal, you are prepare to register the job definition, so take the following template and supplant the placeholders SOLUTION_MANAGER_CONF_B64, repositoryUri, and ecsTaskExecutionRoleARN to start a Denodo container in the ECS cluster:

{

"containerDefinitions": [

{

"surroundings": [

{

"name": " SOLUTION_MANAGER_CONF_B64 ",

"value": " <SOLUTION_MANAGER_CONF_B64> "

}

"entryPoint": [

"sh",

"-c",

"echo $SOLUTION_MANAGER_CONF_B64 | base64 -d > /opt/denodo/conf/SolutionManager.properties; ./denodo-container-start.sh --vdpserver"

"essential": truthful,

"image": " <repositoryUri> :latest",

"name": "denodo",

"portMappings": [

{

"containerPort": 9999,

"hostPort": 9999,

"protocol": "tcp"

{

"containerPort": 9996,

"hostPort": 9996,

"protocol": "tcp"

}

]

}

"executionRoleArn": " <ecsTaskExecutionRoleARN> ",

"family": "denodo-task-definition",

"retentivity": "1024",

"cpu": "256"

}

denodo-task-definition.json with Solution Manager

Once the task definition is saved to the JSON file denodo-task-definition.json you tin can create the chore definition with the following command:

$ aws ecs register-task-definition --cli-input-json file://denodo-job-definition.json

And so, you lot can list the task definitions created with the following control:

$ aws ecs list-task-definitions

You tin also check the details of a task definition:

$ aws ecs describe-task-definition --task-definition <taskDefinitionArn>

Standalone Licenses

To use a standalone license file, we are going to apply the aforementioned solution equally with the SolutionManager.properties file, but instead of defining an surround variable for the Solution Director configuration file, it will be created for the license file.

The following template uses the surroundings variable LICENSE_B64 to create the license file in /opt/denodo/conf/denodo.lic . Y'all just demand to get the base64 encoding for your license file and replace <LICENSE_B64> with it:

{

"containerDefinitions": [

{

"surround": [

{

"proper name": " LICENSE_B64 ",

"value": " <LICENSE_B64> "

}

"entryPoint": [

"sh",

"-c",

"echo $LICENSE_B64 | base64 -d > /opt/denodo/conf/denodo.lic; ./denodo-container-start.sh --vdpserver"

"essential": true,

"image": " <repositoryUri> :latest",

"name": "denodo",

"portMappings": [

{

"containerPort": 9999,

"hostPort": 9999,

"protocol": "tcp"

{

"containerPort": 9996,

"hostPort": 9996,

"protocol": "tcp"

}

]

}

"executionRoleArn": " <ecsTaskExecutionRoleARN> ",

"family": "denodo-task-definition",

"retention": "1024",

"cpu": "256"

}

denodo-chore-definition.json with standalone licenses

Creating the Network Load Balancer

Create the load balancer

After you have registered your chore and take launched a EC2 example that is registered to your cluster, y'all tin create the Network Load Balancer that volition exist used to distribute the requests to the Virtual DataPort server from the Cyberspace. Since the cluster may take several instances, this step is mandatory to decouple the access to the task from the instances pool. In a Fargate ECS cluster, this is non required because the task gets assigned to a public IP accost automatically.

$ aws elbv2 create-load-balancer \

--name <network-load-balancer>\

--blazon network \

--subnets <subnetIdList>

Output:

{

"LoadBalancers": [

{

"LoadBalancerArn": "<nlbARN>",

"DNSName": "<nlbDNS>",

...

}

]

}

Create the target group

The load balancer needs a target group to redirect the requests from the load balancer to the registered instances. However, nosotros do non need to register whatever instance in the target group because the ECS service that will be created later on will do that for united states, registering the EC2 instances from the cluster in the target group automatically.

$ aws elbv2 create-target-group \

--proper name <target-grouping-name> \

--protocol TCP \

--port 9999 \

--vpc-id <vpc-id>

Output:

{

"TargetGroups": [

{

"TargetGroupArn": "< targetGroupARN >",

"TargetGroupName": "<target-grouping-name>",

"Protocol": "TCP",

"Port": 9999,

...

}

]

}

Create the listener

The load balancer needs a listener in society to listen for new requests and to redirect them to the right target group.

$ aws elbv2 create-listener \

--load-balancer-arn <nlbARN> \

--protocol TCP \

--port 9999 \

--default-actions Type=forward,TargetGroupArn=<targetGroupARN>

Create an ECS Service

After registering the job and the load balancer, the next footstep is creating the service that volition run the task on our container instance. Notation that the task requires a road to the internet, as we want the Denodo server to be accessed from outside AWS.

$ aws ecs create-service \

--service-proper noun <denodo-service> \

--cluster <denodo-ecs-cluster> \

--task-definition denodo-chore-definition \

--desired-count 1 \

--launch-type "EC2" \

--load-balancers "targetGroupArn=< targetGroupARN >,containerName=denodo,containerPort=9999"

The create-service command returns a description of the task definition afterwards it completes its registration. The following command gets a list of services created in the cluster:

$ aws ecs list-services --cluster <denodo-ecs-cluster>

Y'all tin can get the clarification of the service running using the service name retrieved earlier to get more information about the task:

$ aws ecs describe-services --cluster <denodo-ecs-cluster> --services <denodo-service>

Connecting to the Denodo Platform

Finally, in order to connect to the server from a VDP customer, yous will need to ensure that the VDP customer is able to resolve this hostname to the public IP address of the load balancer, and you can do that by adding an entry in the local hosts configuration file to map that hostname with the public IP address.

You tin obtain the public IP address of the load balancer from its public DNS Name, which can be retrieved with the following AWS CLI statement:

$ aws elbv2 depict-load-balancers --names <network-load-balancer> --query "LoadBalancers[0].DNSName"

Then, add together the entry with the AWS Public IP address in the local hosts file:

# Denodo ECS service

<public-ip> <denodo-hostname>

At present you tin can open a new Virtual DataPort Assistants Tool and connect to the server using the post-obit Denodo Server URI:

//<denodo-hostname>:9999/admin

Virtual DataPort Administration Tool

References

AWS Control Line Interface

How to create your own Docker images for Denodo Platform Containers