How to setup Service Discovery in Amazon Elastic Container Service

One of the core benefits of containers is that they enable dynamic deployments. In the old days an application server might stay hosted on the same EC2 server for weeks, months, or even years. Application servers are no longer static in the modern containerized environment. They start up to serve traffic when demand is high and stop when demand is low. They get moved around to different EC2 instances to optimize EC2 usage and save costs. The lifespan of a container may be only hours, especially in a cost sensitive organization that is taking advantage of spot instances for savings, and constantly shuffling their workloads around onto the cheapest available compute instances.

What is service discovery?

Service discovery is what enables this new dynamic paradigm. There are many different ways to do service discovery but what they all have in common is that they provide a way for a client to reliably communicate with one or more instances of a server.

Service discovery keeps track of the list of the backend servers so that requests from the client can be distributed across the backend servers. One of the traditional models for service discovery is via a load balancer.

In this architecture the load balancer is an application load balancer created and updated by Amazon Elastic Container Service (ECS). As each container starts and becomes healthy ECS updates the load balancer so that it knows the address of the new container. Whenever the client wants to talk to one of the servers it sends its request to the load balancer using its CNAME, and the load balancer sends the request to one of the servers.

This approach is highly reliable and scalable but one downside is that it does have slightly more latency compared to direct communication from client to server. For this reason some architectures use a DNS based service discovery solution such as Hashicorp Consul.

With this approach the client makes a DNS request that resolves to the direct IP address of one of the servers, so that the request travels directly from the client to the server. The DNS request is served by a Consul server which the containers have self-registered with.

This approach works well but it could be easier. We already have ECS orchestrating the container launches, and ECS is monitoring the health and availability of each container. It would be better if ECS also managed whether or not the container is registered in DNS. This is why we launched service discovery built into ECS and powered by Amazon Route 53.

This architecture is much easier to setup and operate because it does not require any application changes or external components that must be managed. In addition to launching the containers ECS also updates Route 53 so it always has an up to date list of the healthy containers. When the client does a DNS resolution it is served from Route 53 with all the same high availability and scalability that is guaranteed by Route 53.

Setting up ECS Service Discovery

Enabling service discovery for an ECS service is easy using a new section in the create service wizard:

You need to specify a namespace and a service name. Here I’ve named the namespace local and the service service-four

When I launch the service I see a new section in the details for my service:

The service discovery DNS endpoint for my service is service-four.local so whenever a client needs to talk to this service they can use this as the DNS address. The DNS will be resolved from Route 53, which has a list of the internal IP addresses of the containers that ECS launched.

It’s important to note that this DNS endpoint is in a private DNS namespace, which means that the DNS name service-four.local can only be resolved by another EC2 instance inside the VPC. I can’t resolve it from a terminal running on my home computer. So to test out the service discovery I instead use my AWS Cloud9 development environment which is running on an EC2 instance in the same VPC:

First I use the dig system utility to do a DNS lookup on the service discovery endpoint service-four.local and this gives me a list of IP addresses of the containers that ECS launched. I use the shuf utility to randomly select one of the IP addresses. Then I use curl to send a request to the randomly chosen IP address. As you can see it is easy to implement basic client side load balancing powered by DNS service discovery, in one simple line of bash.

Conclusion

Built-in service discovery in ECS is another feature that makes it easy to develop a dynamic container environment without needing to manage as many resources outside of your application. ECS and Route 53 combine to provide highly available, fully managed, and secure service discovery without the need to install, setup, or maintain third party tools.