Open in app

Sign in

Write

Sign in

Consul Connect + Envoy

Rafael Pirolla
6 min readApr 4, 2019

--

Connect + Envoy

Using Envoy with Connect - Consul by HashiCorp

This guide walks though getting started running Envoy as a Connect Proxy.

www.consul.io

This tutorial is a working example of consul connect using envoy to load balance traffic to a redis cluster. I feel it's a more real life scenario then Hashicorp's example and this one's working on Docker for Mac.

Building docker image with needed binaries

As we need to run envoy through consul the docker image should have both binaries available. Also, we will need a consul agent in the same container as redis — let's just make one image to rule them all.

We will use envoy 1.9.1 to be able to use redis filter with dynamic cluster. As a side note envoy 1.10 is not yet supported by consul (#5323) [20190403].

Create the Dockerfile:

FROM consul
FROM envoyproxy/envoy-alpine:v1.9.1
FROM redis
COPY --from=0 /bin/consul /usr/local/bin/consul
COPY --from=1 /usr/local/bin/envoy /usr/local/bin/envoyRUN set -ex; \
 apt-get update; \
 apt-get install -y --no-install-recommends dumb-init curl iproute2 gawk vim; \
 rm -rf /var/lib/apt/lists/*COPY docker-entrypoint.sh /usr/local/bin/ENTRYPOINT ["dumb-init", "docker-entrypoint.sh"]

And the docker-entrypoint.sh:

#!/bin/sh
set -eredis-server &# generate random id
UUID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1)CONSUL_DATA_DIR=/consul/data
CONSUL_CONFIG_DIR=/consul/config
CONSUL_BIND_ADDRESS=$(ip -o -4 addr list eth0 | head -n1 | awk '{print $4}' | cut -d/ -f1)mkdir -p ${CONSUL_DATA_DIR} ${CONSUL_CONFIG_DIR}# run a local agent
consul agent -bind=${CONSUL_BIND_ADDRESS} -grpc-port=8502 -data-dir="$CONSUL_DATA_DIR" -config-dir="$CONSUL_CONFIG_DIR" -join=consul-server &
echo "Ran consul"
sleep 15s# create the service definition with the envoy config
cat <<EOF >redis.json
{
  "name": "redis",
  "id": "${UUID}",
  "address": "127.0.0.1",
  "port": 6379,
  "connect": {
    "sidecar_service": {
      "port": 6000,
      "proxy": {
        "destination_service_name": "redis",
        "destination_service_id": "${UUID}",
        "local_service_address": "127.0.0.1",
        "local_service_port": 6379,
        "config": {
          "envoy_public_listener_json": "
          {
            \"@type\": \"type.googleapis.com/envoy.api.v2.Listener\",
            \"name\": \"public_listener:${CONSUL_BIND_ADDRESS}:6000\",
            \"address\": {
              \"socketAddress\": {
                \"address\": \"0.0.0.0\",
                \"portValue\": 6000
              }
            },
            \"filterChains\": [
              {
                \"filters\": [
                  {
                    \"name\": \"envoy.redis_proxy\",
                    \"config\": {
                      \"stat_prefix\": \"ingress_redis\",
                      \"cluster\": \"local_app\",
                      \"settings\": {
                        \"op_timeout\": \"5s\"
                      }
                    }
                  }
                ]
              }
            ]
          }"
        }
      }
    }
  }
}
EOF# register the service
# JSON strings cannot have line breaks so we just make it a one liner
tr -d '\n' < redis.json | curl -s -X PUT -d @- http://localhost:8500/v1/agent/service/register
echo "Registered"
sleep 10s# run consul connect
export CONSUL_HTTP_ADDR="http://localhost:8500"
export CONSUL_GRPC_ADDR="localhost:8502"# we expect the ADMIN_PORT to be defined as docker --env
consul connect envoy -sidecar-for ${UUID} -admin-bind 0.0.0.0:${ADMIN_PORT} -- -l debug

Build the image:

$ docker build -t redis-envoy .

Start a consul server

We need a bridge network to take advantage of DNS aliases:

$ docker network create --driver bridge consul-net

Start the consul server:

$ docker run --rm -d -p 8500:8500 --name consul-server --network consul-net -e CONSUL_BIND_INTERFACE=eth0 consul agent -dev -client=0.0.0.0 -ui

You can access http://localhost:8500 now.

Starting the redis envoy cluster

We will start three instances of redis but we could add and remove instances at will (given you know what you are doing).

There is a side note further below about redis cluster vs envoy redis cluster.

$ docker run --rm -d --network consul-net -p 19001:19001 --env "ADMIN_PORT=19001" --name redis01 redis-envoy
$ docker run --rm -d --network consul-net -p 19002:19002 --env "ADMIN_PORT=19002" --name redis02 redis-envoy
$ docker run --rm -d --network consul-net -p 19003:19003 --env "ADMIN_PORT=19003" --name redis03 redis-envoy

Creating a client to envoy redis cluster

We have the instances of redis available with a sidecar proxy running. How to access them through consul connect? We will create a client service definition and use redis-cli to connect to the local port that will communicate with the instances.

Let's start with a shell inside a redis-envoy container:

$ docker run -p 19000:19000 --rm -it --network consul-net --entrypoint /bin/bash --name redis-client redis-envoy

Create the dummy-client.json file:

{
  "name": "dummy-client",
  "id": "dummy-client",
  "connect": {
    "sidecar_service": {
    "port": 8282,
      "proxy": {
        "destination_service_name": "dummy-client",
        "upstreams": [
          {
            "destination_name": "redis",
            "local_bind_address": "127.0.0.1",
            "local_bind_port": 8383,
            "config": {
              "envoy_cluster_json": "
              {
                \"@type\": \"type.googleapis.com/envoy.api.v2.Cluster\",
                \"name\": \"redis\",
                \"type\": \"EDS\",
                \"lb_policy\": \"MAGLEV\",
                \"eds_cluster_config\": {
                  \"eds_config\": {
                    \"ads\": {}
                  }
                },
                \"connect_timeout\": \"5s\",
                \"outlier_detection\": {
                  \"consecutive_5xx\": 5,
                  \"base_ejection_time\": \"30s\"
                }
              }",
              "envoy_listener_json": "
              {
                \"@type\": \"type.googleapis.com/envoy.api.v2.Listener\",
                \"name\": \"redis:127.0.0.1:8383\",
                \"address\": {
                  \"socketAddress\": {
                    \"address\": \"127.0.0.1\",
                    \"portValue\": 8383
                  }
                },
                \"filterChains\": [
                  {
                    \"filters\": [
                      {
                        \"name\": \"envoy.redis_proxy\",
                        \"config\": {
                          \"stat_prefix\": \"egress_redis\",
                          \"cluster\": \"redis\",
                          \"settings\": {
                            \"op_timeout\": \"5s\"
                          }
                        }
                      }
                    ]
                  }
                ]
              }"
            }
          }
        ]
      }
    }
  }
}

Notice that I am using the maglev lb_type for the redis cluster (we could also use ring_hash) and redis filter for the upstream listener port.

Now we need:

  1. a local consul agent
  2. register the service
  3. run envoy through consul connect

Here are the commands:

CONSUL_DATA_DIR=/consul/data
CONSUL_CONFIG_DIR=/consul/config
CONSUL_BIND_ADDRESS=$(ip -o -4 addr list eth0 | head -n1 | awk '{print $4}' | cut -d/ -f1)mkdir -p ${CONSUL_DATA_DIR} ${CONSUL_CONFIG_DIR}# run a local agent
consul agent -bind=${CONSUL_BIND_ADDRESS} -grpc-port=8502 -data-dir="$CONSUL_DATA_DIR" -config-dir="$CONSUL_CONFIG_DIR" -join=consul-server &# register the service
tr -d '\n' < dummy-client.json | curl -s -X PUT -d @- http://localhost:8500/v1/agent/service/register# then finally run envoy
consul connect envoy -sidecar-for dummy-client -admin-bind 0.0.0.0:19000 &

We can now connect to the envoy redis cluster through the local upstream port that we defined in the service definition:

$ redis-cli -p 8383
127.0.0.1:8383> set bar foo
OK
127.0.0.1:8383> get foo
"bar"
127.0.0.1:8383> get bar
"foo"

You can check envoy stats at http://localhost:1900[0–3]/stats?usedonly.

Side notes

Envoy redis cluster vs redis cluster

As per envoy documentation:

Envoy can act as a Redis proxy, partitioning commands among instances in a cluster.

This means that envoy itself will choose the redis server (available in the envoy cluster definition) to access using the lb_type hash of your choosing. You can use redis cluster (i.e. redis-cli — cluster create …) or envoy redis cluster; not both.

Here is a good article about it:

http://dmitrypol.github.io/redis/2019/03/18/envoy-proxy.html

Consul JSON service files

There are two ways to register services in consul:

  1. Through config file (JSON or HCL) and sending consul process a HUP
  2. Through the agent/service API

I used option 2 in this documentation.

The configuration files can have top level "service" or "services" definition. Through the API we cannot and therefore we need to specify one service per JSON file.

There are two kind of services: connect-proxy and service. You can define them separately or you can embed the connect-proxy definition inside the service.

Here is the separate version:

redis01.json

{
  "name": "redis",
  "id": "redis01",
  "port": 7001
}

redis01-proxy.json

{
  "name": "redis01-proxy",
  "port": 6001,
  "kind": "connect-proxy",
  "proxy": {
    "destination_service_name": "redis",
    "destination_service_id": "redis01",
    "local_service_address": "127.0.0.1",
    "local_service_port": 7001
  }
}

We can merge them together like this:

redis01.json

{
  "name": "redis",
  "id": "redis01",
  "port": 7001
  "connect": {
    "sidecar_service": {
      "port": 6001,
      "proxy": {
        "destination_service_name": "redis",
        "destination_service_id": "redis01",
        "local_service_address": "127.0.0.1",
        "local_service_port": 7001
      }
    }
  }
}

Envoy Filters

If you use the last service definition you will get only basic TCP routing and metrics from envoy. Since we are using redis we can take advantage of the redis filter from envoy — we need it for the cluster, of course.

As of today consul provides “escape hatches" to inject envoy configurations. The last service definition with the redis filter in place would then become:

redis01-filter.json

{
  "name": "redis",
  "id": "redis01",
  "port": 7001,
  "connect": {
    "sidecar_service": {
      "port": 6001,
      "proxy": {
        "destination_service_name": "redis",
        "destination_service_id": "redis01",
        "local_service_address": "172.18.0.2",
        "local_service_port": 7001
      },
      "config": {
          "envoy_public_listener_json": "
          {
            \"@type\": \"type.googleapis.com/envoy.api.v2.Listener\",
            \"name\": \"public_listener:0.0.0.0:6001\",
            \"address\": {
              \"socketAddress\": {
                \"address\": \"0.0.0.0\",
                \"portValue\": 6001
              }
            },
            \"filterChains\": [
              {
                \"filters\": [
                  {
                    \"name\": \"envoy.redis_proxy\",
                    \"config\": {
                      \"stat_prefix\": \"ingress_redis\",
                      \"cluster\": \"local_app\",
                      \"settings\": {
                        \"op_timeout\": \"5s\"
                      }
                    }
                  }
                ]
              }
            ]
          }"
      }
    }
  }
}

Hyperkit

To spin up a shell inside the hyperkit VM in MacOS:

docker run -it — rm — privileged — pid=host busybox nsenter -t1 -m -u -i -n

Docker
Envoy Proxy
Consul
Connect
Service Mesh

--

--

Rafael Pirolla

Written by Rafael Pirolla

51 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams