This is a quick blog on how we use AWS services from inside of an Istio Service Mesh. Why does it matter that you’re inside the mesh? Because the service mesh wants to manage all the traffic in/out of your application. This means it needs to be able to inspect the traffic and parse it if it is HTTP. Nothing too fancy here, just writing it down in case it can save you a few keystrokes.
Our example is for programs written in Go.
Step 1: Define an Egress Rule
You need to make an egress to allow the application to talk to the AWS service at all. Here’s an example egress rule to allow dynamo:
Step 2: Delegate encryption to the sidecar
This part is the trick we were missing. If you want to get maximum service mesh benefits, you need to pass unencrypted traffic to the sidecar. The sidecar will inspect it, apply policy and encrypt it before egressing to the AWS service (in our case Dynamo).
Don’t worry, your traffic is not going out on any real wires unencrypted. Only the loopback wire from your app container to the sidecar. In Kubernetes, this is its own network namespace so even other containers on the same system cannot see it unencrypted.
AwsConfig() is the core - you need to make a new aws.Session with these options.
The first option,
WithDisableSSL(true), tells the AWS libraries to not use
HTTPS and instead just speak plain HTTP. This is very bad if you are not in
the mesh. But, since we are in the mesh, we’re only going to speak plain HTTP
over to the sidecar, which will convert HTTP into HTTPS and send it out over
the wire. In Kubernetes, the sidecar is in an isolated network namespace with
your app pod, so there’s no chance for other pods or processes to snoop this
When you set the first option, the library will try to talk to
http://dynamodb.<region>.us-west-2.amazonaws.com on port 80 (hey, you asked it
to disable SSL). But that’s not what we want - we want to act like we’re
talking to 443 so that the right egress rule gets invoked and the sidecar
encrypts traffic. That’s what
istioEgressEPResolver is for.
We do it this way for a little bit of belts-and-suspenders safety - we really want to avoid ever accidentally speaking HTTP to dynamo. Here are the various failure scenarios:
Our service is in Istio, and the user properly configured InMesh=true: everything works and is HTTPS via the sidecar.
Our service is not in Istio, and the user properly configured InMesh=false: everything works and is HTTPS via the AWS go library.
Our service is not in Istio, but oops! the user set InMesh=true: the initial request goes out to dynamo on port 443 as plain HTTP. Dynamo rejects it, so we know it’s broken before sending a bunch of data via plain HTTP.
Our service is in Istio, but oops! the user set InMesh=false: the sidecar rejects the traffic as it is already-encrypted HTTPS that it can’t make any sense of.
OK, now you’ve got an
aws.Session instance ready to go. Pass it to your
favorite AWS service interface and go:
p.s. What’s up with
removeTracingHeaders()? Check out
Neeraj’s post. While you’re at it, you can add
just a few more lines and get great end-to-end distributed tracing.