Direct integration with third-party API using AWS Step Functions
Being able to quickly integrate external APIs can facilitate a lot the first iteration (or MVP) of your product/idea. Third party APIs can also being involved in some batch/asyncronous process and even there, having a fast and reliable way to integrate them avoids a lot of worries.
AWS Step Functions offers the new State “Call third-party API” that fits perfectly our needs.
The integration is quite easy but hides some tricky parts that I will explain. As usual, you can find the whole project on my Github
I will stick with my beloved AWS SAM to build with.
Express vs Standard (AWS Step Functions)
Express Step Functions differs from the Standard one for some key points:
- Express support the synchronous invocation (you can expect some dynamic and complex response from it), Standard only asynchronous.
- Express can run for maximum 5 minutes, Standard form an entire year.
- Express need explicit logging definition, Standard gets it built-in.
- Express is priced per number of executions, their duration and memory consumption. Standard is priced per State Transition.
Complete comparation between Standard vs Express is available at this Official AWS link.
Since in this project I want to use API Gateway to trigger my Step Function and demonstrate that it can provide the response that gets from the third-party API, I will use the Express Workflow.
Third party API: Twilio
As I told you in another article you can directly use AWS SNS to send out SMS but i need an external API for this project so i choose Twilio because of their Free Trial offer, their easy dashboard and their detailed documentation.
Step Function Express Definition
First I want to take a look to the Step Function Express definition:
IntegratorStateMachine:
Type: AWS::Serverless::StateMachine
Properties:
Type: EXPRESS
DefinitionUri: statemachine/statemachine.asl.json
DefinitionSubstitutions:
ConnectionArn: !GetAtt TwilioConnection.Arn
Here you can see that there is the explicit definition of which type is this Step Function.
You can also see how you can define some parameters (DefinitionSubstitutions) to “inject” into the Step Function workflow that is detailed in a different file (statemachine.asl.json). The substitution is key-value and you can use the AWS Intrinsic Functions to manipulate or obtain the parameter value.
{
"Comment": "A description of my state machine",
"StartAt": "Call third-party API",
"States": {
"Call third-party API": {
"Type": "Task",
"Resource": "arn:aws:states:::http:invoke",
"Parameters": {
"Method": "POST",
"Transform": {
"RequestBodyEncoding": "URL_ENCODED"
},
"Headers": {
"content-type": "application/x-www-form-urlencoded"
},
"ApiEndpoint": "https://api.twilio.com/2010-04-01/Accounts/YOURTWILIOSID/Messages.json",
"Authentication": {
"ConnectionArn": "${ConnectionArn}"
},
"RequestBody.$": "$.sendsms"
},
"Retry": [
{
"ErrorEquals": [
"States.ALL"
],
"BackoffRate": 2,
"IntervalSeconds": 1,
"MaxAttempts": 3,
"JitterStrategy": "FULL"
}
],
"End": true
}
}
}See that the correct syntax for the parameter is ${parameterName}. Take a look also to “RequestBody.$”: “$.sendsms”, is the main takeaway here: how to pass the API Gateway request to the Step function!
Policies:
- Statement:
- Sid: InvokeHTTPEndpointPolicy
Effect: Allow
Action:
- states:InvokeHttpEndpoint
Resource: '*'
Condition:
StringEquals:
states:HTTPMethod: POST
StringLike:
states:HTTPEndpoint: "https://api.twilio.com/*"
- Statement:
- Sid: RetrieveConnectionCredentialsPolicy
Effect: Allow
Action: events:RetrieveConnectionCredentials
Resource: '*'
- Statement:
- Sid: AccessSecretsManagerPolicy
Effect: Allow
Action:
- secretsmanager:GetSecretValue
- secretsmanager:DescribeSecret
Resource: "arn:aws:secretsmanager:*:*:secret:events!connection/*"
- Statement:
- Effect: Allow
Action:
- "cloudwatch:*"
- "logs:*"
Resource: "*"Now let’s analyze the Policies attached to the resource:
InvokeHTTPEndpointPolicy: this policy allows the step function to “go outside” the VPC (Virtual Private Cloud) to call the external API.
RetrieveConnectionCredentialsPolicy: to manage the external API integration you will need to create an EventBridge Connection. This policy allows the Step Function to “use” it.
AccessSecretsManagerPolicy: the EventBridge Connection stores the connection details into AWS Secret Manager so you need the access to it.
The last one is the explicit allow to log on CloudWatch, with Express Step Function you need to fill in also this with the Logging property:
Logging:
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt StateMachineLogGroup.Arn
IncludeExecutionData: false
Level: 'ALL'You will also need a separate resource as LogGroup:
StateMachineLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Join [ "/", [ "stepfunctions", IntegratorStateMachine]]EventBridge API Connection
TwilioConnection:
Type: AWS::Events::Connection
Properties:
AuthorizationType: BASIC
Description: 'Connection with an API key'
AuthParameters:
BasicAuthParameters:
Username: "{{resolve:ssm:/twilio_sid:1}}"
Password: "{{resolve:ssm:/twilio_secret:1}}"This is the SAM definition of the Connection: I am using the AWS Parameter Store to store my Twilio Free Trial connection credentials:
API Gateway definition
SmsServiceAPIGateway:
Type: AWS::Serverless::Api
Properties:
StageName: dev
Name: sms-service
DefinitionBody:
'Fn::Transform':
Name: 'AWS::Include'
Parameters:
Location: 'api.yaml'This is the SAM syntax to have an AWS API Gateway defined in a different Openapi spec. file (api.yaml):
openapi: "3.0.1"
info:
title: "SmsServiceAPIGateway"
version: "2022-06-20T05:39:09Z"
servers:
variables:
basePath:
default: "/dev"
paths:
/sms:
post:
responses:
"200":
description: "200 response"
content:
application/json:
schema:
$ref: "#/components/schemas/Empty"
x-amazon-apigateway-integration:
type: "aws"
credentials:
Fn::Sub: "${RestApiRole.Arn}"
httpMethod: "POST"
uri:
"arn:aws:apigateway:${AWS::Region}:states:action/StartSyncExecution"
responses:
default:
statusCode: "200"
responseTemplates:
application/json: "#set ($parsedPayload = $util.parseJson($input.json('$.output')))\n\
$parsedPayload"
requestTemplates:
application/json:
Fn::Sub: "#set($data = $util.escapeJavaScript($input.json('$')))\n\
\ {\n \"input\": \"$data\",\n \"stateMachineArn\": \"\
${IntegratorStateMachine.Arn}\"\n }"
passthroughBehavior: "when_no_templates"
components:
schemas:
Empty:
title: "Empty Schema"
type: "object"This Openapi definition is simple but the integration between AWS API Gateway and the Step Function needs the “tricky” part that involves responseTemplates and requestTemplates.
I would expect that this part was plug&play passing by default the POST request to the step function without the need of further definition.
Let’s try it
Deploy it with AWS SAM as explained in this article.
I will use Postman to call the SmsServiceAPIGateway endpoint:
Postman keeps waiting for the response from the API Gateway and this last gets it from the synchronous execution of the Express Step Functions:
On my phone I just got my SMS from Twilio:
Now you know how to integrate without even a single line of code a third-party API.
Other details are available in the official AWS guide (for example, with Twilio I needed the BASIC auth but others kind of auth are supported as well).
Another thing to point out is that I used API Gateway to trigger the Step Functions but this integration fits easily an EDA (Event Driver Architecture) using EventBridge to trigger the Step Function: tailor the correct way on your own project.
Hope this was interesting for you, see you next time! 😎
Lorenzo
