AWS Integration Services
Below is a concise, organized reference to AWS services commonly used to integrate components, systems, and data across applications and organizations. Each service entry includes a short description and common use cases.
Amazon API Gateway
- Description: Fully managed service to create, publish, secure, and monitor REST, HTTP, and WebSocket APIs.
- Use cases: Fronting microservices/lambdas, BFF (backend-for-frontend), real-time WebSocket APIs.
- When to choose: You need auth/traffic control, request transformation, throttling, or a managed API lifecycle.
AWS AppSync
- Description: Managed GraphQL service with real-time subscriptions and built-in resolvers for DynamoDB, Lambda, HTTP.
- Use cases: GraphQL APIs for mobile/web apps, offline sync, merging multiple backends into one schema.
- When to choose: Client-driven queries, complex aggregation from multiple sources, real-time updates.
Amazon EventBridge (CloudWatch Events)
- Description: Serverless event bus for application, SaaS, and AWS service events with filtering and routing.
- Use cases: Event-driven architectures, cross-account event routing, SaaS integrations (Salesforce, Datadog).
- When to choose: You need reliable, schema-aware event routing and decoupled microservices.
Amazon SNS (Simple Notification Service)
- Description: Pub/sub messaging for fan-out, mobile push, SMS, email, and HTTP endpoints.
- Use cases: Broadcast notifications, fan-out patterns (SNS -> multiple SQS endpoints), mobile alerts.
- When to choose: You need push-style notifications and multi-protocol delivery.
Amazon SQS (Simple Queue Service)
- Description: Fully managed message queuing (standard and FIFO) for decoupling producers and consumers.
- Use cases: Asynchronous job processing, buffering spikes, decoupling microservices.
- When to choose: You need durable queue semantics, retry/backoff, and DLQs (dead-letter queues).
Amazon MQ
- Description: Managed message broker (ActiveMQ, RabbitMQ) for JMS/AMQP/MQTT/STOMP compatibility.
- Use cases: Lift-and-shift of legacy apps using broker protocols, interoperable enterprise messaging.
- When to choose: You require broker features (transactions, JMS) or protocol compatibility.
Amazon Kinesis (Data Streams / Firehose / Data Analytics)
- Description: Real-time streaming data ingestion and processing platform.
- Use cases: Clickstream ingestion, real-time analytics, ETL into S3/Redshift, stream processing.
- When to choose: High-throughput streaming, ordered event processing, complex streaming pipelines.
AWS Lambda (as integration runtime)
- Description: Serverless compute often used as event handlers, transformers, or connectors.
- Use cases: Event consumers (EventBridge, S3, API Gateway), lightweight transforms, glue logic.
- When to choose: Short-lived integration logic, custom routing, or transformations.
AWS Step Functions
- Description: Serverless workflow orchestration (Standard and Express) for composing services with stateful workflows.
- Use cases: Long-running orchestrations, human approval flows, error-handling and retries across steps.
- When to choose: Orchestration of multi-step processes with observability and retry control.
AWS Simple Workflow Service (SWF)
- Description: Older workflow service for coordination of distributed components (largely legacy).
- Use cases: Existing SWF workloads; prefer Step Functions for new designs.
Amazon AppFlow
- Description: Managed integration service for SaaS applications and AWS (two-way flows).
- Use cases: Sync Salesforce, Slack, Google Analytics to S3, Redshift, or Salesforce-to-S3 workflows.
- When to choose: Low-code SaaS-to-AWS integrations with secure data handling.
AWS DataSync
- Description: Managed data transfer for on-premises to AWS (NFS/SMB) and between AWS storage services.
- Use cases: Bulk data migration, recurring syncs between datacenter and S3/EFS.
- When to choose: High-performance, automated file transfers with minimal custom code.
AWS Transfer Family
- Description: Managed SFTP/FTPS/FTP endpoint backed by S3/EFS.
- Use cases: Lift-and-shift file transfer workflows, partner file exchange.
- When to choose: You need managed traditional file transfer protocols integrated with S3/EFS.
AWS Glue
- Description: Serverless ETL/data integration service with crawlers, schema discovery, and job orchestration.
- Use cases: Data cataloging, ETL pipelines, prepare data for analytics.
- When to choose: Batch/stream ETL into data lake or data warehouse with cataloging.
Database & Migration Integration
- AWS DMS (Database Migration Service): Continuous replication/migration between heterogeneous DBs.
- AWS SCT (Schema Conversion Tool): Convert schemas between DB engines.
- Use cases: Database migrations, heterogeneous replication.
AWS App Mesh
- Description: Service mesh for microservices networking with observability and traffic control.
- Use cases: Fine-grained traffic routing, mTLS, telemetry across microservices (ECS/EKS/EC2).
- When to choose: You need service-to-service visibility and traffic control at L7.
Common Integration Patterns and Examples
- Pub/Sub: SNS (publisher) -> SQS / Lambda (subscribers) for fan-out and durable processing.
- Event Bus: Application emits events -> EventBridge routes to multiple targets (Lambda, Step Functions, SQS, Kinesis).
- Queue-based decoupling: Microservice A pushes jobs to SQS; consumer scales independently.
- Stream processing: Kinesis Data Streams -> Kinesis Data Analytics / Lambda -> S3/Redshift.
- API-based integration: API Gateway -> Lambda -> backend services (DynamoDB, RDS).
- Orchestration: Step Functions coordinates multiple Lambdas, ECS tasks, and human approvals.
- SaaS sync: AppFlow to move data from SaaS apps into S3/Redshift for analytics.
Example: EventBridge rule triggering a Lambda
- Event producer publishes a custom event to EventBridge.
- EventBridge rule filters events (by source/detail-type) and invokes a Lambda function for processing.
Example: Reliable async processing
- Producer -> SNS Topic -> multiple SQS queues (fan-out). Each consumer polls its SQS and uses DLQ for failures.
