Amazon SQS Adapter

With the Amazon SQS or Amazon Simple Queue Service adapter, a user can access Amazon Simple Queue Services. The adapter connects to the fully managed Amazon Simple Queue Service system, using AWS keys. The SQS adapter can send and fetch messages to and from the queues to which a user has access.

This document provides an overview and the functional specifications of the Amazon SQS Adapter. The Amazon Simple Queue Service (SQS) is a distributed message-queuing service supporting programmatic sending of messages via web service applications. This message-queuing service is entirely manageable and scalable and can dissociate microservices, distributed systems, and server-less applications. The function of an Amazon SQS adapter is to access queues located at various Amazon Web Services-served geographic regions to fetch messages from standard queues and also store messages in standard queues in Amazon SQS. IAM users can access SQS messages if they have the correct AWS access key and secret key, and also the queue-access permission. The SQS Adapter supports the sending and fetching of message attributes along with the message texts. SQS adapter supports standard queues having at-least-once delivery. For more information referhttps://aws.amazon.com/sqs/.

The Amazon SQS adapter does not support the predefined-schema-import option. An AWS Identity and Access Management (IAM) user can be created in AWS for accessing the SQS services. This user represents a person or application that will interact with AWS. It is not the same as the AWS account root user. For more information, refer: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html

Adapter properties and commands

This section provides the details of the adapter properties to be set to define connections and actions, and import schemas in the new web UI of HIP. The following table lists their properties, scope, whether they are required or not, and the adapter commands they map to for use in the compiled maps and map executions. “DI” scope means the property is dynamically enumerated - the adapter is invoked during action configuration to provide the set of allowed values for that property. A detailed description of the properties is provided after the table.

Table 1.
Adapter property Scope
  • CO: Connection
  • DI: Discovery
  • IC: Input card
  • OC: Output card
  • G: GET function
  • P: Put function
Required Corresponding adapter command must be used directly in GET and PUT functions
accessKey= access_key CO Yes -AK access_key
secretKey= secret_key CO Yes -SK secret_key
region= region_name CO Yes -R region_name
queue= queue_name DI/IC/OC/G/P Yes -Q queue_name
createQueue= create_queue OC/P No -CQ
attributes= attributes IC/OC/G/P No -A
timeout= timeout IC/G No -LSN wait_time_in_seconds
limit= limit IC/G No -QTY limit
logging=info|errors|verbose|off IC/OC/G/P No -T[E|V]?[+]? [file_name]
append_log=true|false IC/OC/G/P No -T[E|V]?[+]? [file_name]
log_file_name=file_name IC/OC/G/P No -T[E|V]?[+]? [file_name]

accessKey= access_key

This property specifies the access key of AWS user for the connection. The access key is used to sign programmatic requests to AWS API calls through the adapter. The access key can be used with -ACCESSKEY or -AK command.

secretKey= secret_key

This property specifies the secret key of AWS user for the connection. Like the user name and password, both the access key ID and secret access key are mandatory to authenticate SQS queue GET or PUT request. The secret key can be passed to the adapter with -SECRETKEY or -SK command.

region= region_name

The adapter uses the region name to connect to an AWS region for the IAM user account. A region can be passed to the adapter using -REGION or -R command.

queue= queue_name

This property states the SQS queue name to or from which messages need to be accessed. During action configuration, SQS Adapter provides options to select queue name among available queues in the provided region. Also, queue name property can be passed to adapter input or output cards using -QUEUE or -Q command.

createQueue= create_queue

If specified, this property indicates the SQS adapter to create a queue with the name passed in the ‘queue_name’ property, in case it doesn’t already exist. This property is optional. When the create_queue property is not specified, adapter checks if the queue already exists; if not: it throws an error. Create queue flag can be passed to adapter input or output cards using -CREATEQUEUE or -CQ command.

attributes= attributes

Attribute property specifies if the SQS message should be accessed along with the corresponding message attributes. When the attributes flag is set to true in output card, the passed data should be in predefined JSON format, else data should be in a plain text message.

{"message": "text message", "attributes": [{"name": "attribute_name", "type= "String", "value": "attribute_value"}]

When the attributes property is present in input card, SQS adapters get the data in the above JSON format; otherwise,only text message is fetched form SQS queue. Attribute flag can be passed to adapter input or output cards using -ATTRIBUTES or -A command. If it is omitted, attributes are not used. For more information on Amazon SQS message attributes, refer:https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-attributes.html

timeout= timeout

The timeout property of SQS adapter is applicable and utilized only in HIP/ITX deployments. It specifies the time duration (in seconds) to wait for before retrieving the message from the queue. The SQS adapter would listen to the queue for specified seconds. The timeout property can be used with -LSN.

limit= limit

The limit property is applicable and utilized only in HIP/ITX deployments. It specifies the total number of messages to consume. Minimum value should be 1. The value S means all available messages. If no limit is specified, adapter fetches only one message at a time from the queue. The limit can be specified using with -QTY command.

logging=info|errors|verbose|off

This property specifies the level of logging to be used for the log (trace) file produced by the adapter. The default is off. The value info means log informational and error messages, the value error means log error messages only, and the value verbose means log debug and trace level messages along with the informational and error messages.

append_log=true|false

This property specifies the flag that indicates the action to take when the specified log file already exists. When set to true, the log messages are appended to the file. When set to false: the file is truncated, and the messages are written to the empty file. The default value is true.

log_file_name=file_name

This is the name of the log file, where the log messages are written. If not specified, the default log file name m4sqs.trc is used, and the file is stored to the directory in which the executed compiled map resides.


Examples

Examples of GET map function

In the following example, assume that the adapter being used in a GET function is defined as follows:

GET("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test -A", inputdata). Since the -ATTRIBUTES/-A command is not present, the adapter only text message from the queue.

Now assume the GET map function was defined as follows: GET("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test -A", inputdata). In this case the adapter retrieves message attributes along with the text message and the output data would be like below: {"messages": "It's get example", "attributes": [{"name": "attribute1", "type"= "String", "value": "attribute example"}]}