Set up a New AWS Ingest Feed

The steps required to create a new ingest feed are partially on LogScale and partially on AWS as ID's created during the creation of an ingest feed must be used during the creation of policies on AWS.

The configuration has been divided into three phases to simplify the process. Phase 1 and 3 is performed on LogScale and phase 2 must be performed on AWS.

LogScale Ingest Feeds Steps Phase 1

The first step is to create the new ingest feed configuration:

  1. Go to Repositories and views page and select a relevant repository in which you want to ingest the data.

  2. Click Settings, under Ingest on the side menu click Ingest feeds.

    Screenshot of the LogScale Ingest feeds settings page showing the administrative interface where users begin the AWS ingest feed configuration process. The interface displays a tabular list of any existing configured ingest feeds with columns showing feed names, preprocessing methods used, assigned parsers, last activity timestamps, and current operational status. The page includes a prominent '+ New ingest feed' button at the top that users click to initiate the creation of a new AWS S3 ingest feed.

    Figure 108. Ingest Feeds


  3. On the Ingest feeds page, click + New ingest feed and specify the following:

    • Name

      Enter a name for the feed (required).

    • Description

      Enter a meaningful description for the feed.

    • Compression

      Select a compression type from the list. Auto (default) uses decompression based on MIME type and file extension.

    • Preprocessing

      Define how the data is processed prior to ingestion, it can be split by AWS records or by new line. The way data should be preprocessed depends on the log source.

    • Parser

      Select the parser to apply to ingest data (required). The parser can be any parser that is on repository, predefined, from a package, or custom. See Parsing Data for more information.

    Screenshot of the LogScale 'New ingest feed' configuration dialog showing the first step of the AWS S3 ingest feed setup process. The form displays required and optional input fields for creating a new feed configuration, including: a Name field for providing a unique identifier for the feed (marked as required), a Description field for adding contextual information about the feed's purpose, a Compression dropdown menu defaulting to 'Auto' for specifying how data files should be decompressed, a Preprocessing selection for defining how incoming data should be processed (with options for splitting by AWS records or by new line), and a Parser selection field (marked as required) for choosing which parser to apply to the ingested data. The dialog includes 'Next' and 'Cancel' buttons at the bottom to either proceed with the configuration or abort the process.

    Figure 109. Ingest Feeds - Add


  4. Click Next.

  5. In the Add credentials dialog box, click the copy icon to copy your IAM identity as you will need this identity along with your External ID to configure AWS.

    Screenshot of the LogScale AWS credentials configuration dialog showing the first phase of setting up an AWS S3 ingest feed. The interface displays the 'Add credentials' step where LogScale generates an IAM identity and External ID that must be copied and used when configuring permissions in AWS. The dialog includes copy buttons next to these values to facilitate transferring them to the AWS console. This is a critical step in the AWS ingest feed setup workflow, as these credentials must be used to create proper trust relationships between LogScale and AWS before proceeding with the configuration.

    Figure 110. Ingest Feeds - Add Part 1


AWS Steps Phase 2

The following is an example configuration with minimal permission to access the data to ingest. You will need:

  • To create a role in AWS.

  • To make sure that STS is activated on your AWS account.

  • The ARN of the role you create for the ingest feed in LogScale.

  • To configure an SQS queue or an SNS topic with event notifications.

    The following SQS permissions are required:

    ini
    sqs:ReceiveMessage
sqs:GetQueueAttributes
sqs:DeleteMessage
sqs:ChangeMessageVisibility

  1. Create a standard SQS queue in AWS with the following configurations:

    • Name

      Specify a name for your queue.

    • Message retention period

      We recommend setting this to 14 days (max).

      Screenshot of the AWS SQS queue configuration interface showing the creation form for a standard queue to be used with LogScale ingest feeds. The interface displays configuration fields including the queue name input and message retention period setting, which is recommended to be set to the maximum of 14 days.

      Figure 111. Ingest Feeds - SQS Queue


  2. Create a dead-letter queue (recommended). See Amazon documentation for more information.

    Screenshot of the AWS SQS console interface showing the dead-letter queue configuration section within the queue creation workflow. The interface displays configuration options for setting up a dead-letter queue, which is a recommended component when creating an SQS queue for LogScale's AWS S3 ingest feeds. The panel includes fields for specifying which queue to use as the dead-letter queue, along with settings for defining the maximum receives threshold that determines when messages are moved to the dead-letter queue after failed processing attempts. This configuration step is part of creating a robust error-handling mechanism for the S3 ingest process, ensuring that problematic messages don't block the main processing queue and can be separately analyzed for troubleshooting purposes. The AWS interface shows radio button options for enabling this feature along with dropdown selectors for choosing existing queues or creating new ones specifically for this dead-letter functionality.

    Figure 112. Dead-letter Queue


  3. You need to grant access to the S3 Bucket or SNS topic, click Advanced under Access policy and copy and paste this snippet and:

    Screenshot of the AWS SQS queue access policy configuration interface showing where administrators configure permission settings for LogScale's S3 ingest functionality. The image displays the JSON policy editor within the AWS Management Console where users must paste the custom access policy that grants S3 bucket permission to send notifications to the SQS queue. The policy editor shows a text area where administrators enter the required JSON policy document with the appropriate Principal, Action, Resource, and Condition elements to establish the trust relationship between the S3 bucket and SQS queue. This configuration step is critical in the LogScale ingest feed setup workflow as it establishes the permissions needed for event notifications to flow from S3 storage to the SQS queue that LogScale monitors. The Advanced section of the access policy configuration is expanded, showing where users must replace the 'MY_ARN' placeholder with their actual S3 bucket or SNS Topic ARN to complete the security configuration.

    Figure 113. Ingest Feeds - Policy


    • Replace the MY_ARN with the ARN of your S3 bucket or SNS Topic.

    json
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sqs:SendMessage",
            "Resource": "arn:aws:sqs:*",
            "Condition": {
                "ArnEquals": {
                    "aws:SourceArn": "MY_ARN"
                }
            }
        }
    ]
}

  4. Click Create the queue.

  5. Configure the SQS queue or SNS Topic to create an event notifications, go to Create event notification on you SQS Bucket and configure it so that All objects create events.

    Screenshot of the AWS S3 bucket properties interface showing the 'Create event notification' configuration panel. This image displays the initial step in setting up event notifications that will trigger when new objects are created in the S3 bucket. The interface shows the AWS Management Console where administrators configure which S3 bucket events will be sent to the SQS queue that LogScale monitors. The panel includes options for creating a new event notification rule that will inform LogScale's ingest system when new log data is available for processing. This configuration step is essential in the LogScale AWS ingest feed setup workflow, as it establishes the automated mechanism that notifies LogScale when new data appears in the bucket. Without properly configured event notifications, LogScale would be unable to detect and process new log files as they arrive in the S3 storage. The interface highlights where users should select 'All objects create events' to ensure LogScale can ingest all new data added to the bucket.

    Figure 114. Ingest Feeds - Events


    Screenshot of the AWS S3 event notification configuration interface showing the event type selection screen. This image displays the second step in the event notification setup process where administrators specify which types of S3 object events will trigger notifications to the SQS queue. The interface presents checkboxes for different event categories, with 'All object create events' option highlighted as the recommended selection for LogScale's ingest feed configuration. This panel allows administrators to precisely define which actions on S3 objects (such as uploads, completions of multipart uploads, or copy operations) should generate notifications that LogScale will monitor. The configuration ensures that only relevant storage events trigger the notification pipeline, optimizing the efficiency of the LogScale ingest process. This middle step in the three-part event notification setup is critical for establishing the correct event filtering that determines when LogScale will be notified about new data available for ingestion from the S3 bucket.

    Figure 115. Ingest Feeds - Events 2


    Screenshot of the AWS S3 event notification destination configuration interface. This image displays the critical final step in the three-part event notification setup process where administrators specify where the selected S3 events will be sent. The interface shows the destination selection panel where users must configure the SQS queue as the target for all S3 bucket event notifications. The panel includes options for selecting the notification destination type (SQS queue) and a field for specifying the exact queue ARN or name that LogScale will monitor for new data alerts. This configuration creates the essential connection between the S3 bucket that stores the log data and the SQS queue that LogScale polls to detect when new data is available for ingestion. After completing this configuration and clicking 'Save changes', the notification pathway is established, enabling automated data discovery and ingestion into LogScale. This final configuration step completes the event-based notification system that powers LogScale's AWS S3 ingest feed functionality.

    Figure 116. Ingest Feeds - Events 3


  6. Set the SQS Queue or SNS Topic as the destination, then click Save changes. Notifications will now appear in the bucket.

    Screenshot of the AWS S3 bucket properties interface displaying the successfully configured event notifications. This image shows the completed event notification setup as it appears in the AWS Management Console after saving the configuration. The interface presents a summary view of the bucket's event notification rules, listing the newly created notification that links S3 object creation events to the SQS queue destination. This verification screen confirms that the notification pathway has been properly established between the S3 bucket containing log data and the SQS queue that LogScale will monitor. The configured notification is now active and will trigger automatically whenever new objects are added to the bucket, enabling LogScale to detect and ingest new log data. This completed configuration serves as a visual confirmation that the event notification system is properly set up and ready to support LogScale's AWS S3 ingest feed functionality. Administrators should verify that the notification appears as expected in this interface before proceeding with the final LogScale configuration steps.

    Figure 117. Ingest Feeds - data


  7. If you are using an SNS Topic you need to create a SNS topic subscription that points to the SQS bucket you created previously with an access policy that allow SourceArn on your S3 bucket.

  8. The next step is to create an IAM role, go to Identity and Access Management (IAM), which allows you to read data. Select Roles.

  9. Create a role and add a Custom trust policy using this snippet, where you must:

    json
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLogScaleAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "LOGSCALE_AWS_IDENTITY_ARN"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "LOGSCALE_EXTERNAL_ID"
                }
            }
        }
    ]
}
    Screenshot of the AWS IAM custom trust policy configuration interface showing where administrators create the security role for LogScale S3 ingest. The image displays the JSON policy editor within the AWS IAM Console where users must paste and customize the trust policy that allows LogScale to assume the role. The policy editor shows a text area containing the JSON trust policy with placeholders for 'LOGSCALE_AWS_IDENTITY_ARN' and 'LOGSCALE_EXTERNAL_ID' that must be replaced with the specific values copied from the LogScale ingest feed configuration. This critical security configuration establishes the trust relationship that authorizes LogScale to assume this role when accessing AWS resources. The interface includes the custom trust policy option selected, indicating that administrators must manually enter the policy rather than using pre-defined templates. This security configuration is essential for creating the proper authentication mechanism that allows LogScale to securely access both the S3 bucket containing log data and the SQS queue that provides notifications, while maintaining strict access control through the external ID condition.

    Figure 118. Ingest Feeds - Custom trust policy


  10. Click Next.

  11. Click Next again, you do not add permissions as we will add an inline policy later.

  12. Specify a name for the role and click Create role.

  13. Find and select the role, click the Add permissions drop down and select Create inline policy

    Screenshot of the AWS IAM console showing the role summary page with the 'Add permissions' dropdown menu expanded. This image displays the interface after successfully creating a new IAM role for LogScale, highlighting where administrators must now add specific permissions to the role. The dropdown menu shows various permission options with 'Create inline policy' selected, which is the required next step in the configuration process. This interface represents a critical transition point in the setup workflow - moving from role creation with the trust policy (who can assume the role) to defining permission policies (what the role can do). The console shows the newly created role's summary information with the permissions section that must be populated with the specific S3 and SQS access permissions required for LogScale's ingest functionality. This step enables administrators to add the precise resource-level permissions that will allow LogScale to read from the S3 bucket and receive messages from the SQS queue while maintaining the principle of least privilege by granting only the minimum necessary access rights.

    Figure 119. Ingest Feeds - Role


    Screenshot of the AWS IAM inline policy creation interface showing the policy editor with multiple tabs. This image displays the initial view of the policy editor that appears after selecting 'Create inline policy' from the Add permissions dropdown. The interface presents different methods for defining the policy, with tabs for Visual Editor, JSON editor, and Templates visible at the top of the panel. The editor shows the default starting view where administrators can choose how they want to construct the permission policy for the LogScale IAM role. This interface is where administrators must select the JSON tab (as indicated in the next instruction) to manually enter the specific permission policy that grants LogScale access to read objects from the S3 bucket and interact with the SQS queue. The policy editor provides the foundation for defining the precise resource-level permissions with the correct actions (s3:GetObject, sqs:ReceiveMessage, etc.) and resources (specific ARNs) that LogScale requires to function correctly while maintaining security best practices through the principle of least privilege.

    Figure 120. Ingest Feeds - Inline Policy


  14. Click the JSON tab and copy and paste the snippet below which allows your read data and;

    • Replace S3_BUCKET_ARN -- with the ARN of your S3 bucket this must be followed by /* to access the objects in the bucket.

    • Replace SQS_QUEUE_ARN with the ARN of your SQS queue

    json
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReceiveS3Notifications",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteMessage",
                "sqs:ChangeMessageVisibility"
            ],
            "Effect": "Allow",
            "Resource": "SQS_QUEUE_ARN"
        },
        {
            "Sid": "ReadS3Objects",
            "Action": "s3:GetObject",
            "Effect": "Allow",
            "Resource": "S3_BUCKET_ARN/*"
        }
    ]
}

  15. Click Next. Specify a name for the policy and click Create policy. Copy and save the ARN of the role and the URL of the SQS bucket.

LogScale Ingest Feeds Steps Phase 3

  1. Go back to the ingest feed you started to create in part 1.

    Screenshot of the LogScale AWS credentials configuration dialog where users enter AWS integration details to complete ingest feed setup. The form displays fields for Role ARN and SQS Queue URL that must be populated with values created in AWS during the previous configuration step. The interface also shows the IAM Identity and External ID values that were used to create the AWS role, and includes a Test button to validate the connection before saving the configuration.

    Figure 121. Ingest Feeds - Add Part 2


  2. Copy and paste the ARN of the role you created in AWS.

  3. Copy and paste the URL of your SQS queue from AWS. The AWS Region is normally auto-filled but if empty, select a region based on your AWS Console.

  4. Optionally, click test to see if the configuration is valid. Click Save.