7 thoughts on “[Exercise] Kinesis Firehose, Part 3”

  1. Ramakrishna Neti says:
    January 14, 2022 at 7:26 pm

    I don’t see any code in the agent.jason file

    1. Ramakrishna Neti says:
      January 14, 2022 at 7:31 pm

      Please disregard my earlier comment. I typed json incorrectly!

      1. Frank Kane says:
        January 14, 2022 at 7:40 pm

        Glad you figured it out 🙂

  2. Ramakrishna Neti says:
    January 15, 2022 at 12:08 am

    Hi Frank, I am now getting some errors when i did tail -f of the aws-kinesis-agent.log. May be I need to check the config file.

    2022-01-15 05:06:07.606+0000 (FileTailer[fh:PurchaseLogs:/var/log/cadabra/*.log].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [WARN] FileTailer[fh:PurchaseLogs:/var/log/cadabra/*.log]: Tailer is 71.600586 MB (75078855 bytes) behind. There are 2 file(s) newer than current file(s) being tailed.
    2022-01-15 05:06:07.611+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 107001 records parsed (8993550 bytes), and 0 records sent successfully to destinations. Uptime: 14640087ms
    2022-01-15 05:06:07.611+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [WARN] Agent: Tailing is 71.600586 MB (75078855 bytes) behind. There are 2 file(s) newer than current file(s) being tailed.
    2022-01-15 05:06:11.520+0000 (sender-931) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher [ERROR] AsyncPublisher[fh:PurchaseLogs:/var/log/cadabra/*.log]:RecordBuffer(id=23,records=500,bytes=40287) Retriable send error (com.amazonaws.SdkClientException: Unable to execute HTTP request: firehose.us-east-2: Name or service not known). Will retry.

  3. Ramakrishna Neti says:
    January 15, 2022 at 12:42 am

    This is how my json config file looks like:
    GNU nano 2.9.8 agent.json

    {
    “cloudwatch.emitMetrics”: true,
    “kinesis.endpoint”: “”,
    “firehose.endpoint”: “firehose.us-east-2.amazonaws.com”,

    “awsAccessKeyId”: “”,
    “awsSecretAccessKey”: “”,

    “flows”: [
    {
    “filePattern”: “/var/log/cadabra/*.log”,
    “deliveryStream”: “PurchaseLogs”
    }
    ]
    }

    1. Frank Kane says:
      January 15, 2022 at 6:41 am

      The error message is saying: “firehose.us-east-2: Name or service not known” But,
      firehose.us-east-2.amazonaws.com is a valid endpoint for Firehose according to https://docs.aws.amazon.com/general/latest/gr/fh.html. You’re sure that’s your region though, right?

      My guess would be that you edited the agent.json file but didn’t restart the Kinesis agent to pick up the changes. Perhaps originally you had “firehose.us-east-2” instead of “firehose.us-east-2.amazonaws.com” in there when you started it up?

  4. Ramakrishna Neti says:
    January 15, 2022 at 2:21 pm

    Yes, I got the region from the ARN (arn:aws:firehose:us-east-2:141656874697:deliverystream/PurchaseLogs) of the delivery stream and you are right in that I did not restart the Kinesis agent after making edits to the agent.json to inculde “firehose.us-east-2.amazonaws.com”.

    Here’s the final tail -f of the aws-kinesis-agent.log:
    2022-01-15 19:13:50.839+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 500000 records parsed (46225698 bytes), and 500000 records sent successfully to destinations. Uptime: 390083ms

    I am also able to see the PurchaseLog files in the S3 bucket folders!

    Thank you Frank!

Leave a Reply