Create AWS IoT Analytics Setup

To facilitate the creation of an IoT Analytics setup AWS provides a quick create wizard which creates a pipeline, channel and data store just with a some clicks.

To create your AWS IoT Analytics resources

  1. Navigate to the AWS IoT Analytics console
  2. Under Get started with AWS IoT Analytics, under Resources prefix, enter sitewise_workshop.
  3. Do not enter a topic under MQTT topic - optional.
  4. Choose Create resources.
  5. Under Get started with AWS IoT Analytics you should see that the following resources are created:
    • sitewise_workshop_channel
    • sitewise_workshop_datastore
    • sitewise_workshop_pipeline
    • sitewise_workshop_dataset
  • In the left pane under AWS IoT Analytics browse Channels, Pipelines, Data stores and Data sets to verify that the IoT Analytics resources have been created.

Create a Lambda

Create the Lambda function to flatten the JSON objects in the AWS Cloud9 IDE.

In the left pane:

  1. Under FILE SYSTEM
  2. Expand SiteWiseWorkshop
  3. Expand IoTAFlattenSiteWiseData
  4. You should see now lambda_function.py
  5. Right click on lambda_function.py
  6. Open
  7. Replace the existing code with the following:
import json
import logging
import sys
import time

# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
streamHandler = logging.StreamHandler(stream=sys.stdout)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
streamHandler.setFormatter(formatter)
logger.addHandler(streamHandler)


def lambda_handler(event, context):
    logger.info("event: {}".format(event))
    print(json.dumps(event, indent=2))
    transformed = []

    for e in event:
        logger.info("e: {}".format(json.dumps(e, indent=2)))
        swtype=e['type']
        asset_id = e['payload']['assetId']
        property_id = e['payload']['propertyId']
        logger.info("type: {} asset_id: {} property_id: {}".format(swtype, asset_id, property_id))
        row = {"type": swtype, "asset_id": asset_id, "property_id": property_id}
        for v in e['payload']['values']:
            logger.info("v: {}".format(v))

            time_in_seconds = int(time.time())
            if 'timestamp' in v and 'timeInSeconds' in v['timestamp']:
                logger.debug("timeInSeconds in payload")
                time_in_seconds = v['timestamp']['timeInSeconds']

            value = ""
            valuetype = ""
            if 'doubleValue' in v['value']:
                logger.debug("doubleValue in payload")
                value = v['value']['doubleValue']
                valuetype= "double"
            if 'integerValue' in v['value']:
                logger.debug("integerValue in payload")
                value = v['value']['integerValue']
                valuetype= "integer"
            if 'booleanValue' in v['value']:
                logger.debug("booleanValue in payload")
                value = v['value']['booleanValue']
                valuetype= "boolean"
            if 'stringValue' in v['value']:
                logger.debug("stringValue in payload")
                value = v['value']['stringValue']
                valuetype= "string"

            quality = ""
            if 'quality' in v:
                logger.debug("quality in payload")
                quality = v['quality']

            row['timestamp'] = time_in_seconds
            row['quality'] = quality
            row['value'] = value
            row['valuetype'] = valuetype
            logger.debug("row: {}".format(row))
            transformed.append(row)

    logger.info("transformed: {}\n".format(json.dumps(transformed, indent=2)))

    return transformed
  1. File (in the menu bar)
  2. Save
  3. In the left pane, choose the aws symbol. You should see AWS: Explorer and the AWS Region you are in. In case you are stuck in Connecting, choose the three bars right from AWS: Explorer, then choose Connect to AWS, and then choose profile:default.
  4. Expand the region.
  5. Right click Lambda, and then choose Deploy SAM Application.
  6. Under Step 1 of 4, choose IoTAFlattenSiteWiseData/template.yaml.
  7. Under Step 2 of 4, choose the bucket which name starts with sitewiseworkshop-miscresources.
  8. Under Step 4 of 4, enter IoTFlattenSiteWiseData, and then hit Enter.
  9. An additional window/tab will be opened with the name AWS Toolkit.
  10. You will see messages about the deployment progress of your Lambda function.
  11. After a successful deployment you should see messages similar to:
...
2021-08-09 14:30:16 [INFO]:
Successfully created/updated stack - IoTFlattenSiteWiseData in eu-west-1


2021-08-09 14:30:16 [INFO]:

2021-08-09 14:30:16 [INFO]: Successfully deployed SAM Application to CloudFormation Stack: IoTFlattenSiteWiseData

Verify that the Lambda function has been created

Use the AWS Explorer to verify that the Lambda function has been created.

Expand Lambda.

You should find a Lambda function with a name which looks similar to IoTFlattenSiteWiseData-IoTAFlattenSiteWiseData-UNIQUE_STRING.

If you do not find your newly created function, refresh the AWS: Explorer. To refresh the explorer, choose the three bars right from AWS: Explorer, then choose Refresh Explorer.

Add permissions

Add a permission to the Lambda function to allow IoT Analytics to invoke the function. The permission is added through the command line. To add permission you need to find the exact name of your Lambda function.

In a Cloud9 terminal:

aws lambda list-functions --query 'Functions[?starts_with(FunctionName, `IoTFlattenSiteWiseData-IoTAFlattenSiteWiseData`) == `true`].FunctionName' --output text

The output of this command should look similar to IoTFlattenSiteWiseData-IoTAFlattenSiteWiseData-UNIQUE_STRING

In the next command replace [YOUR_LAMBDA_FUNCTION_NAME] with the name of your Lambda function.

aws lambda add-permission --statement-id $(date '+%Y%m%d%H%M%S') \
    --principal iotanalytics.amazonaws.com --action lambda:InvokeFunction \
    --function-name [YOUR_LAMBDA_FUNCTION_NAME]

The output of the command should look similar to:

{
    "Statement": "{\"Sid\":\"20210809144349\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"iotanalytics.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:AWS_REGION:AWS_ACCOUNT_ID:function:IoTFlattenSiteWiseData-IoTAFlattenSiteWiseData-UNIQUE_STRING\"}"
}

Add the Lambda to pipeline activities

With pipeline activities you can modify your incoming data before they are persisted into a data store.

  1. Navigate to the AWS IoT Analytics console
  2. Choose Pipelines, and then choose sitewise_workshop_pipeline.
  3. Choose Activities, and under Activities, choose Edit.
  4. Choose Next.
  5. Under Add a pipeline activity, choose Transform message with Lambda function, and then choose Add.
  6. Under Lambda function, choose the Lambda funcion whose name starts with IoTFlattenSiteWiseData-IoTAFlattenSiteWiseData.
  7. Under Batch size, enter 10.
  8. Choose Next, and then choose Update.

To verify that the pipeline activity has been added navigate to Pipelines, and then choose sitewise_workshop_pipeline. Under Activities you should find Execute AWS Lambda function.

Create an IoT topic rule

Create an IoT topic rule to transfer messages from AWS IoT SiteWise to AWS IoT Analytics.

As you have already created a rule in AWS IoT Core not every single step is explained to create the rule but only the important ones.

  • For Rule name, choose SiteWiseToIoTARule.
  • For Rule query statement, choose SELECT * FROM '$aws/sitewise/asset-models/+/assets/+/properties/+'.
  • For Action, choose Send a message to IoT Analytics.
  • For Configure action, choose Manually select IoT Analytics Channel and role.
  • For Channel name, choose sitewise_workshop_channel.
  • For Role, choose SiteWiseWorkshop-MiscResources-UNIQUE_STRING-IoTServiceRole-UNIQUE_STRING.

Watch data in AWS IoT Analytics

Wait some minutes for IoT Analytics to receive data.

To verify that data are ingested into IoT Analytics you can use the channel monitor

  1. Navigate to the AWS IoT Analytics console
  2. Choose Channels, and then choose sitewise_workshop_channel.
  3. Under Monitoring, choose 1h.
  4. The graph should show incoming messages. You might need to refresh the IncomingMessages graph.

Verify data format

To verify that your data are stored as expected create a data set and look at the preview

  1. Navigate to the AWS IoT Analytics console
  2. Choose Datasets, and then choose sitewise_workshop_dataset.
  3. Choose Run now.
  4. After the dataset has been created you can find it under Content, and then Dataset content.
  5. Choose the linke under Name to retrieve a preview of your dataset content. The name looks like uuid-formatted and should look similar to 48eec233-72bd-47cf-a2ed-25aeaf3f9b02.
  6. The result preview should look similar to:

iota_result_preview.png

Troubleshooting

In case you are not getting data in IoT Analytics there could be several causes like an error in the IoT topic rule or an error in the Lambda function that flattens the data.

You should first look for errors in Amazon CloudWatch.

Go to the Amazon CloudWatch console

  1. Logs
  2. Log groups
  • You find the logs for the IoT service under AWSIoTLogs or AWSIoTLogsV2
  • Lambda logs are found under /aws/lambda/[YOUR_LAMBDA_FUNCTION_NAME]