Code
Introduction
The ClearBlade IoT Platform utilizes a microservices model powered by a Code Engine
. The code engine allows for JavaScript (es5) and has several built-in integrations to interact with your IoT assets. Adhering to microservice practices: a new execution context is sandboxed and created upon invocation. Once the process is complete, it is destroyed.
There are four types of Code assets:
Micro Service: A short-lived service that is expected to be completed within a fixed time (or is killed otherwise); default execution timeout is 60 seconds.
Stream Service: A service configured to continuously handle requests with an infinite execution timeout.
Configuration: A library that contains a key-value object to be used as constants.
Library: A custom javascript library that can be imported into one or more code services
For a tutorial on how to set up code services, click here
Purpose
Microservices are a software development technique that allows for a high degree of modularity, scale, and parallelization - all of which are essential for effectively building and running full-scale IoT Applications.
Libraries
Libraries are reusable sets of code that can be imported by one or more code services.
Native Libraries
Native Libraries are available in every System and have an underlying implementation in Golang. Each library is optimized for computational performance.
These libraries can be imported using Requires
Library | Desc | Docs |
---|---|---|
clearblade | Interacts with Collections, Messaging, Users, Edges, Devices | View docs |
log | Appends messages to a log | View docs |
http | Makes outbound HTTP/S requests | View docs |
geo | Geospatial Computation Library | View docs |
crypto | Creates cryptographical hashes | View docs |
file_writer | Write files to host filesystem | View docs |
jsutil | JS Utility Library | View docs |
mailer | Send email directly from host machine | View docs |
net_cb | Net library | View docs |
oauth2_lib | OAuth Utility Library | View docs |
Custom Libraries
A developer can create their custom library with custom logic.
Services
A Code Service
, or Service
, is a short-lived microservice that is expected to complete in a finite time capped by the execution timeout field in its settings (it is killed after the timeout). The default is 60 seconds.
Lifecycle
Invocation
A Service
can be invoked in any of the three ways:
- A REST request to the API Endpoint
- Trigger Event
- Timer Event
Our SDKs can be used as a REST Client to accomplish the first REST option
Context
Each Code Service service has a name, and that name corresponds to an autogenerated function name.
For example, a code service named analyze
will have an autogenerated function like so:
function analyze(req, resp) {}
This function will have two parameters, similar to Node’s Express library:
req
req
is an object containing the following contextual values:
key | desc | type | example |
---|---|---|---|
userEmail | Email of user who invoked the service | string | “user@clearblade.com” |
params | JSON with parameters, structure is based upon type of Invocation | Object | {“key”:“value”} |
systemKey | System Key for system in which code service is running | string | e095a19d0b94e48dee9bff87e5fd01” |
systemSecret | System Secret for system in which code service is running | string | “E095A19D0B9ACD9EFA86AEEDFCC001” |
userToken | Invoking user’s OAuth token | string | “U1hNa7o4bEXpiE4aGhb-79MQv4…” |
isLogging | Whether logging is enabled | boolean | true |
userid | internal uid associated with invoking user | string | “fc8cfcea0a96ebebc7f5d4edd414” |
resp
resp
is an object used to return values and exit the Code Service
. Following methods are available in the resp
object:
- resp.success
- resp.error
- resp.set
- resp.status
- resp.send
resp.success
resp.success = function (object | string){}
function analyze(req, resp) {
var message = "Everything went well!";
resp.success(output); // Service exits successfully with "Everything went well!"
}
resp.error
resp.error = function (object | string ){}
function analyze(req, resp) {
var message = "It broke!";
resp.error(error); // Service exits with failure with "It broke!"
}
resp.set This method has multiple signatures:
- resp.set = function(string, string){}
resp.set(keyString, valueString)
: adds or updates a key in the header to the HTTP response from a service. Example: resp.set("Content-Type", "text/html")
- resp.set = function(object){}
-
resp.set({})
: will set the response headers to those in the map. It replaces the header object so it removes any headers added withresp.set(k,v)
. Example:
resp.set({
"Content-Type": "application/json",
"ClearBlade-UserToken: req.userToken"
})
resp.status
resp.status = function(int){}
resp.status(int)
: changes the HTTP response code from a service. Example: resp.status(400)
resp.send
resp.send = function(object|string){}
resp.send(value)
: exit the service and send the value back as the body of the response. value
will be stringified if not already a string. value
will be sent as-is and not wrapped in our typical json response object.
Example: resp.send({valueA: "ABC-123-DEF", valueB: 123})
Exit
The Code Service
will terminate its execution with resp.success
, resp.error
, or resp.send
.
Note: By default, if no exit method is called and execution reaches the end, resp.success
will be called to exit.
Logs
Logs record and store the history of service responses. Each service can store up to 25 of the latest logs with a max size of 100KB each.
Requires
A Code Service can import a Library using the Requires
feature. In the Service settings, you can select one or more libraries to import, and that code is accessible from within the Code Service.
The same library can be used for multiple code services
Debugging
There is no native support for debugging line-by-line. Recommended to use log
method extensively, like so:
function ServiceA(req, resp) {
log("Something happened");
resp.success("All done");
}
Events
Events are actions or occurrences that happen in the ClearBlade System, response to them through execution of the associated Code Service. ClearBlade supports two types of events:
Timers
Timers provide a mechanism where a ClearBlade code service can be scheduled to execute at certain time intervals. Timers are similar to Unix cron jobs, with a similar scheduling mechanism.
Some example Timer configurations:
- Run every five minutes starting now forever
- Run 26 times every two weeks starting Friday, December 25th, 2018
- Run once at midnight
Triggers
Triggers are a mechanism that ties an action, called a Trigger Source
, to the invocation of a code service. The trigger source and payload information can be accessed in the code service through the req.params
object. When creating a trigger, add log(req)
to the Code Editor
to log the payload information.
"params": {
"topic": "ClearBladeTopic",
"body": "214.312",
"userId": "8e89e2af0fc9cbeadfd6afb501",
"trigger": "Messaging::Publish"
}
Event Source
The platform publishes for every event, regardless of whether the event is created programmatically in a service or created externally in the console.
$trigger
and $timer
are custom ClearBlade topics.
It is advisable to use a stream service instead of micro-service when working with $timer
and $trigger
topics that fire often to prevent overloading the platform.
For more information on publishing event topics, click here
Asset | Category | Action | Topic |
---|---|---|---|
Messaging | Publish | Publish | $trigger/messaging/publish |
Messaging | Subscriptions | Upon Subscribe | $trigger/messaging/subscribe |
Messaging | Subscriptions | Upon Unsubscribe | $trigger/messaging/unsubscribe |
Messaging | Connection | User Connected | $trigger/messaging/user/connected |
Messaging | Connection | User Disconnected | $trigger/messaging/user/disconnected |
Messaging | Connection | Device Connected | $trigger/messaging/device/connected |
Messaging | Connection | Device Disconnected | $trigger/messaging/device/disconnected |
Data | Table | Created | $trigger/collection/created |
Data | Table | Updated | $trigger/collection/updated |
Data | Table | Deleted | $trigger/collection/deleted |
Data | Item | Created | $trigger/data/created |
Data | Item | Updated | $trigger/data/updated |
Data | Item | Deleted | $trigger/data/deleted |
Data | Item | Upserted | $trigger/data/upserted |
User | n/a | Created | $trigger/user/created |
User | n/a | Updated | $trigger/user/updated |
User | n/a | Deleted | $trigger/user/deleted |
Device | n/a | Created | $trigger/device/created |
Device | n/a | Updated | $trigger/device/updated |
Device | n/a | Deleted | $trigger/device/deleted |
Edge/Platform | Platform | Platform Started | $trigger/startstopdisconnect/platformstarted |
Edge/Platform | Platform | Platform Connected on Edge | $trigger/startstopdisconnect/platformconnectedonedge |
Edge/Platform | Platform | Platform Disconnected on Edge | $trigger/startstopdisconnect/platformdisconnectedonedge |
Edge/Platform | Edge | Edge Started | $trigger/startstopdisconnect/edgestarted |
Edge/Platform | Edge | Edge Connected on Platform | $trigger/startstopdisconnect/edgeconnectedonplatform |
Edge/Platform | Edge | Edge Disconnected on Platform | $trigger/startstopdisconnect/edgedisconnectedonplatform |
Timer | n/a | Create Timer | $timer/{timerName} |
Examples of Incoming Object
Ensure the relevant permissions for code service are set before using triggers and timers.
Following is a list of categories grouped by action types, showing structures for incoming object:
Messaging (Publish/Subscribe/Unsubscribe)
:
When there is a Publish/Subscribe/Unsubscribe made to the ClearBlade’s MQTT broker.
// Template
{
"params": {
"topic": "<message_topic>",
"body": "<message_body>",
"userId": "<user_id>",
"trigger": "Messaging::<action_type>"
}
}
// Example
{
"params": {
"topic": "ClearBladeTopic",
"body": "example message",
"userId": "8e89e2af0fc9cbeadfd6afb501",
"trigger": "Messaging::Publish"
}
}
Messaging (UserConnected/UserDisconnected)
:
When a user Connects/Disconnects to the ClearBlade’s MQTT broker.
// Template
{
"params": {
"email": "<user_email>",
"trigger": "Messaging::<action_type>"
}
}
// Example
{
"params": {
"email": "example@clearblade.com",
"trigger": "Messaging::MQTTUserDisconnected"
}
}
Messaging (DeviceConnected/DeviceDisconnected)
:
When a device Connects/Disconnects to the ClearBlade’s MQTT broker.
// Template
{
"params": {
"deviceName": "<device_name>",
"trigger": "Messaging::<action_type>"
}
}
// Example
{
"params": {
"deviceName": "exampleDevice",
"trigger": "Messaging::MQTTDeviceDisconnected"
}
}
Data-Table (CollectionCreated/CollectionUpdated/CollectionDeleted)
:
// Template
{
"params": {
"collectionId": "<collection_id>",
"collectionName": "<collection_name>",
"trigger": "Data::<action_type>"
}
}
// Example
{
"params": {
"collectionId": "ceafb4cf0bbaf5f4ddfbf1b48730",
"collectionName": "ExampleCollection",
"trigger": "Data::CollectionCreated"
}
}
Data (ItemCreated)
:
// Template
{
"params": {
"items": [
{
"item_id": "<item_id>"
}
],
"collectionId": "<collection_id>",
"collectionName": "<collection_name>",
"trigger": "Data::ItemCreated"
}
}
//Example
{
"params": {
"items": [
{
"item_id": "083196bf-8059-46c0-8280-38234aa73fc1"
}
],
"collectionId": "ceafb4cf0bbaf5f4ddfbf1b48730",
"collectionName": "ExampleCollection",
"trigger": "Data::ItemCreated"
}
}
Data (ItemUpdated/ItemDeleted)
:
// Template
{
"params": {
"collectionId": "<collection_id>",
"collectionName": "<collection_name>",
"trigger": "Data::<action_type>",
"items": [
// First row
// This includes all of the data from the row
{
"<column1_name>": <column1_data>,
"item_id": "<item_id1>",
"<column2_name>": <column2_data>
}
// A second row
{
"<column1_name>": <column1_data>,
"item_id": "<item_id1>",
"<column2_name>": <column2_data>
}
]
}
}
// Example
{
"params": {
"collectionId": "ceafb4cf0bbaf5f4ddfbf1b48730",
"collectionName": "ExampleCollection",
"trigger": "Data::ItemUpdated",
"items": [
{
"hi": 6,
"item_id": "14b12adf-4c32-4740-8461-546daa660393",
"low": 3
},
{
"hi": 7,
"item_id": "3700fa91-ef49-41e8-9fe7-64e9ee6aa3b2",
"low": 5
}
]
}
}
User (UserCreated)
// Template
{
"params": {
"user": {
"creation_date": <date_time>,
"email": "<user_email>",
"user_id": "<user_id>"
},
"trigger": "User::UserCreated"
}
}
//Example
{
"params": {
"user": {
"creation_date": 2019-06-20T19:43:10Z,
"email": "example@clearblade.com",
"user_id": "9c87dfd00beefbd8a0f3cc82f0c001"
},
"trigger": "User::UserCreated"
}
}
User (UserUpdated/UserDeleted)
// Template
{
"params": {
"query": {
"FILTERS": [
[
{
"EQ": {
"user_id": "<user_id>"
}
}
]
],
"PAGENUM": <page_number>,
"PAGESIZE": <page_size>,
"SELECTCOLUMNS": <columns_selected>,
"SORT": []
},
// Includes all of the data from the row
"user": {
"email": "<user_email>"
<column1_name>: <row_data>
<column2_name>: <row_data>
"user_id": "<user_id>"
},
"trigger": "User::<action_type>"
}
}
// Example
{
"params": {
"query": {
"FILTERS": [
[
{
"EQ": {
"user_id": "9c87dfd00beefbd8a0f3cc82f0c001"
}
}
]
],
"PAGENUM": 0,
"PAGESIZE": 0,
"SELECTCOLUMNS": null,
"SORT": []
},
"user": {
"email": "example@clearblade.com"
"name": "Example"
"age": 22
"user_id": "9c87dfd00beefbd8a0f3cc82f0c001"
},
"trigger": "User::UserUpdated"
}
}
Device (DeviceCreated/DeviceDeleted)
// Template
{
"params": {
"trigger": "Device::<action_type>",
"device": {
"allow_certificate_auth": "<boolean>",
"allow_key_auth": "<boolean>",
"certificate": "<certificate>",
"created_date": "<date_time>",
"description": "<device_description>",
"device_key": "<device_key> :: <device_name>",
"enabled": "<boolean>",
"last_active_date": "<date_time>",
"<columnX_name>": "<columnX_data>",
"<columnY_name>": "<columnY_data>",
"system_key": "<system_key>",
"type": "<device_type>"
},
"deviceName": "<device_name>"
}
}
//Example
{
"params": {
"trigger": "Device::DeviceCreated",
"device": {
"allow_certificate_auth": true,
"allow_key_auth": true,
"certificate": "",
"created_date": 2019-06-20T19:43:10Z,
"description": "",
"device_key": "ccafb4cf0bd0dcbcadaccaf9ebba01 :: ExampleDevice",
"enabled": true,
"last_active_date": 2019-06-20T19:43:10Z,
"name": "ExampleDevice",
"state": "active",
"system_key": "ccafb4cf0bd0dcbcadaccaf9ebba01",
"type": "Sensor"
},
"deviceName": "ExampleDevice"
}
}
Device (DeviceUpdated)
// Template
{
"params": {
"changes": {
"description": "<device_description>"
},
"deviceName": "<device_name>",
"trigger": "Device::DeviceUpdated"
}
}
// Example
{
"params": {
"changes": {
"description": "device description"
},
"deviceName": "example_device",
"trigger": "Device::DeviceUpdated"
}
}
Edge/Platform
// Template
{
"params": {
"edgeName": "edge_name",
"trigger": "StartConnectDisconnect::<action_type>"
}
}
// Example
{
"params": {
"edgeName": "example",
"trigger": "StartConnectDisconnect::<Edge Started>"
}
}
Webhooks
A webhook is a mechanism that allows you to execute a code service by targeting a public endpoint. Any HTTP method, such as GET, POST, PUT, DELETE, can be used and will result in the same action, executing the service. The URL we provide can be used by third parties to push real-time data to ClearBlade’s server. A code service can be invoked by multiple webhooks and are sync-able. See tutorial.
Preloaded Services
A micro-service can be preloaded if we turn on the preloaded toggle for it. Preloading allows it to be significantly efficient. Especially when it is being invoked frequently (through a trigger or a webhook). Similar to a regular stream service, you can set the concurrency and auto-balance to true. This way, the requests to invoke a service will be load-balanced across multiple handlers.
Internally these services are getting loaded into a stream service, which has the ability to ingest http
requests and returns responses back to the client. Since the stream service is always running, it solves the problem of continuously creating and destroying execution engines. This makes it faster and memory efficient.
Important note: Since your microservice code is part of the internal stream service, we recommend that you use caution if you have globals in your code. Globals will store their state across multiple invocations of the service, which might give undesirable results.
Authorization
These requests are to be added when executing a webhook.
Auth Method | Request | |
---|---|---|
clearblade_auth |
ClearBlade-Usertoken: <USER_TOKEN> |
See Example |
http_basic_auth |
Authorization: Basic <USER_TOKEN> |
See Example |
payload_auth (GET) |
Query String: token=<USER_TOKEN> |
See Example |
payload_auth (POST) |
Request Body: “token”: <USER_TOKEN> |
See Example |
no_auth |
No Authorization Required. The “Anonymous” role should have Run permission checked for the service. This allows for the no auth webhook to work. |
See Example |
Authorization Examples
ClearBlade Auth
function ExampleWebhookInvoker(req,resp){
// These are parameters passed into the code service
var params = req.params
var http = Requests();
var options = {
uri:"https://platform.clearblade.com/api/v/4/webhook/execute/f6edffd60bdc95a5c80f79d0a/ExampleWebhook",
headers:{
"ClearBlade-UserToken":"Id19kSWr2vkB4CUcxa5lm6Akzh1jXr6Fw3sxW-ynaaV67EJRLmrHvkqUD8vnYu52XGErDNRpSNJsD5w=="
},
body:{
"data":"hello"
}
}
http.post(options, function(err, response){
if(err){
resp.error("Failed to call the webhook");
}
resp.success("Success"+ response);
})
}
HTTP Basic Auth
function ExampleWebhookInvoker(req,resp){
// These are parameters passed into the code service
var params = req.params
var http = Requests();
var options = {
uri:"https://platform.clearblade.com/api/v/4/webhook/execute/84c3e6d10bbae683c7ace4dfd701/Webhook_2_HTTP_Auth",
headers:{
"Authorization":"Basic Id19kSWr2vkB4CUc5lm6Akzh1jXr6Fw3sxW-ynaaV67EJRLmrHvkqUD8vnYu52XGErPEjDNRpSNJsD5w=="
},
body:{
"data":"Authenticated using Http Auth"
}
}
http.post(options, function(err, response){
if(err){
resp.error("Failed to call the webhook");
}
resp.success("Success"+ response);
});
}
Payload Auth GET
function ExampleWebhookInvoker(req,resp){
// These are parameters passed into the code service
var params = req.params
var http = Requests();
var options = {
uri:"https://platform.clearblade.com/api/v/4/webhook/execute/f6edffd60bdc95a5c19ff79d0a/Webhook_3_Payload_Auth",
qs:{
"token":"Id19kSWr2vkB4CUcxa5lm6Akzh1jXr6Fw3sxW-ynaaV67ELmrHvkqUD8vnYu52XGErPEjDNRpSNJsD5w==",
"data":"Authenticated using Get method"
}
}
//Using get method
http.get(options, function(err, response){
if(err){
resp.error("Failed to call the webhook");
}
resp.success("Success"+ response);
});
}
Payload Auth POST
function ExampleWebhookInvoker(req,resp){
// These are parameters passed into the code service
var params = req.params
var http = Requests();
var options = {
uri:"https://platform.clearblade.com/api/v/4/webhook/execute/84c3e6d10b8ebae3c7ace4dfd701/Webhook_3_Payload_Auth",
body:{
"token":"Id19kSWr2vkB4CUcxa5lm6Akzh1jXr6Fw3sxW-ynaaV67EJRLmrHvkqUDnYu52XGErPEjDNRpSNJsD5w==",
"data":"Authenticated using Http Payload"
}
}
http.post(options, function(err, response){
if(err){
resp.error("Failed to call the webhook");
}
resp.success("Success"+ response);
});
}
No Auth
function ExampleWebhookInvoker(req,resp){
// These are parameters passed into the code service
var params = req.params
var http = Requests();
var options = {
uri:"https://platform.clearblade.com/api/v/4/webhook/execute/84c3e6d10b8ebae3c7ace4dfd701/Webhook_4_No_Auth"
}
http.post(options, function(err, response){
if(err){
resp.error("Failed to call the webhook");
}
resp.success("Success"+ response);
});
}
Advanced
Execution Timeout
Code Services are usually used as microservices and are expected to have a finite lifecycle with a default of 60 seconds. However, this execution timeout can be extended to make the services run longer. When execution timeout is set to infinity(“Never” in the console), it’s called a Stream Service. This means the code service will run indefinitely. It allows for the following high-performance use cases:
- Exponentially faster rate for any operation which is otherwise performed in an event-triggered micro-service.
- Users can also design stream services to perform bulk operations on collections.
- Data generation
Best Practices:
For Stream Services, set Execution Timeout to
Never
For Micro Service, set Execution Timeout to its default of
Infinite
Concurrency
Concurrency allows the developer to cap the maximum number of instances of code services that can run on each of the ClearBlade nodes. Any more than this number will kill the oldest code service instance.
Best Practices:
For Stream Services, set Concurrency to
Other
, and then set the number of instances you need. Recommended is 4.- For Micro Service, set Concurrency to its default of
Infinite
, to optimize performance
- For Micro Service, set Concurrency to its default of
Run as
This feature is useful when the user who invokes the service needs different permissions than what is required for the service to run.
If a role has permission to execute a code service, then a user with that role can start the execution of the service. If the run as
parameter is set, it runs as the user that is stated in the parameter, regardless of the user that invokes it. If the user set in the run as
parameter doesn’t have permissions to execute the service (and the permissions to execute all the operations performed within the code service), then the service will fail. If the user invoking the service does not have permission to execute the code service, then the service will fail as well.
For example
An end-user (eg: customer1@customer.com) can request for his account information through an accountsInfo
service. The user the service executes as needs permissions to the “accounts” collection. The end-user is not supposed to have permissions to the “accounts” collection. This is where the “Run as” feature allows us to have a separation between the end-user and the user the service is “run as”. For the above scenario, the end-user (here customer1@customer.com) will need to execute permissions for the code service. The run as user (here accountInfo_service_user@clearblade.com) will need to execute permissions to the code service along with other permissions for assets accessed in the code service (in this case, “read” permissions to the “accounts” collection).
Auto Restart
This property is valid when the service is a Stream Service. When the box is checked, it restarts service automatically if it stops for any reason, including successes.
Best Practice: Do not check the box until fully testing the service. There could be undesired deadlocks if there are errors in the code.
Auto Balance
This property is valid when the service is a Stream Service. The platform automatically starts a service on each available node with the number of concurrency instances. It kills and restarts services when saved with a new code.
If auto_balance = true
:
req.params
are not allowed- A normal code service must be developed first to confirm that it is functioning properly before developing it as a stream service. Do not check the auto-balance box until the code service works.
Failed Runs
If a Code Service exits with resp.error, or throws an uncaught error, it will be stored in Failed Runs. A developer can then fix any issue and retry that failed run with the same input parameters.
Standard Javascript Functions (es5.1)
Method | Usage |
---|---|
setTimeout(),setInterval() | Available natively in version 9.3.0 or later |
atob(), btoa() | buffer-node-js library |
promises | q-promise-library/Available natively in version 9.3.0 or later |