Skip to content

Toro Cloud Dev Center

Toro Cloud Login

Logging data to Tracker

Martini exposes a small API that you can use to search and store data to the Tracker search index. Using the API, you can customize the way service calls look in the Tracker interface, as well as pick and choose which data will be indexed by Tracker for searching later.

Logging HTTP calls

Martini includes a file called trackers.xml. This configuration file is located under the conf/ directory, and is used by Tracker to determine which requests and responses are tracked. The distribution contains the following content by default:

1
2
3
4
5
6
7
8
<?xml version="1.0" encoding="UTF-8" ?>
<trackers
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://www.torocloud.com/toro-martini/schema/trackers"
        xsi:schemaLocation="http://www.torocloud.com/toro-martini/schema/trackers trackers.xsd">

    <tracker pattern="/api/**" scope="none"/>
</trackers>

The <tracker> declaration above means that all requests that start with /api/ are matched, but skips tracking because the scope is set to none. To see this in action, try removing the scope attribute:

1
    <tracker pattern="/api/**"/>

This configuration will enable tracking for all requests that start with /api/.

The <tracker> element also contains the following attributes, for further customization:

Attribute Default Description
pattern (required) The URL pattern, or the exact URL to match
method all A comma-separated value of the request methods (e.g., GET, POST) to match
scope all Indicates whether to log the request, response, all, or none
internal-id (generated) The identifier used for the document that tracks a single request-response lifecycle
sender-id Client Addr The value to be used as the Tracker document senderId
receiver-id Path Info The value to be used as the Tracker document receiverId
document-type HTTP The value to be used as the Tracker document type
More example Tracker configurations 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8" ?>
<trackers>
    <tracker pattern="/exact-match"/>
    <tracker pattern="/pattern/**"/>
    <tracker pattern="/by-post-method" method="POST"/>
    <tracker pattern="/by-post-or-put-method" method="POST,PUT"/>

    <tracker pattern="/scope/request-only" scope="request"/>
    <tracker pattern="/scope/response-only" scope="response"/>
    <tracker pattern="/scope/all" scope="all"/>
    <tracker pattern="/scope/none" scope="none"/>

    <tracker pattern="/custom-document"
            internal-id="harcoded-internal-id"
            sender-id="8888"
            receiver-id="09158888888"
            document-type="Globe"/>
</trackers>

How granular should I get?

While tracking generally enables you to have invaluable traceability, there are certain use cases where it might not be applicable. This may be one of, but not limited to:

Note that when configuring <tracker> elements, the order of declaration is important. The elements should be added, with the most specific at the top, and the most general at the bottom. For example:

1
2
3
4
5
6
<?xml version="1.0" encoding="UTF-8" ?>
<trackers>

    <tracker pattern="/api/**"/>
    <tracker pattern="/api/pets"/>
</trackers>

In this case, the second <tracker>, is never matched because the first <tracker> takes priority. The correct configuration should be:

1
2
3
4
5
6
7
8
<?xml version="1.0" encoding="UTF-8" ?>
<trackers>

    <tracker pattern="/api/pets"/>

    <!-- Catch-all example -->
    <tracker pattern="/api/**"/>
</trackers>

Restrictions

Out of the box, only the following base URLs are tracked:

Thus, Tracker patterns can only start with any of the above.

Logging from HTTP client services

Logging HTTP requests that invoke the io.toro.martini.HttpFunctions.http(...) service is done by configuring the trackable input model. This model can be found in the Input side of the Mapper view when the invoke step is selected. This model also has a field called trackingScope which changes the scope of the tracking for the HTTP request.

HTTP invoke step showing the `trackable` input property

HTTP invoke step showing the `trackable` input property

Logging from Flux

The Flux engine adds a Tracker document when a Flux service is executed. A document state is added for each of the following scenarios:

  • Start of execution, where the document state contains the context of the Flux service in a JSON file.
  • Execution of a state, where the document state contains the context of the invoked state in a JSON file.
  • Occurrence of an exception, where the document contains the stacktrace of the exception from the {stateName}-error.log file.
  • End of execution.

No tracking for certain Flux invocations

A Flux invocation will not be logged to Tracker if it is debugged or run using the IDE.

We can track Flux executions invoked via REST API by configuring the base path /flux-api in the trackers.xml. For example:

1
2
3
4
<?xml version="1.0" encoding="UTF-8" ?>
<trackers>
    <tracker pattern="/flux-api/**" document-type="FluxExecution"/>
</trackers>

Or if per-package tracking is desired:

1
2
3
4
5
<?xml version="1.0" encoding="UTF-8" ?>
<trackers>
    <!-- where 'examples' is the name of the package -->
    <tracker pattern="/flux-api/examples/**" document-type="FluxExecution"/>
</trackers>

Differentiating by type

The document-type attribute defaults to HTTP. While this value would work for all cases, it is recommended to supply a customized, sensible value if you want an easier time searching and filtering documents.

The above setup enables tracking, but the document will only record states related to the request-response lifecycle, and not the Flux states. To include Flux states in the Tracker document, declare a GloopString input property named $trackerId on your Flux service.

The Tracker document may further be updated, by calling services from TrackerMethods and using $trackerId from the input.

Logging from Martini endpoints

To log data processed by Martini endpoints to Tracker, the Logging property of the endpoint you want to track must be checked in the Endpoint configuration editor. You will be able to set this property when creating or configuring an endpoint. Unchecking this property means data processed by the endpoint is left unindexed.

Endpoint logging property

Endpoint logging property

Programmatic logging

To manually create or update Tracker documents in the Tracker search index, developers can use the services in Tracker. The TrackerMethods class is located in the core package, under the services/io/toro/martini directory, as seen in the Navigator view.

Martini showing the available methods in Gloop for Tracker

Martini showing the available methods in Gloop for Tracker

To update a Tracker document, you would need its internalId. For certain invocations that are already tracked, the document may be further populated by invoking functions on the internal ID injected to the service being used. This automatically becomes available by declaring:

Tracker ID

$trackerId is analogous to internalId. For backwards-compatibility, declaring internalId as the variable name will also work, albeit only on Gloop and Groovy services.

Discrepancies

There are a few discrepancies among Tracker documents, depending on how a service is exposed. To show you these differences, the following examples below1 are provided:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
  "internalId": "fa30f599-6030-4ad5-836e-2a18926ee1a6",
  "externalId": "1534393691669",
  "type": {
    "id": "hello",
    "name": "hello"
  },
  "stateName": "Processed",
  "senderId": "0:0:0:0:0:0:0:1",
  "receiverId": "/api/hello/world",
  "username": "Anonymous",
  "timestamp": 1534393691629,
  "logs": [],
  "properties": [
    {
      "key": "Martini_Server",
      "value": "Martini@TORO-MM64:8080 (fe80:0:0:0:42cd:e074:a3d4:4fdc%utun0)"
    },
    {
      "key": "Duration(ms)",
      "value": "29"
    }
  ],
  "states": [
    {
      "id": 1,
      "name": "Received",
      "timeReceived": 1534393691629,
      "canBeResubmitted": false,
      "contentFileName": "request.xml",
      "contentSize": 5347
    },
    {
      "id": 2,
      "name": "Processed",
      "timeReceived": 1534393691793,
      "canBeResubmitted": false,
      "contentFileName": "response.txt",
      "contentSize": 13
    }
  ]
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
  "internalId": "7ae04eee-d419-404d-ada6-da623e3e9ed5",
  "externalId": "1534394377149",
  "type": {
    "id": "hello",
    "name": "hello"
  },
  "stateName": "Processed",
  "senderId": "0:0:0:0:0:0:0:1",
  "receiverId": "/api/hello/world",
  "username": "Anonymous",
  "timestamp": 1534394377149,
  "logs": [],
  "properties": [
    {
      "key": "Martini_Server",
      "value": "Martini@TORO-MM64:8080 (fe80:0:0:0:42cd:e074:a3d4:4fdc%utun0)"
    },
    {
      "key": "Duration(ms)",
      "value": "208"
    }
  ],
  "states": [
    {
      "id": 3,
      "name": "Received",
      "timeReceived": 1534394377149,
      "canBeResubmitted": false,
      "contentFileName": "request.xml",
      "contentSize": 5195
    },
    {
      "id": 4,
      "name": "Processed",
      "timeReceived": 1534394377366,
      "canBeResubmitted": false,
      "contentFileName": "response.txt",
      "contentSize": 15
    }
  ]
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
{
  "internalId": "d3d21653-e75c-46c1-a6a3-dfc82b0db9a6",
  "externalId": "1534394572980",
  "type": {
    "id": "hello",
    "name": "hello"
  },
  "stateName": "Processed",
  "senderId": "0:0:0:0:0:0:0:1",
  "receiverId": "/api/hello/world",
  "username": "Anonymous",
  "timestamp": 1534394572979,
  "logs": [],
  "properties": [
    {
      "key": "Martini_Server",
      "value": "Martini@TORO-MM64:8080 (fe80:0:0:0:42cd:e074:a3d4:4fdc%utun0)"
    },
    {
      "key": "Duration(ms)",
      "value": "303"
    }
  ],
  "states": [
    {
      "id": 5,
      "name": "Received",
      "timeReceived": 1534394572979,
      "canBeResubmitted": false,
      "contentFileName": "request.xml",
      "contentSize": 5195
    },
    {
      "id": 6,
      "name": "Processed",
      "timeReceived": 1534394573295,
      "canBeResubmitted": false,
      "contentFileName": "response.txt",
      "contentSize": 15
    }
  ]
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  "internalId": "97bdc10b-237f-4c27-97f9-867145ee7ad7",
  "externalId": "1534400984618",
  "type": {
    "id": "Scheduler",
    "name": "Scheduler"
  },
  "stateName": "Processed",
  "senderId": "SendScheduledEmail",
  "receiverId": "gloop:endpointServices.gloop.schedulerExamples.SendScheduledEmail/endpointServices.gloop.schedulerExamples.SendScheduledEmail",
  "username": "",
  "timestamp": 1534400984618,
  "logs": [],
  "properties": [
    {
      "key": "Duration(ms)",
      "value": "6350"
    }
  ],
  "states": [
    {
      "id": 7,
      "name": "Started",
      "timeReceived": 1534400984618,
      "canBeResubmitted": false,
      "contentFileName": "",
      "contentSize": 0
    },
    {
      "id": 8,
      "name": "Processed",
      "timeReceived": 1534400990995,
      "canBeResubmitted": false,
      "contentFileName": "",
      "contentSize": 0
    }
  ]
}

As you may notice, Groovy service, Gloop API, and ad hoc Gloop service endpoints have fairly similar fields.

  • By default, these types have set the leading URL mapping as the name and ID of the type of each document. In the example, all endpoints are accessible via the path /hello/{name}, wherein the leading URL path is hello.
  • The senderId is the IP address where the request came from.
  • The receoverId is the URL mapping where the request was received at.

Martini endpoints service invocations differ in that:

  • The default name of their starting state is "Started" instead of "Received".
  • The senderId is the name of the endpoint itself.
  • The receiverId is the qualified name of the service of the endpoint.

  1. Displayed JSON data are obtained via the /tracker/document/{id} endpoint. ↩