Setting up the Marketo Observability Data Stream

These instructions describe how to set up and get started using Adobe I/O Events to subscribe to Marketo Observability events.

• Introduction
• Prerequisite Setup
• Setup Adobe I/O
• Developer Guidelines
• Debug

Introduction

The Marketo Observability Data Stream (MODS) provides insights about the data flow into Marketo through the Data Ingestion Service. The stream offers several different event types:

• Metrics - Periodic aggregated metrics showing all data sent to the API and processed into Marketo
• Results - Per API request results for tracking and verification
• Status - Periodic notices providing insight into the remaining API quota and any request processing backlog

Note: MODS (Marketo Observability Data Stream) is currently a Beta Product

Prerequisite Setup

The following are required to be able to subscribe to the data stream:

• Marketo Engage subscription that is IMS-enabled
• Marketo Engage subscription provisioned with a Performance Tier or CET package
• Developer or admin account in the Experience Cloud

Getting Started with Adobe I/O

See the Adobe I/O Events Docs

Basic instructions for this use case, starting from the developer console:

When prompted, click the designated button to proceed

• Select Create new project

  

• Select Add event

  

• Filter by Experience Cloud

• Select Marketo Observability Data Stream

  

• Subscribe to observability events that you're interested in

  

• Set up authentication (OAuth Server-to-Server) to be used for accessing the Journaling API

  

• Set the name and description for your event registration

  

• Finish Event Registration Setup

  

  • Provide a name and description for this event subscription

  • Optionally choose whether to enable Webhook or Runtime action

    • Enable Webhook

      • We recommend batch over single webhooks
      • For Webhook URL a public https endpoint must be provided
      • The endpoint must be able to handle get and post requests
      • The get request must respond with the challenge query if it exists
      • The post request must respond that it received the message or the webhook will re-attempt to send several times before giving up and automatically disabling the webhook sends

    • Enable Runtime action

      • Getting Started with Adobe I/O Runtime
      • Runtime Actions as Webhook

• After Saving

  

  • Verify that the Status is Active
  • If Webhook was selected, verify that it successfully passed the challenge without errors

Developer Guidelines

When setting up a project to subscribe to events, there are three ways to interact with those event subscriptions in order to receive the events. The first is Journaling, which provides a pull model in which events can be pulled via API and stores up to 7 days of past events. The second is Webhooks, which can be configured to send events either as single events or batched to a webhook endpoint in near real-time with the event occurrence. Third is Runtime, where you can set up your own custom function within Adobe that events will automatically run through near-real time.

Journaling

Getting Started with Journaling

Important Takeaways:

• Stores up to 7 days of history
• Can be iterated through from any previous event within the history
• Will still receive and store events even if webhook is failing
• Useful for fetching events that were missed due to webhook issues or for a pulling mechanism instead of webhook push

Webhooks

Getting Started with Adobe I/O Events Webhooks

Webhook Endpoint Requirements:

• Handle GET and POST requests

• Respond with a 200-type response within a reasonable time period

• Challenge Request

  • GET request with challenge query parameter
  • Must respond with value of challenge query parameter

• Webhook Events

  • POST request with JSON data body with one or more events
  • Recommended to set up webhook as batch

Event Data Structure Examples

Events are structured in JSON format using the CloudEvents spec

Metrics

{
    "id": "b90382d8-6b23-11ee-b962-0242ac120002",
    "specversion": "1.0",
    "type": "com.adobe.platform.marketo.observability.metrics",
    "source": "urn:data_ingestion_service",
    "time": "2023-08-14T18:00:00Z",
    "datacontenttype": "application/json",
    "data": {
        "munchkinId": "123-ABC-456",
        "windowStart": "2023-08-14T17:00:00Z",
        "requests": {
            "received": 3,
            "processed": 1,
            "rejected": {  
                "606": 1,
                "607": 1
            }
        },
        "records": {
            "person": {
                "high": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                },
                "normal": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                }
            },
            "customObject": {
                "high": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                },
                "normal": {
                    "received": 100,
                    "created": 90,
                    "updated": 5,
                    "failed": {
                        "503": 5
                    }
                }
            }
        }
    }
}

Results

Result events contain an array of processed requests indicating what was processed. These may be complete or partial results depending on the size of the request payload. Request payloads are processed in chunks, and if a request is split into multiple chunks, the processing results could potentially be reported across multiple events.

{
    "id": "b90382d8-6b23-11ee-b962-0242ac120002",
    "specversion": "1.0",
    "type": "com.adobe.platform.marketo.observability.results",
    "source": "urn:data_ingestion_service",
    "time": "2023-08-14T17:30:00Z",
    "datacontenttype": "application/json",
    "data": {
        "munchkinId": "123-ABC-456",
        "requests": [
            {
                "requestId": "cf0a1a20-668e-492a-8ec2-ce8747507068",
                "requestTime": "2023-08-14T17:20:00Z",
                "clientId": "foo@marketo.com",
                "correlationId": "6180bb48-8dc7-4fc5-85ca-a59dd1edb0f3",
                "requestSource": "Adobe Journey Optimizer",
                "objectType": "person",
                "priority": "high",
                "records": {
                    "received": 100,
                    "created": 90,
                    "updated": 10,
                    "failed": {}
                }
            },
            {
                "requestId": "67b28858-f5ee-45ac-aa40-63a04085e6be",
                "requestTime": "2023-08-14T17:20:00Z",
                "clientId": "foo@marketo.com",
                "correlationId": "6180bb48-8dc7-4fc5-85ca-a59dd1edb0f3",
                "requestSource": "Public API",
                "objectType": "customObject",
                "priority": "normal",
                "records": {
                    "received": 100,
                    "created": 10,
                    "updated": 10,
                    "failed": {
                        "503": 10,
                        "404": 10
                    }
                }
            }
        ]
    }
}

Status

{
    "id": "b90382d8-6b23-11ee-b962-0242ac120002",
    "specversion": "1.0",
    "type": "com.adobe.platform.marketo.observability.status",
    "source": "urn:data_ingestion_service",
    "time": "2023-08-14T18:00:00Z",
    "datacontenttype": "application/json",
    "data": {
        "munchkinId": "123-ABC-456",
        "quota": 65432100,
        "queue": {
            "high": {
                "lagSeconds": 15,
                "requestBacklog": 1,
                "recordBacklog": 100
            },
            "normal": {
                "lagSeconds": 135,
                "requestBacklog": 10,
                "recordBacklog": 1000
            }
        },
    }
}

Data Field Definitions

Debug

Debug Tracing

Once you have successfully completed your setup and event subscription registration, events should start being stored in the journal. In addition, if you have webhooks or runtime set up, the events will go through those flows. From the project's page in the event registration details, you should see a tab for Debug Tracing. For webhooks, this will show a record of failed and successful challenge attempts as well as webhook attempts. Each request includes the request/response details to help debug.

There is also an Event Browser tab which lets you manually look at events in the Journal from the Developer Console UI