Get Credentials

Get credentials

After signing in, you can create credentials that can be used to call the Platform APIs.

Batch Ingestion API (1.0)

Download OpenAPI specification:Download

Batch ingestion allows you to ingest data into Adobe Experience Platform as batch files. Batches are units of data that consist of one or more files to be ingested as a single unit. Once ingested, batches provide metadata that describes the number of records successfully ingested, as well as any failed records and associated error messages.

Use the Batch Ingestion API to create batches, upload files, check on upload status, and more.

(NEW) Interactive API documentation

You can now interact with the Experience Platform API endpoints directly from this API reference page. Get your authentication credentials and use the Try it functionality in the right rail. Note that by using this functionality, you are making real API calls. Keep this in mind when you interact with production sandboxes.

Batch Ingestion

Batch ingestion is used to ingest data into Experience Platform as batch files. For example, data being ingested can be the profile data from a flat file in a CRM system (for example: Parquet or JSON) or data that conforms to a known Experience Data Model (XDM) schema within the Schema Registry.

Create a new batch

Request
header Parameters
Authorization
required
string

The access token which can be copied from your Experience Platform integration, prefixed with "Bearer ". For more information on how to obtain this value, visit the authentication tutorial.

x-api-key
required
string

The API key which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-gw-ims-org-id
required
string

The Organization ID which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-sandbox-name
required
string

The name of the sandbox in which the operation will take place. See the sandboxes overview documentation for more information.

Content-Type
required
string

The type of content being sent in the body of the request. The value should be 'application/json`.

Request Body schema: application/json
required

The payload that contains information needed to create a batch.

datasetId
required
string

The ID of the dataset that you are creating a batch for. When creating the batch, the files uploaded must conform to the XDM schema of the dataset ID that you provided.

externalId
string

A client supplied batch identifier, that is expected to be unique within your organization. This defaults to the batch ID if not specified.

required
object

The format of the ingested batch files. It can be any of the following values; JSON, Parquet, or CSV. The ingestion of a multi-line JSON file is only supported when the inputFormat field format is set to JSON and the isMultiLineJson flag is set to true. For more information, please read the batch ingestion troubleshooting guide.

object

Provide a list of client supplied IDs to identify the previous batches that this new batch is replacing. Any ID supplied from the client as part of the request body must be prefixed with a $.

object

Additional batch tags. This object contains any properties associated with the previous batches.

Responses
201

The batch was successfully created.

400

A bad request. The request is malformed.

401

Access is unauthorized. The user needs to provide a valid bearer token.

414

The URL length exceeds the allowed 2000 characters.

500

An internal server error has occurred.

503

The service is currently unavailable.

post/batches
Request samples
application/json
{
  • "datasetId": "5da9452f7de80400007jc52a",
  • "externalId": "01EQ1C48TR4GD4FE7NH2522TCQ",
  • "inputFormat": {
    },
  • "replay": {
    },
  • "tags": {
    }
}
Response samples
application/json
{
  • "id": "01EQ1C48TR4GD4FE7NH2522TCQ",
  • "imsOrg": "5C1328435BF324E90A49402A@AdobeOrg",
  • "created": "2022-11-09T18:45:31.256Z",
  • "createdClient": "acme_foundation_push",
  • "createdUser": "acme_foundation_dataTracker@AdobeID",
  • "updatedUser": "acme_foundation_dataTracker@AdobeID",
  • "updated": "2022-11-09T18:45:31.256Z",
  • "started": "2022-11-09T18:45:31.256Z",
  • "completed": "2022-12-09T18:45:31.256Z",
  • "status": "active",
  • "recordCount": 57,
  • "failedRecordCount": 3,
  • "errors": [
    ],
  • "size": "1,342,576",
  • "version": "1.0.0",
  • "availableDates": {
    },
  • "relatedObjects": [
    ],
  • "metrics": [
    ],
  • "tags": [
    ],
  • "inputFormat": {
    }
}

Upload a small file to a dataset

This endpoint lets you upload files under datasets for a batch initialized by the bulk ingestion API. The PUT method creates or updates the entire request stream as the file bytes under the path denoted by the filePath and is idempotent. Subsequent calls to this endpoint will replace the existing file on the referenced path. Note: Due to gateway size and timeout limits, there is currently a size threshold of 256MB. If the file is larger than that, you must use the large file upload mechanism.

Request
path Parameters
batchId
required
string

The ID of the batch that the data is being uploaded to.

datasetId
required
string

The ID of the dataset that the data is being uploaded to.

filePath
required
string

The location where the file will be uploaded to, on the Adobe side. Note: The directory structure denoted in this path is preserved and can be used to denote partitions that the file data belongs to.

header Parameters
Authorization
required
string

The access token which can be copied from your Experience Platform integration, prefixed with "Bearer ". For more information on how to obtain this value, visit the authentication tutorial.

x-api-key
required
string

The API key which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-gw-ims-org-id
required
string

The Organization ID which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-sandbox-name
required
string

The name of the sandbox in which the operation will take place. See the sandboxes overview documentation for more information.

Content-Type
required
string

The type of content being sent in the body of the request. The value should be 'application/octet-stream'.

Value: "application/octet-stream"
Responses
200

The file was successfully updated.

201

The file was successfully created.

400

The request is malformed.

401

Access is unauthorized. The IMS token provided is invalid.

404

The batch or dataset resource is not found.

414

The URL length exceeds the allowed 2000 characters.

415

The uploaded file media type isn't supported.

503

The service is unavailable.

put/batches/{batchId}/datasets/{datasetId}/files/{filePath}

Upload part of a large file to a batch

Files larger than 256MB are uploaded in parts. The PATCH endpoint provides you a way to upload a part of your large file that was initialized via the POST endpoint. File sizes of between 0-512000000 bytes are accepted.

Request
path Parameters
batchId
required
string

The ID of the batch that the data is being uploaded to.

datasetId
required
string

The ID of the dataset that the data is being uploaded to.

filePath
required
string

The location where the file will be uploaded to, on the Adobe side. Note: The directory structure denoted in this path is preserved and can be used to denote partitions that the file data belongs to.

header Parameters
Authorization
required
string

The access token which can be copied from your Experience Platform integration, prefixed with "Bearer ". For more information on how to obtain this value, visit the authentication tutorial.

x-api-key
required
string

The API key which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-gw-ims-org-id
required
string

The Organization ID which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-sandbox-name
required
string

The name of the sandbox in which the operation will take place. See the sandboxes overview documentation for more information.

Content-Type
required
string

The type of content being sent in the body of the request. The value should be 'application/octet-stream'.

Value: "application/octet-stream"
Content-Range
required
string

The lowest and highest value of bytes in the file being uploaded with this request.

Request Body schema: multipart/form-data
required
file
required
string <binary>

The full path and name of the file you are trying to upload. This file path is the local file path with the file format suffix. For example, @{FILE_PATH_AND_NAME}.parquet. Note, as the 'application/octet-stream' content type is a binary file that contains data in the form of bytes instead of text, no example request body is provided here.

Responses
200

The file was successfully uploaded.

400

The request is malformed.

401

Access is unauthorized. The IMS token provided is invalid.

404

The batch, dataset, or file resource is not found.

414

The URL length exceeds the allowed 2000 characters.

415

The uploaded file media type isn't supported.

500

An internal server error has occurred.

503

The service is unavailable.

patch/batches/{batchId}/datasets/{datasetId}/files/{filePath}
Response samples
application/json
{
  • "name": "acme/customers/campaigns/summer.json",
  • "sizeInBytes": 3682
}

Either initialize or finish uploading a large file

This endpoint should only be used when uploading a large file to the batch. Large file upload should be used for files larger than 256MB.

Request
path Parameters
batchId
required
string

The ID of the batch that the data is being uploaded to.

datasetId
required
string

The ID of the dataset that the data is being uploaded to.

filePath
required
string

The location of the file you want to initialize or complete within the Adobe data lake. Note: The directory structure denoted in this path is preserved and can be used to denote partitions that the file data belongs to.

query Parameters
action
required
string

The action to perform on the file. The currently supported values are INITIALIZE and COMPLETE. To begin initializing the file for upload, use the value INITIALIZE. To mark the file as finished, use the value COMPLETE.

Enum: "INITIALIZE" "COMPLETE"
header Parameters
Authorization
required
string

The access token which can be copied from your Experience Platform integration, prefixed with "Bearer ". For more information on how to obtain this value, visit the authentication tutorial.

x-api-key
required
string

The API key which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-gw-ims-org-id
required
string

The Organization ID which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-sandbox-name
required
string

The name of the sandbox in which the operation will take place. See the sandboxes overview documentation for more information.

Content-Type
required
string

The type of content being sent in the body of the request. The value should be 'application/json'.

Responses
200

The file is now finished uploading.

400

The request is malformed.

401

Access is unauthorized. The IMS token provided is invalid.

404

The batch, dataset, or file resource is not found.

414

The URL length exceeds the allowed 2000 characters.

500

An internal server error has occurred.

503

The service is unavailable.

post/batches/{batchId}/datasets/{datasetId}/files/{filePath}
Response samples
application/json
{
  • "name": "acme/customers/campaigns/summer.json",
  • "sizeInBytes": 3682
}

Get the current status of the large file

Files larger than 512MB are uploaded in parts. The HEAD endpoint provides a way to get the byte range information received by the server so far.

Request
path Parameters
batchId
required
string

The ID of the batch that the data is being uploaded to.

datasetId
required
string

The ID of the dataset that the data is being uploaded to.

filePath
required
string

The location of the file within the Adobe data lake that you want to retrieve status information for. Note: The directory structure denoted in this path is preserved and can be used to denote partitions that the file data belongs to.

header Parameters
Authorization
required
string

The access token which can be copied from your Experience Platform integration, prefixed with "Bearer ". For more information on how to obtain this value, visit the authentication tutorial.

x-api-key
required
string

The API key which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-gw-ims-org-id
required
string

The Organization ID which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-sandbox-name
required
string

The name of the sandbox in which the operation will take place. See the sandboxes overview documentation for more information.

Responses
200

The status information was successfully returned.

400

The request is malformed.

401

Access is unauthorized. The IMS token provided is invalid.

404

The batch, dataset, or file resource is not found.

414

URL length exceeds the allowed 2000 characters.

415

The uploaded file media type isn't supported.

500

An internal server error has occurred.

503

The service is unavailable.

head/batches/{batchId}/datasets/{datasetId}/files/{filePath}

Retrieve a preview for a batch

This endpoint generates a data preview for the files uploaded to the batch so far. The preview can be generated either collectively for all the batch datasets, or for only selected datasets.

Request
path Parameters
batchId
required
string

The ID of the batch that you are trying to preview.

datasetId
required
string

The ID of the dataset that you are trying to preview.

query Parameters
format
required
string

The file format for the uploaded file.

isMultiLineJson
boolean
Default: false

This flag indicates whether the files to be ingested are in a multi-line json format.

delimiter
string
Default: ","

The character used to specify boundaries between separate strings when parsing column values.

quote
string
Default: "\""

The character used to quote a value when parsing data.

escape
string
Default: "\\"

The character used to escape a reserved character when parsing data.

charset
string
Default: "UTF-8"

The character encoding used for parsing data.

header
boolean
Default: true

The flag to indicate if the header is supplied in the dataset files.

nrow
integer <int32>
Default: 0

The number of rows to parse.

header Parameters
Authorization
required
string

The access token which can be copied from your Experience Platform integration, prefixed with "Bearer ". For more information on how to obtain this value, visit the authentication tutorial.

x-api-key
required
string

The API key which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-gw-ims-org-id
required
string

The Organization ID which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-sandbox-name
required
string

The name of the sandbox in which the operation will take place. See the sandboxes overview documentation for more information.

Responses
200

The preview for the batch was successfully generated.

400

The request is malformed.

401

Access is unauthorized. The IMS token provided is invalid.

404

The batch or dataset resource is not found.

414

The URL length exceeds the allowed 2000 characters.

415

The uploaded file media type isn't supported.

500

An internal server error has occurred.

503

The service is unavailable.

get/batches/{batchId}/datasets/{datasetId}/preview
Response samples
application/json
{
  • "formatParams": {
    },
  • "preview": {
    }
}

Complete a batch

After you have finished uploading all of the different parts of the file, signal that the batch ingestion is complete to trigger the downstream data promotion workflow.

Request
path Parameters
batchId
required
string

The ID of the batch that you want to perform the signal action on.

query Parameters
action
required
string

The action to take on the batch. To signal that the batch file ingestion is complete, use the 'COMPLETE' action query parameter.

Enum: "COMPLETE" "ABORT" "FAIL" "REVERT"
header Parameters
Authorization
required
string

The access token which can be copied from your Experience Platform integration, prefixed with "Bearer ". For more information on how to obtain this value, visit the authentication tutorial.

x-api-key
required
string

The API key which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-gw-ims-org-id
required
string

The Organization ID which can be copied from your Experience Platform integration. For more information on how to obtain this value, visit the authentication tutorial.

x-sandbox-name
required
string

The name of the sandbox in which the operation will take place. See the sandboxes overview documentation for more information.

Responses
200

The batch has been successfully promoted.

400

The request is malformed.

401

Access is unauthorized. The IMS token provided is invalid.

404

The batch is not found.

414

The URL length exceeds the allowed 2000 characters.

500

An internal server error has occurred.

503

The service is unavailable.

post/batches/{batchId}
Response samples
application/json
{
  • "id": "01EQ1C48TR4GD4FE7NH2522TCQ",
  • "imsOrg": "5C1328435BF324E90A49402A@AdobeOrg",
  • "created": "2022-11-09T18:45:31.256Z",
  • "createdClient": "acme_foundation_push",
  • "createdUser": "acme_foundation_dataTracker@AdobeID",
  • "updatedUser": "acme_foundation_dataTracker@AdobeID",
  • "updated": "2022-11-09T18:45:31.256Z",
  • "started": "2022-11-09T18:45:31.256Z",
  • "completed": "2022-12-09T18:45:31.256Z",
  • "status": "active",
  • "recordCount": 57,
  • "failedRecordCount": 3,
  • "errors": [
    ],
  • "size": "1,342,576",
  • "version": "1.0.0",
  • "availableDates": {
    },
  • "relatedObjects": [
    ],
  • "metrics": [
    ],
  • "tags": [
    ],
  • "inputFormat": {
    }
}