Bulk Data API (v 3.0)

Learn about how to export the bulk Aurora Community Analytics data using Bulk Data API v3.0

Added in 23.12.

Bulk Data API provides the log (including the metadata) of different community actions such as Page Views, Posts, and Likes. Bulk Data API can be used to store the Aurora community event logs for specific days in your own data store and then use your own report builder tool to build custom metrics, KPIs, and reports that you can schedule and run on a regular and automated basis. You may fetch up to seven (7) days of data in a single request.

Notes and recommendations

Get started with these Bulk Data queries

Our Requesting essential community metrics via the Bulk Data API guide in Atlas provides useful Bulk Data API queries to measure the health and activity of your community at different phases of your community's maturity.

• We recommend specifying application/json in the Accept field of your request header. Without specifying JSON in the header, the API defaults to CSV.
• We recommend requesting specific fields in the order you want them returned in your API call. If no fields are requested, all fields are returned. Requesting specific fields allows you to limit what the user requests, thus reducing the load on the server and allowing faster response time.
• fromDate and toDate support the following (case-sensitive) formats: yyyy-MM-dd and yyyy-MM-ddTHH:mm
• Although supported, we do not recommend sending requests with a fromDate that defines a time other than 00:00 UTC.
• If the fromDate does not start at 00:00 UTC, set a client timeout of 5 minutes. Otherwise, 30 seconds is acceptable.

Known limitations

At this time, we strongly recommend limiting to one request at a time.

Example export

We provide an example export as an attachment to Using the Khoros Bulk Data API in the Khoros Communities Documentation Knowledge Base.

API Application for Bulk Data API

Before making an API request, you must create a web application for the bulk data API.

1. Log in to your Aurora Community.
2. Go to Account > Dev Tools > Create App.
3. Create a bulk data API with Analytics role.

   

4. After the Bulk Data API application is created, you can find the client ID and Client Secret in the API Apps section of the Dev Tools.
5. Download the Client Shared Key for Access Token generation.

   

6. Generate the access token using the OAuth 2.0 Client Credentials Grant Flow.

Example

Retrieving all the community analytics data

Request

The sample Bulk Data API call shows the general format to retrieve the community analytics data.

curl --location 'https://[COMMUNITY DOMAIN]/t5/s/api/2.1/analytics/bulk/export?bda.fromDate=[start_date]&bda.toDate=[end_date]' \
--header 'Authorization: Bearer[TOKEN]'

import uuid
import hashlib
import requests
import time

# Update values as per requirements
client_id = '[CLIENT ID OF YOUR APPLICATION]'
client_secret = '[CLIENT SECRET KEY OF YOUR APPLICATION]'
secret_key = "[SECRET KEY OF YOUR APPLICATION]"
callback_url = 'https://[COMMUNITY DOMAIN]/api/auth/callback'
url_base = 'https://[COMMUNITY DOMAIN]/t5/s'
api_url = url_base + "/api/2.0/auth/accessToken"
access_token = None
from_date = '2023-12-20'
to_date = '2023-12-23'

def cc_hash(client_id, client_secret, nonce, secret_key):
    message = client_id + ":" + client_secret + ":" + nonce + ":" + str(int(time.time()/60))
    checksum = hashlib.sha256()
    checksum.update(bytes(message, 'utf-8'))
    checksum.update(bytes(secret_key, 'utf-8'))
    checksum = checksum.hexdigest()
    print(checksum)
    return checksum

if access_token is None:
    nonce = uuid.uuid4().hex
    print(nonce)

    digest = cc_hash(client_id, client_secret, nonce, secret_key)
    headers = {'nonce': nonce, 'Content-Type': 'application/json'}
    payload = {
        "client_id": client_id,
        "client_secret": client_secret,
        "redirect_uri": callback_url,
        "grant_type": "client_credentials",
        "cc_hash": digest,
        "hash_algorithm": "sha256"
    }
    response = requests.post(api_url, headers=headers, json=payload)
    if response.ok:
        result = response.json()
        access_token = result['data']['access_token']
        print('Access: ' + access_token)
        print('Refresh: ' + result['data']['refresh_token'])
    else:
        print("Could not fetch access token: " + response.text)

if access_token is not None:
    accept = 'text/csv'
    headers = {'Authorization': 'Bearer ' + access_token, 'Accept': accept}
    response = requests.get(
        url_base + '/api/2.1/analytics/bulk/export?bda.fromDate=' + from_date + '&bda.toDate=' + to_date,
        headers=headers)
    data = response.text
    print(data)

where

• community domain is your community url.
• token is the authentication token generated using the OAuth 2.0 Client Credentials Grant Flow.

Response

The response for the above API call is provided below.

document.id,action.key,event.time.ms,message.uid,message.subject,message.is_topic,message.type,labels,tags,conversation.uid,parent.conversation_uid,conversation.title,board.uid,board.title,community_app,node.ancestor_path,action.is_solution_view,grouphub.uid,status,user.uid,user.login,user.sso.id,user.community.role_name,user.community.rank_name,user.registration_status,visitor.id,visit.id,target.user.uid,target.user.login,target.user.sso_id,target.user.community.role_name,target.user.community.rank_name,target.user.registration_status,action.duration,action.span.seconds,action.weight,action.is_moderator_action,badge.uid,badge.title,badge.is_retroactive,badge.activation_date,search.entity.type,search.terms,search.has_results,search.page_number,search.filters,search.location,search.type,search.is_promoted_search_result,search.results,privatemessage.type,privatemessage.newthread,occasion.start_time,occasion.end_time,occasion.rsvp_response,occasion.is_invited,subscription.is_direct,subscription.type,request.url.host,request.headers.user_agent,request.headers.referrer_url,request.headers.referrer_host,request.device,request.geo.city,request.geo.country,request.geo.country_code,request.geo.geohash,request.geo.postal,request.geo.region,request.geo.timezone,request.geo.latitude,request.geo.longitude,badges.award,badges.revoke

Exporting this information to Excel produces a file like this.

If you prefer to get your data in JSON format, add “Accept: application/json” to the header of your request.

Retrieving the data of the specific fields

Request

The sample bulk data API shows the request to retrieve only the action.key and visitor.id fields.

curl --location 'https://[COMMUNITY DOMAIN]/t5/s/api/2.1/analytics/bulk/export?bda.fromDate=[YYYY-mm-dd]&bda.toDate=[YYYY-mm-dd]&bda.fields=action.key,visitor.id'
--header 'Authorization: Bearer[TOKEN]'

Response

The response for the above API call is provided below.

action.key,visitor.id

Exporting this information to Excel produces a file like this.

See the Export Reference section below for definitions and examples of each column value.

Connecting to the API

The following examples show how to connect to the Bulk Data API using:

• Shell
• Java - HttpClient 4.5
• Python 3

Shell

curl --location 'https://[COMMUNITY DOMAIN]/t5/s/api/2.1/analytics/bulk/export?bda.fromDate=[start_date]&bda.toDate=[end_date]' \
--header 'Authorization: Bearer [TOKEN]'

Java - HttpClient 4.5

String urlBase       = "https://[COMMUNITY DOMAIN]/t5/s";
String accept   = "[text/csv or application/json]";
String accessToken = "Bearer [access_token]";
 
String fromDate   = "[fromDate]";
String toDate   = "[toDate]";
 
String url = urlBase + "/api/2.1/analytics/bulk/export?bda.fromDate=" + fromDate + "&bda.toDate=" + toDate;
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
    HttpGet httpget = new HttpGet(url);
    httpget.addHeader("Authorization", accessToken);
    httpget.addHeader("Accept", accept);
    CloseableHttpResponse response = httpclient.execute(httpget);
    try {
        System.out.println(EntityUtils.toString(response.getEntity()));
    } finally {
        response.close();
    }
} finally {
    httpclient.close();
}

Python 3

# Usage: python3 <filename> <start_date> <end_date>
import requests
import sys

# TODO: Update script to set these values
url_base     = 'https://[COMMUNITY DOMAIN]/t5/s'
accept       = '[text/csv or application/json]'
access_token = 'Bearer [TOKEN]'

if(accept != "application/json" and accept != "text/csv"):
    print("Invalid accept value. It must be either 'text/csv' or 'application/json'")
    exit()

# These values are passed as command line arguments
from_date   = sys.argv[1]
to_date     = sys.argv[2]

response = requests