Search to Content Workflow

This guide walks you through the complete workflow of using the Tollbit API to search for content, check rates, and retrieve content. You can use either cURL commands or the Python SDK.

Prerequisites

Before you begin, make sure you have:

cURL installed: cURL should be available on most systems. Check with:
```
curl --version
```
Obtained your API key: You'll need a Tollbit organization API key. Replace <key> in the examples below with your actual API key.
Set your user agent: This identifies your application. The user agent should be added to the dev dashboard in the My Agents tab

Complete Workflow

The typical workflow consists of three main steps:

Search for content
Get rates for a specific result
Get the content (token acquisition is handled automatically with Python SDK, or separately with cURL)

Let's walk through each step with examples.

Step 1: Make a Search

Start by searching for content across the TollBit platform.

Basic Search

curl --location 'https://gateway.tollbit.com/dev/v2/search?q=DIY%20projects%20for%20Millenials' \
--header 'TollBitKey: <key>'

Search with Additional Parameters

You can customize your search with additional query parameters:

size: Number of results to return (maximum 20)
next-token: Token for pagination to get the next page
properties: Comma-separated list of domains (max 20) to filter results to only those properties

# Search with size limit
curl --location 'https://gateway.tollbit.com/dev/v2/search?q=python%20tutorial&size=5' \
--header 'TollBitKey: <key>'

# Search on specific properties (filter to only these domains)
curl --location 'https://gateway.tollbit.com/dev/v2/search?q=tutorial&size=10&properties=example.com,tutorial.com' \
--header 'TollBitKey: <key>'

# Pagination - get next page
curl --location 'https://gateway.tollbit.com/dev/v2/search?q=DIY%20projects&next-token=<next_token_from_previous_response>' \
--header 'TollBitKey: <key>'

Understanding the Response

The search response is a JSON object with the following structure:

{
  "nextToken": "string",
  "items": [
    {
      "title": "string",
      "url": "string",
      "publishedDate": "string",
      "publisher": {
        "domain": "string",
        "name": "string"
      },
      "availability": {
        "discoverable": true,
        "readyToLicense": true
      }
    }
  ]
}

Understanding Search Result Flags:

discoverable: Indicates the content is available in Tollbit
readyToLicense / ready_to_license: Indicates the property owner has already set rates and the content is ready to transact

Example: Parsing Search Results

# Save the response to a file for easier parsing
curl --location 'https://gateway.tollbit.com/dev/v2/search?q=DIY%20projects%20for%20Millenials' \
--header 'TollBitKey: <key>' \
--output search_results.json

# Using jq to parse (if installed)
cat search_results.json | jq '.items[0].url'  # Get first result URL
cat search_results.json | jq '.items[0].title'  # Get first result title

Step 2: Get Rates for a Result

Once you've found a result you're interested in, check the available rates and licenses for that content.

Get Rates

curl --location 'https://gateway.tollbit.com/tollbit/dev/v2/rate/https://www.forbes.com/sites/jefffromm/2018/03/27/marketing-to-millennial-new-home-buyers-lowes-innovative-diy-training-program/' \
--header 'TollbitKey: <key>'

Note: The URL path in the rate endpoint should be the full URL of the content (including protocol), URL-encoded if necessary.

Understanding the Response

The rates response is a JSON array of rate objects:

[
  {
    "price": {
      "priceMicros": 11000000,
      "currency": "USD"
    },
    "license": {
      "id": "string",
      "licenseType": "ON_DEMAND_LICENSE",
      "licensePath": "string",
      "permissions": [
        {
          "name": "string"
        }
      ]
    }
  }
]

Note: The priceMicros / price_micros field represents the price in micros (1/1,000,000 of the currency unit). To convert to dollars, divide by 1,000,000. For example, 11000000 micros = 11.00 USD.

Handling Access Restrictions

If you receive a 400 Bad Request error with a message like "access to this page is not allowed", it means the property exists in Tollbit but is not enabled for your organization. In this case, you should reach out to your Tollbit contact to request access to that property.

Example error response:

{
    "detail": "access to this page is not allowed",
    "instance": "/tollbit/dev/v2/rate/https://www.example.com/article",
    "status": 400,
    "title": "Bad Request",
    "type": "about:blank"
}

Check the HTTP status code in your response. A 400 status indicates access restrictions.

Example: Parsing Rates

# Save the response
curl --location 'https://gateway.tollbit.com/tollbit/dev/v2/rate/<url>' \
--header 'TollbitKey: <key>' \
--output rates.json

# Extract price using jq
cat rates.json | jq '.[0].price.priceMicros'  # Get first rate price in micros
cat rates.json | jq '.[0].license.licenseType'  # Get first rate license type

Step 3: Get the Content

To retrieve content, you need to:

Obtain an access token
Use that token to retrieve the content

Note: With the Python SDK, token acquisition is handled automatically by the get_sanctioned_content method. With cURL, you need to obtain the token separately.

Step 3a: Get a Token (cURL only)

First, obtain an access token by specifying:

The URL of the content
Maximum price you're willing to pay (in micros)
Currency
License type
User agent

curl --location 'https://gateway.tollbit.com/dev/v2/tokens/content' \
--header 'Content-Type: application/json' \
--header 'TollbitKey: <key>' \
--data '{
    "url": "https://www.forbes.com/sites/jefffromm/2018/03/27/marketing-to-millennial-new-home-buyers-lowes-innovative-diy-training-program/",
    "userAgent": "test-agent",
    "maxPriceMicros": 11000000,
    "currency": "USD",
    "licenseType": "ON_DEMAND_LICENSE"
}'

Note: For CUSTOM_LICENSE type, you'll also need to include licenseCuid in the request body.

Understanding the Token Response

The token response is a JSON object:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}

Save this token value - you'll need it in the next step.

Step 3b: Get the Content

Use the token obtained in Step 3a to retrieve the actual content:

curl --location 'https://gateway.tollbit.com/dev/v2/content/https://www.forbes.com/sites/jefffromm/2018/03/27/marketing-to-millennial-new-home-buyers-lowes-innovative-diy-training-program/' \
--header 'TollbitToken: <Token>' \
--header 'User-Agent: test-agent'

Note: The URL path in the content endpoint should be the full URL of the content (including protocol).

Content Format Options

You can request different content formats by adding an Accept header:

# Get Markdown format (default)
curl --location 'https://gateway.tollbit.com/dev/v2/content/<url>' \
--header 'TollbitToken: <Token>' \
--header 'User-Agent: test-agent' \
--header 'Accept: text/markdown'

# Get HTML format
curl --location 'https://gateway.tollbit.com/dev/v2/content/<url>' \
--header 'TollbitToken: <Token>' \
--header 'User-Agent: test-agent' \
--header 'Accept: text/html'

Understanding the Content Response

The content response is a JSON object:

{
  "content": {
    "header": "string",
    "body": "string",
    "footer": "string"
  },
  "metadata": {
    "title": "string",
    "description": "string",
    "imageUrl": "string",
    "author": "string",
    "published": "string",
    "modified": "string"
  },
  "rate": {
    "price": {
      "priceMicros": 11000000,
      "currency": "USD"
    },
    "license": {
      "id": "string",
      "licenseType": "ON_DEMAND_LICENSE",
      "licensePath": "string",
      "permissions": [
        {
          "name": "string"
        }
      ]
    }
  }
}

Available License Types

ON_DEMAND_LICENSE: Standard on-demand license
ON_DEMAND_FULL_USE_LICENSE: Full-use on-demand license
CUSTOM_LICENSE: Custom license (requires licenseCuid parameter in token request for cURL, or license_id for Python SDK)

Available Content Formats

Markdown (default): text/markdown for cURL, content_formats.MARKDOWN for Python
HTML: text/html for cURL, content_formats.HTML for Python

Complete Example: End-to-End Workflow

Here's a complete example that ties all the steps together:

#!/bin/bash

# Configuration
API_KEY="<your-api-key>"
USER_AGENT="test-agent"
SEARCH_QUERY="DIY projects for Millenials"

# Step 1: Search for content
echo "Step 1: Searching for content..."
SEARCH_RESPONSE=$(curl -s --location "https://gateway.tollbit.com/dev/v2/search?q=$(echo $SEARCH_QUERY | jq -sRr @uri)" \
  --header "TollBitKey: $API_KEY")

# Extract first result URL (requires jq)
SELECTED_URL=$(echo $SEARCH_RESPONSE | jq -r '.items[0].url')
SELECTED_TITLE=$(echo $SEARCH_RESPONSE | jq -r '.items[0].title')

if [ "$SELECTED_URL" == "null" ] || [ -z "$SELECTED_URL" ]; then
  echo "No results found."
  exit 1
fi

echo "Selected: $SELECTED_TITLE"
echo "URL: $SELECTED_URL"

# Step 2: Get rates
echo ""
echo "Step 2: Getting rates..."
# URL encode the URL for the rate endpoint
ENCODED_URL=$(echo "$SELECTED_URL" | jq -sRr @uri)
RATES_RESPONSE=$(curl -s --location "https://gateway.tollbit.com/tollbit/dev/v2/rate/$ENCODED_URL" \
  --header "TollbitKey: $API_KEY")

# Extract first rate price (requires jq)
MAX_PRICE_MICROS=$(echo $RATES_RESPONSE | jq -r '.[0].price.priceMicros')
LICENSE_TYPE=$(echo $RATES_RESPONSE | jq -r '.[0].license.licenseType')

if [ "$MAX_PRICE_MICROS" == "null" ] || [ -z "$MAX_PRICE_MICROS" ]; then
  echo "No rates available for this content."
  exit 1
fi

echo "Price: $((MAX_PRICE_MICROS / 1000000)).$((MAX_PRICE_MICROS % 1000000 / 10000)) USD"
echo "License Type: $LICENSE_TYPE"

# Step 3a: Get token
echo ""
echo "Step 3a: Getting token..."
TOKEN_RESPONSE=$(curl -s --location 'https://gateway.tollbit.com/dev/v2/tokens/content' \
  --header 'Content-Type: application/json' \
  --header "TollbitKey: $API_KEY" \
  --data "{
    \"url\": \"$SELECTED_URL\",
    \"userAgent\": \"$USER_AGENT\",
    \"maxPriceMicros\": $MAX_PRICE_MICROS,
    \"currency\": \"USD\",
    \"licenseType\": \"$LICENSE_TYPE\"
  }")

TOKEN=$(echo $TOKEN_RESPONSE | jq -r '.token')

if [ "$TOKEN" == "null" ] || [ -z "$TOKEN" ]; then
  echo "Failed to get token."
  exit 1
fi

# Step 3b: Get content
echo ""
echo "Step 3b: Getting content..."
CONTENT_RESPONSE=$(curl -s --location "https://gateway.tollbit.com/dev/v2/content/$ENCODED_URL" \
  --header "TollbitToken: $TOKEN" \
  --header "User-Agent: $USER_AGENT")

# Display content
TITLE=$(echo $CONTENT_RESPONSE | jq -r '.metadata.title')
BODY_PREVIEW=$(echo $CONTENT_RESPONSE | jq -r '.content.body' | head -c 200)

echo ""
echo "Content retrieved successfully!"
echo "Title: $TITLE"
echo ""
echo "Content preview (first 200 chars):"
echo "$BODY_PREVIEW..."

Error Handling

The API returns standard HTTP status codes:

200 OK: Request successful
400 Bad Request: Invalid request (e.g., access not allowed, invalid parameters)
401 Unauthorized: Invalid or missing API key
500-599 Server Error: Server-side errors

Example Error Response

{
    "type": "about:blank",
    "title": "Bad Request",
    "status": 400,
    "detail": "access to this page is not allowed",
    "instance": "/tollbit/dev/v2/rate/https://www.example.com/article"
}

# Check HTTP status code
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  --location 'https://gateway.tollbit.com/dev/v2/search?q=test' \
  --header 'TollBitKey: <key>')

if [ $HTTP_CODE -ne 200 ]; then
  echo "Error: HTTP $HTTP_CODE"
  exit 1
fi

Additional Resources

API Documentation: Visit docs.tollbit.com

Summary

The typical workflow is:

Search → Find content using the search endpoint
Get Rates → Check pricing for a specific result
Get Token (cURL only) → Obtain access token
Get Content → Retrieve the actual content

With Python SDK: The get_sanctioned_content method handles both token acquisition and content retrieval in a single call, simplifying your code.

With cURL: You need to obtain the token separately before requesting content.