Tokens

Tokens are one-time access grants tied to specific pages and user agents. The token system allows authenticated access for retrieving content, with a one-time use model to ensure security.

Specification

TollBit tokens are JWTs used to authorize a specific content request. They include authentication details, request intent, and, when applicable, pricing and license terms. TollBit generates and validates tokens automatically; this section explains the token claims for reference. Tokens expire 5 minutes after issuance.

TollBit has two tokens: content and crawl, which map to Content Intents.

Indexing tokens (sometimes referred to as crawl tokens) allow bots to retrieve content for indexing and performing analysis on the content.

Content Retrieval tokens allow bots to retrieve content to display to users. They contain licensing information and payment details.

The tokens are generated by TollBit's servers so you will never create these manually.

Common Claims (All Token Types)

Content Token (tt = content)

Crawl Token (tt = crawl)

No additional claims beyond the common claims.

SDKs

The SDK handles token generation automatically when making API requests. Consequently, there is no documentation on generating tokens within the SDK.

Generate Content Token

Generate a valid token for sanctioned, paid access to a specific page, with the terms of the license type or CUID that you include in the request payload.

Endpoint

POST /dev/v2/tokens/content

Base URL: https://gateway.tollbit.com

Authentication: Requires API key via the TollbitKey header.

Request headers

• TollbitKey (string, required)
  • Your API key, available in the access page of the dashboard.

Request body (JSON)

• url (string, required)

  • The specific page URL for the intended content to access.

• userAgent (string, required)

  • Unique AgentID string for who is allowed to access the page. Use the AgentID that you will use within your application. You must use a user agent you registered in the developer portal.

• maxPriceMicros (integer, required)

  • The maximum price, in micro units, that you are willing to pay for this piece of content. Leaving this unset defaults the field to 0, which only allows retrieval of free content. $1 = 1,000,000 micros.

• currency (string, required)

  • The three-letter currency code. Only USD is supported at the moment.

• licenseType (string, required)

  • The type of license you are requesting. Supported values: ON_DEMAND_LICENSE, ON_DEMAND_FULL_USE_LICENSE, CUSTOM_LICENSE.

• licenseCuid (string, optional)

  • Required only for CUSTOM_LICENSE types.

Response

• token (string)
  • The JWT token that can be used to make a content request.

Example request

curl https://gateway.tollbit.com/dev/v2/tokens/content \
  -H "Content-Type: application/json" \
  -H "TollbitKey: YOUR_API_KEY_HERE" \
  -d '{"url": "https://example.com/article", "userAgent": "my-agent/1.0", "maxPriceMicros": 11000000, "currency": "USD", "licenseType": "ON_DEMAND_LICENSE"}'

Example response (success)

{
  "token": "eyJhbGciOiJFUzI1..."
}

Example response (error)

{
  "detail": "The license...",
  "instance": "/dev/v2/tokens/content",
  "status": 400,
  "title": "Bad Request",
  "type": "about:blank"
}

Generate Indexing Token

Generate a valid token for a specific page for your user agent to fetch content for indexing purposes. Note that while you may be able to generate the token, the data provider has controls to allow or disallow access through this method of fetching content.

Endpoint

POST /dev/v2/tokens/crawl

Base URL: https://gateway.tollbit.com

Authentication: Requires API key via the TollbitKey header.

Request headers

• TollbitKey (string, required)
  • Your API key, available in the access page of the dashboard.

Request body (JSON)

• url (string, required)

  • The specific page URL for the intended content to access.

• userAgent (string, required)

  • Unique AgentID string for who is allowed to access the page. Use the AgentID that you will use within your application. You must use a user agent you registered in the developer portal.

Response

• token (string)
  • The JWT token that can be used to make a content request (e.g. for index content).

Example request

curl https://gateway.tollbit.com/dev/v2/tokens/crawl \
  -H "Content-Type: application/json" \
  -H "TollbitKey: YOUR_API_KEY_HERE" \
  -d '{"url": "https://example.com/article", "userAgent": "my-agent/1.0"}'

Example response (success)

{
  "token": "eyJhbGciOiJFUzI1..."
}

Example response (error)

{
  "detail": "The license...",
  "instance": "/dev/v2/tokens/crawl",
  "status": 400,
  "title": "Bad Request",
  "type": "about:blank"
}

Error responses

• 400 Bad Request: Invalid request (e.g. license not available, invalid parameters).
• 401 Unauthorized: Missing or invalid TollbitKey header.

Error responses follow the standard format with detail, instance, status, title, and type.