How to Manage Web Sources via the API

In this article:

• Overview
• Before you start
• Add web sources
• Retrieve web sources
• Delete a web source
• Validation and errors
• Rate limits

Overview

If your institution maintains a large list of web sources, or already keeps that list in another system, you can manage web sources through the Originality API instead of adding each one by hand on the Web Sources page.

Through the API you can:

• Add web sources, one at a time or many in a single request
• Retrieve the web sources you have added
• Delete web sources you no longer need

Web sources added through the API appear on the Web Sources page with API integration in the Added by column. They follow the same requirements as sources added in the interface (see Web Sources → Requirements).

Before you start

You need an API client for your Originality license. License administrators create API clients in Originality under License settings → API. Each client has a client ID and a client secret. Store the secret securely.

Add web sources

Send the sources as a list. You can submit one or many in the same request.

{
  "sources": [
    {
      "description": "Faculty publication index",
      "url": "https://example.com/publications"
    }
  ]
}

What happens after submission

Sources are queued and crawled by a periodic job, so they are not crawled immediately. Only the page at each submitted URL is fetched. Originality does not follow links from it, either to other pages on the same site or to other sites.

Submission limits

Per-source results

An invalid URL does not fail the whole submission. The response reports each source individually, matched to its position (index) in the request:

Originality normalizes URLs before comparing and storing them:

• Fragments (#...) are removed.
• Host names are converted to lowercase.
• URLs without a scheme get https://.

The url in each result is the normalized form. Use it when matching results back to your own records.

URL requirements

A URL is rejected if it:

• does not use http or https
• does not resolve to a named host with a top-level domain
• contains embedded credentials (for example, https://user:password@example.com)
• points to localhost or to an IP address

Retrieve web sources

Returns the web sources registered for your license, newest first.

Query parameters

Filtering by URL

The url filter matches any part of the URL and ignores letter case, so Example.COM and example.com return the same sources.

The filter is matched against the normalized URL stored when the source was added, not the value as it was submitted. For example, a source submitted as HTTPS://Example.com/Papers#intro is stored as https://example.com/Papers. Searching for example.com/Papers finds it; searching for #intro does not.

The text is matched literally:

• %, _ and \ have no special meaning, so you can search for percent-encoded characters such as %20 directly.
• Wildcards are not supported.
• Leading and trailing spaces are removed, and an empty value returns all sources.

Paging through results

Use limit and offset to page through your sources. If hasMore is true, increase offset by limit and request the next page.

Example response

{
  "hasMore": true,
  "sources": [
    {
      "crawledAt": "2026-09-29T13:36:48.971Z",
      "createdAt": "2026-09-29T13:36:48.971Z",
      "description": "Faculty publication index",
      "id": 4711,
      "url": "https://example.com/publications"
    }
  ]
}

Delete a web source

Replace {id} with the ID of the web source. You'll find the ID in the results when you add or retrieve sources.

Deleting a web source stops it from being crawled again. Content already indexed from earlier crawls is not removed, so submissions can still match it.

Validation and errors

The API reports problems at two levels.

Per source (when adding web sources)

An invalid URL doesn't fail the whole request. The request returns 200, and that source is reported as rejected, with an error explaining why. A URL is rejected if it:

• does not use http or https
• does not resolve to a named host with a top-level domain
• contains embedded credentials (for example https://user:password@example.com)
• points to localhost or to an IP address

Always check the per-source results, even when the request returns 200.

Per request

When a whole request fails, the response body describes the error:

{
  "title": "Unauthorized",
  "status": 401,
  "detail": "Unauthorized"
}

Rate limits

Each API key is limited to:

• 200 requests per second
• 5,000 requests per minute

Whichever limit is reached first triggers rate limiting, and further requests return 429 until the window resets. If you receive a 429, pause before retrying rather than retrying immediately.

To stay well within the limits:

• Adding sources: submit them in bulk. One request can hold up to 20,000 sources.
• Retrieving sources: use a larger limit to reduce the number of pages.