Skip to main content

Add Index

POST /api/v1/developer/add-index Add a new index with a source URL for indexing SDK, CLI, or API documentation. This endpoint creates a new index and starts the indexing process automatically. Once indexed, you can use the get-context endpoint to retrieve relevant documentation.

Request

Headers

API-Key: your-api-key
Content-Type: application/json

Body Parameters

index_name
string
required
Name or title of the index to create. This should be a descriptive name for the service you’re indexing (e.g., “Stripe API”, “AWS CLI”, “Python Requests Library”).
url
string
required
Source URL to index. This should be the main documentation page or starting point for the service. The system will crawl and index related pages automatically.
notes
string
Optional notes about the index. Use this to add context about what this index contains or any special considerations.

Example Request

curl -X POST \
  "https://www.goharvest.ai/api/v1/developer/add-index" \
  -H "API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "index_name": "Stripe API Documentation",
    "url": "https://stripe.com/docs/api",
    "notes": "Official Stripe API documentation for payment processing"
  }'

Response

Success Response (200)

{
  "ok": true,
  "error": null,
  "data": {
    "index_id": "scrape_abc123",
    "index_name": "Stripe API Documentation",
    "url": "https://stripe.com/docs/api",
    "status": "PENDING",
    "message": "Index created successfully and indexing has started"
  }
}
ok
boolean
Indicates if the request was successful
error
string | null
Error message if the request failed, null otherwise
data
object
Contains the response data

Index Status

After creating an index, it goes through several states:
The index has been created and is queued for processing. Indexing will begin shortly.
The system is actively crawling and indexing the documentation from the provided URL.
The index is complete and ready to use. You can now query it with the get-context endpoint.
The indexing process encountered an error. Check that the URL is valid and accessible.

Error Responses

{
  "detail": "Invalid URL provided",
  "status_code": 400
}
The URL format is invalid or the URL is not accessible.
{
  "detail": "Active subscription required to create indexes",
  "status_code": 403
}
You need an active subscription to create new indexes. Visit your account settings to manage subscriptions.
{
  "detail": "This URL is already indexed",
  "status_code": 409
}
An index already exists for this URL. Use list-indexes to find the existing index.
{
  "detail": "Indexing is already in progress for this URL",
  "status_code": 409
}
This URL is currently being indexed. Wait for the process to complete before attempting to add it again.
{
  "detail": "API key missing",
  "status_code": 401
}
The API key is missing from request.
{
  "detail": "Invalid API key",
  "status_code": 401
}
The API key provided is invalid.
{
  "detail": "User account not found for user_account_id <user_account_id>",
  "status_code": 404
}
The user account associated with the API key doesn’t exist.

Best Practices

Choose the Right Starting URL: Select a documentation home page or API reference as your starting URL. The system will automatically discover and index related pages.
Descriptive Names: Use clear, descriptive names for your indexes to make them easy to identify when using list-indexes.
Indexing Time: The indexing process can take several minutes to hours depending on the size of the documentation. Monitor the status using list-indexes.
Subscription Limits: The number of indexes you can create depends on your subscription plan. Check your account settings for details.

Common Use Cases

Internal Documentation

Index your company’s internal API documentation and developer guides

Third-Party APIs

Add frequently-used third-party service documentation for quick reference

Open Source Libraries

Index open source library documentation for your development stack

CLI Tools

Create indexes for command-line tools and their documentation

Workflow Example

Here’s a typical workflow for adding and using a new index:
  1. Add Index: Call add-index with your documentation URL
  2. Wait for Completion: Indexing begins automatically (status: PENDINGIN_PROGRESS)
  3. Verify: Use list-indexes to check when status changes to INDEXED
  4. Query: Once indexed, use get-context to retrieve relevant documentation
Example Workflow
import requests
import time

api_key = "your-api-key"
base_url = "https://www.goharvest.ai/api/v1/developer"

# Step 1: Add the index
add_response = requests.post(
    f"{base_url}/add-index",
    headers={"API-Key": api_key, "Content-Type": "application/json"},
    json={
        "index_name": "FastAPI Docs",
        "url": "https://fastapi.tiangolo.com"
    }
)
index_id = add_response.json()['data']['index_id']
print(f"Created index: {index_id}")

# Step 2: Wait for indexing to complete
print("Waiting for indexing to complete...")
while True:
    list_response = requests.get(
        f"{base_url}/list-indexes",
        headers={"API-Key": api_key}
    )
    indexes = list_response.json()['data']['indexes']
    index = next((i for i in indexes if i['index_id'] == index_id), None)

    if index and index['status'] == 'INDEXED':
        print("✅ Indexing complete!")
        break

    time.sleep(30)  # Check every 30 seconds

# Step 3: Query the index
context_response = requests.post(
    f"{base_url}/get-context",
    headers={"API-Key": api_key, "Content-Type": "application/json"},
    json={
        "query": "how to create a path parameter",
        "index_id": index_id
    }
)
context = context_response.json()['data']['context']
print(f"Retrieved context:\n{context}")

Next Steps

I