Resources/Search Engines

Search Engine Indexing & Submission APIs: The Complete Reference

Everything developers and AI agents need to know about automated URL submission protocols, rate limits, authentication methods, and content restrictions across major search engines worldwide.

Federated IndexNow
Bing Webmaster API
Google Indexing API Policy
Baidu Active Push

Submission Methods Matrix

Side-by-side comparison of protocols, daily throughput limits, and IndexFast automation status.

Search EnginePrimary MethodBatch SizeDaily QuotaCrawl LatencyIndexFast Status
Microsoft BingMicrosoftIndexNow Protocol10,000 URLs10,000+ URLs/dayInstant (minutes to crawl) Native
GoogleAlphabetGoogle Indexing API (Strictly Gated)100 URLs200 quota units/day default (JobPosting/BroadcastEvent only)Fast (crawls within minutes for eligible types)Gated (Jobs/Live)
YandexYandexIndexNow (Recommended)10,000 URLs10,000 URLs/dayVery fast (crawls within minutes)IndexNow
BaiduBaidu, Inc.Baidu Active Push API (主动推送)2,000 URLsUp to 50,000 URLs/day (level dependent)Fast (fastest crawl speed for Baiduspider)Planned
NaverNaver CorporationIndexNow (Federated Partner)10,000 URLs10,000 URLs/dayFast (hours to 1 day)IndexNow
Seznam.czSeznam.cz, a.s.IndexNow (Official Partner)10,000 URLs10,000 URLs/dayWithin hoursIndexNow
DuckDuckGo, Yahoo & EcosiaSyndicated Search EnginesSyndicated Discovery via Bing & IndexNow10,000 URLsCovered by Bing / IndexNow quotaIndexed when Bing updates its indexSyndicated

Search Engine API Specifications

Technical architecture, authentication methods, payload specifications, and cURL snippets.

Microsoft Bing

Global: ~3.5% | US Desktop: ~13% | Powering Yahoo, DuckDuckGo, Ecosia

Microsoft Bing provides both the open IndexNow protocol for instant push and a dedicated Webmaster API for direct quota-based batch submissions. Discovery on Bing also feeds DuckDuckGo, Yahoo Search, and Ecosia.

Official Docs

Available Submission Methods

IndexNow ProtocolPOSTAuth: indexnow_key
Batch: 10000 URLs•Quota: 10,000+ URLs/day

None (any valid page hosted on the verified domain).

Endpoint:

https://api.indexnow.org/indexnow

cURL Example:

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "your-indexnow-key",
    "keyLocation": "https://example.com/your-indexnow-key.txt",
    "urlList": [
      "https://example.com/new-page-1",
      "https://example.com/updated-page-2"
    ]
  }'

Created jointly by Microsoft and Yandex. Pinging one IndexNow endpoint shares notifications across all participating search engines.

Bing Webmaster API (SubmitUrlBatch)POSTAuth: api_key
Batch: 500 URLs•Quota: 10,000 - 100,000 URLs/day (based on site authority)

Site must be registered and verified in Bing Webmaster Tools.

Endpoint:

https://ssl.bing.com/webmaster/api.svc/json/SubmitUrlBatch?apikey={apiKey}

cURL Example:

curl -X POST "https://ssl.bing.com/webmaster/api.svc/json/SubmitUrlBatch?apikey=YOUR_BING_API_KEY" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "siteUrl": "https://example.com",
    "urlList": [
      "https://example.com/page-1",
      "https://example.com/page-2"
    ]
  }'

Quota can be queried via GetUrlSubmissionQuota endpoint. IndexFast supports automated submissions through both IndexNow and Bing Webmaster API.

How to connect Bing in IndexFast

  1. Add your site to IndexFast by entering your domain.
  2. Deploy the generated IndexNow key file to your root (e.g. /public/<key>.txt) and click Verify.
  3. In Bing Webmaster Tools (bing.com/webmasters), verify your domain ownership.
  4. Go to Bing Webmaster Tools > Settings (gear icon) > API Access > API Key, copy your key.
  5. Paste the API key into your IndexFast site Settings under 'Bing Webmaster API' with auto-submit enabled.

Developer Tips & Gotchas

  • •Using both IndexNow and Bing Webmaster API provides maximum delivery redundancy.
  • •Bing Webmaster API requires the exact protocol and host (including or excluding www) matching your Bing Webmaster property.
  • •Bing powers DuckDuckGo, Yahoo, and Ecosia, so submitting to Bing covers multiple search engines in one call.

Google

Global: ~90% | Mobile: ~94%

Google does NOT offer an open URL submission API for standard content. The Google Indexing API is strictly restricted by policy to JobPosting and BroadcastEvent (live video) schema. Google deprecated sitemap pinging in December 2023. The URL Inspection API is read-only.

Official Docs

Available Submission Methods

Google Indexing API (Strictly Gated)POSTAuth: oauth2
Batch: 100 URLs•Quota: 200 quota units/day default (JobPosting/BroadcastEvent only)

CRITICAL: Strictly permitted ONLY for pages containing JobPosting or BroadcastEvent structured data. Submitting standard pages violates Google policies.

Endpoint:

https://indexing.googleapis.com/v3/urlNotifications:publish

cURL Example:

curl -X POST "https://indexing.googleapis.com/v3/urlNotifications:publish" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/jobs/senior-engineer",
    "type": "URL_UPDATED"
  }'

IndexFast enforces safe guardrails for Google Indexing API to protect domains from API quota revocation or algorithmic penalties.

Google Search Console URL Inspection APIPOSTAuth: oauth2
Batch: 1 URLs•Quota: 2,000 requests/day per Search Console property

None. Provides indexing status, canonical URL, and mobile usability.

Endpoint:

https://searchconsole.googleapis.com/v1/urlInspection/index:inspect

cURL Example:

curl -X POST "https://searchconsole.googleapis.com/v1/urlInspection/index:inspect" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{
    "inspectionUrl": "https://example.com/about",
    "siteUrl": "sc-domain:example.com"
  }'

Useful for checking whether Google has indexed a URL and verifying user-declared canonical tags.

How to handle Google Discovery with IndexFast

  1. Ensure XML sitemaps are referenced in robots.txt and submitted in Google Search Console.
  2. Ensure pages return 200 HTTP status, valid canonical link tags, and proper schema markup.
  3. If your site publishes job listings or livestream broadcasts, you can connect a GCP Service Account with Google Indexing API enabled.
  4. For general pages, discovery occurs organically via sitemaps and inbound web links.

Developer Tips & Gotchas

  • •Beware of services claiming 'instant Google indexing for all pages' via the Indexing API; Google actively revokes quotas for abusive usage.
  • •Google permanently turned off 'google.com/ping?sitemap=...' in December 2023. Keep sitemaps updated on your server instead.
  • •Googlebot crawls sitemaps frequently when lastmod dates are updated accurately.

Yandex

Russia & CIS: ~65% | Central Asia: ~30-50%

Yandex is a co-founder and primary operator of the IndexNow protocol. Submissions sent via IndexNow to api.indexnow.org or yandex.com/indexnow are directly ingested by YandexBot. Yandex also provides a Webmaster Recrawl API.

Official Docs

Available Submission Methods

IndexNow (Recommended)POSTAuth: indexnow_key
Batch: 10000 URLs•Quota: 10,000 URLs/day

Endpoint:

https://yandex.com/indexnow

cURL Example:

curl -X POST "https://yandex.com/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "your-indexnow-key",
    "keyLocation": "https://example.com/your-indexnow-key.txt",
    "urlList": ["https://example.com/page-1"]
  }'

Standard IndexNow endpoint; submissions through IndexFast reach Yandex automatically.

Yandex.Webmaster API (Recrawl Queue)POSTAuth: oauth2
Batch: 1 URLs•Quota: 10 - 100 URLs/day (limited based on SQI score)

Requires Yandex OAuth 2.0 app and low quota.

Endpoint:

https://api.webmaster.yandex.net/v4/user/{userId}/hosts/{hostId}/recrawl/queue

cURL Example:

curl -X POST "https://api.webmaster.yandex.net/v4/user/USER_ID/hosts/HOST_ID/recrawl/queue" \
  -H "Authorization: OAuth YOUR_OAUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/page-1"}'

Due to small daily quotas (often just 20 URLs/day), IndexNow is much better suited for automated indexing.

How to connect Yandex in IndexFast

  1. Add your site in IndexFast and verify your IndexNow key.
  2. All verified IndexFast sites automatically submit URLs to Yandex through the federated IndexNow protocol.
  3. Optionally add your site in Yandex.Webmaster (webmaster.yandex.com) to track Russian and CIS search analytics.

Developer Tips & Gotchas

  • •IndexNow is the official recommendation by Yandex for rapid content indexing.
  • •Yandex strongly penalizes slow mobile pages; ensure mobile TTFB is under 400ms.

Baidu

China: ~65-70% | Global Chinese diaspora

Baidu operates the 'Active Push' (主动推送) API through the Baidu Search Resource Platform (百度搜索资源平台). It allows webmasters to submit up to 50,000 URLs daily via a direct HTTP POST with a site push token.

Official Docs

Available Submission Methods

Baidu Active Push API (主动推送)POSTAuth: token
Batch: 2000 URLs•Quota: Up to 50,000 URLs/day (level dependent)

Only URLs belonging to the registered domain. Requires ICP license for mainland Chinese hosting.

Endpoint:

http://data.zz.baidu.com/urls?site={domain}&token={token}

cURL Example:

curl -H 'Content-Type:text/plain' --data-binary $'https://example.com/page-1\nhttps://example.com/page-2' \
  "http://data.zz.baidu.com/urls?site=https://example.com&token=YOUR_BAIDU_TOKEN"

Returns JSON response: {"remain":49990,"success":10}. Baidu does not currently participate in IndexNow.

How to set up Baidu Active Push

  1. Register on Baidu Search Resource Platform (ziyuan.baidu.com).
  2. Add and verify your website domain via HTML file or DNS TXT record.
  3. Go to Resource Submission (资源提交) > Ordinary Ingestion (普通收录) > API Submission (API提交).
  4. Copy your site push token (16-character alphanumeric string).
  5. Execute direct HTTP POST with plain text newline-delimited URLs to the push endpoint.

Developer Tips & Gotchas

  • •Baidu does not support IndexNow; direct HTTP POST to their active push endpoint is mandatory for real-time discovery.
  • •Baiduspider prioritizes sites with Chinese mainland ICP filing and fast domestic CDN routes.
  • •Use GB2312 or UTF-8 character encoding cleanly to avoid crawl parsing errors on Baidu.

Seznam.cz

Czech Republic: ~10-15%

Seznam.cz is an independent European search engine in the Czech Republic. Seznam officially integrated with IndexNow in 2022 to provide real-time discovery for Czech and European websites.

Official Docs

Available Submission Methods

IndexNow (Official Partner)POSTAuth: indexnow_key
Batch: 10000 URLs•Quota: 10,000 URLs/day

Endpoint:

https://indexnow.seznam.cz/indexnow

cURL Example:

curl -X POST "https://indexnow.seznam.cz/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "your-indexnow-key",
    "keyLocation": "https://example.com/your-indexnow-key.txt",
    "urlList": ["https://example.com/cz/stranka"]
  }'

Submissions to api.indexnow.org or indexnow.seznam.cz are shared with SeznamBot.

How to connect Seznam in IndexFast

  1. IndexFast sites with verified IndexNow keys automatically notify SeznamBot.
  2. Register in Seznam Webmaster (webmaster.seznam.cz) to monitor crawl health in the Czech market.

Developer Tips & Gotchas

  • •SeznamBot heavily values schema.org microdata and standard HTML sitemaps.

DuckDuckGo, Yahoo & Ecosia

Combined Global: ~5% | Privacy-focused users

DuckDuckGo, Yahoo Search, and Ecosia do not operate separate large-scale web crawlers; their search indexes are primarily powered by Microsoft Bing. Pushing URLs to Bing via IndexNow or the Bing Webmaster API immediately makes content discoverable across these engines.

Official Docs

Available Submission Methods

Syndicated Discovery via Bing & IndexNowPOSTAuth: indexnow_key
Batch: 10000 URLs•Quota: Covered by Bing / IndexNow quota

Inherits Bing policies.

Endpoint:

https://api.indexnow.org/indexnow

cURL Example:

# Submitting to IndexNow or Bing automatically covers DuckDuckGo, Yahoo, and Ecosia:
curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json" \
  -d '{"host": "example.com", "key": "KEY", "urlList": ["https://example.com/page"]}'

No separate API keys or registration required. Bing is the underlying data provider.

How to reach DuckDuckGo and Yahoo

  1. Enable IndexFast for your domain.
  2. Verify your IndexNow key and optionally configure your Bing Webmaster API key.
  3. Once Bing indexes your URLs, they appear automatically in DuckDuckGo and Yahoo search results.

Developer Tips & Gotchas

  • •DuckDuckGo uses Wikipedia, Apple Maps, and Bing index results. Clean structured data and fast SSL are critical.

Deep Dive

Automating Bing Search Console with IndexFast

IndexFast provides first-class support for Microsoft Bing. Every connected site automatically uses IndexNow to push changes. In addition, developers can supply their own Bing Webmaster API key to execute direct, authenticated batch submissions using their site's Bing daily quota.

1

Dashboard (Zero Code)

Navigate to your site in the IndexFast Dashboard → Settings → Bing Webmaster API. Paste your key and click “Connect & Verify”. Enable the auto-submit checkbox.

2

REST API

Send a POST request to /api/v1/sites/:siteId/bing with your API key. IndexFast encrypts and validates the credential instantly.

curl -X POST \
  https://indexfast.co/api/v1/sites/example.com/bing \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"apiKey": "BING_KEY", "autoSubmit": true}'
3

AI IDEs via MCP

Inside Cursor or Claude Desktop, your AI agent can manage Bing automation directly using:

call_tool("set_bing_api_key", { site: "example.com", apiKey: "..." })

Start automating your search engine discovery today.

Add your website, place one verification key file, and let IndexFast handle IndexNow, Bing Webmaster API, and discovery monitoring.

Open DashboardExplore All Resources