Scraping API — User Guide

Endpoint

Generate and manage your API key in your SmartScraping account. Send it in the x-api-key header; do not put it in the request body, URLs, or public browser code. A configured demo may allow requests without this header, but use your own key for regular integrations.

Minimal request

Every request must include:

• url: the HTTPS address of the page to scrape.
• schema: an object describing the fields to extract.

{  "url": "https://example.com/article",  "schema": {    "title": {      "type": "string",      "description": "article title"    }  }}

Descriptions should state what each field represents, such as article title or publication date. Use clear field names and descriptions so the extractor can distinguish related fields, such as a title and subtitle.

Supported field types

Object fields are declared with properties. For arrays:

• If items is omitted, each item is treated as a string.
• To declare the item type explicitly, use items: { "type": "..." }.
• For an array of objects, set items.type to object and declare the fields in items.properties.
• Arrays and objects can be nested, subject to the schema depth limit.

Scalar fields

{  "url": "https://medium.com/david-fernando/por-que-usar-typescript-ca15607eed33",  "schema": {    "title": {      "type": "string",      "description": "article title",      "maxLength": 200    },    "author": {      "type": "string",      "description": "author name"    },    "articleUrl": {      "type": "url",      "description": "article URL"    },    "publishedAt": {      "type": "date",      "description": "publication date",      "format": "dd/mm/yyyy"    }  }}

For an image, use a URL field whose name or description identifies it as an image. The API looks for images in og:image/twitter:image metadata and <img> elements; it does not use the page URL as a substitute:

{  "url": "https://example.com/article",  "schema": {    "image": {      "type": "url",      "description": "article image URL"    }  }}

To specifically request the author's avatar, describe that purpose explicitly, for example, "description": "author profile image URL". An avatar URL is image data and does not replace the author's name in the author field.

Example: article title, image, and author name

The author field should request the name as text. The image field, described as an article image, returns the article's main image, usually obtained from og:image or twitter:image metadata. The avatar does not replace the author's name; request it in a separate field only if needed.

{  "url": "https://medium.com/david-fernando/por-que-usar-typescript-ca15607eed33",  "schema": {    "title": {      "type": "string",      "description": "article title",      "maxLength": 200    },    "image": {      "type": "url",      "description": "article image URL"    },    "author": {      "type": "string",      "description": "author name"    },    "description": {      "type": "string",      "description": "article description"    },    "date": {      "type": "date",      "format": "dd/mm/yyyy"    }  }}

Example response:

{  "scrapingResult": {    "title": "Por que usar TypeScript?",    "image": "https://miro.medium.com/v2/resize:fit:1200/example-cover.jpg",    "author": "David Fernando da Mata Souza",    "description": "Veja quais são os benefícios de utiliza-lo em seus projetos",    "date": "24/09/2020"  }}

When articles appear as cards, declare the image inside each item's properties so it can be associated with the correct article:

{  "url": "https://medium.com/@alex.streza",  "schema": {    "articles": {      "type": "array",      "description": "all articles by the author",      "items": {        "type": "object",        "properties": {          "title": {            "type": "string",            "description": "article title"          },          "image": {            "type": "url",            "description": "article image URL"          }        }      }    }  }}

Lazy-loaded images can only be found when their URLs are present in attributes such as src, data-src, or srcset in the retrieved HTML. Images rendered by scripts, CSS, or APIs that do not appear in the HTML may not be available. If no recognizable image is present, the field may be Data not found.

The explicitly supported date formats are dd/mm/yyyy, dd.mm.yyyy, mm/dd/yyyy, yyyy.mm.dd, yyyy/mm/dd, and yyyy-mm-dd. Without format, the output date uses yyyy-mm-dd. For other date formats, validation depends on the environment's date parser and may be inconsistent; prefer one of the listed formats.

Simple arrays of strings

This schema does not need to declare items; the default is an array of strings.

{  "url": "https://medium.com/david-fernando/por-que-usar-typescript-ca15607eed33",  "schema": {    "tags": {      "type": "array",      "description": "all article tags"    }  }}

You can also declare the item type explicitly:

{  "schema": {    "titles": {      "type": "array",      "description": "all article titles",      "items": {        "type": "string"      }    }  }}

For article URLs, use a field name or description that identifies them:

{  "url": "https://medium.com/@alex.streza",  "schema": {    "articleUrls": {      "type": "array",      "description": "all article URLs",      "items": {        "type": "url"      }    }  }}

On Medium profile pages, structured extraction selects links identified as article URLs, excludes tag/profile links, and removes the source tracking parameter. The number of results depends on the articles present in the page content that was loaded.

Arrays of objects

Use an array of objects to get multiple records with related fields:

{  "url": "https://medium.com/@alex.streza",  "schema": {    "articles": {      "type": "array",      "description": "all articles published by the author",      "items": {        "type": "object",        "properties": {          "title": {            "type": "string",            "description": "article title"          },          "url": {            "type": "url",            "description": "article URL"          },          "date": {            "type": "date",            "description": "publication date",            "format": "yyyy.mm.dd"          },          "description": {            "type": "string",            "description": "article description",            "maxLength": 500          }        }      }    }  }}

In Medium article cards, the first heading is treated as the title and a recognized subtitle is treated as the description. When the page provides a description marker, it takes priority over fallbacks. If a field is not present on the page, its value may be Data not found.

Requesting multiple fields

A schema can combine arrays, objects, and scalar fields:

{  "url": "https://medium.com/@alex.streza",  "schema": {    "profileName": {      "type": "string",      "description": "author name"    },    "articles": {      "type": "array",      "description": "all articles published by the author",      "items": {        "type": "object",        "properties": {          "title": {            "type": "string",            "description": "article title"          },          "url": {            "type": "url",            "description": "article URL"          }        }      }    }  }}

Schema limits and validation

• The schema must be a non-empty JSON object.
• Each object can contain at most 20 fields.
• The maximum nesting depth is 5.
• Field names must be 1–40 characters and contain only ASCII letters, numbers, or _.
• description must be text with at most 80 characters and no line breaks. It can be omitted.
• When provided, maxLength must be an integer between 1 and 1000.
• object requires non-empty properties.
• An array without items is interpreted as an array of strings.
• The request url must use HTTPS and point to a safe public host. localhost, direct IP addresses, and local/internal hosts are not accepted.
• Fields required by response validation must be present. When an individual value cannot be found, the API may return Data not found; arrays with no identified items may be empty.

An invalid schema returns HTTP 400 with an error property explaining the validation failure.

Complete request examples

curl -X POST "https://smartscraping.dev/api/scrape" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://medium.com/@alex.streza",
    "schema": {
      "articles": {
        "type": "array",
        "description": "all articles published by the author",
        "items": {
          "type": "object",
          "properties": {
            "title": {
              "type": "string",
              "description": "article title"
            },
            "url": {
              "type": "url",
              "description": "article URL"
            }
          }
        }
      }
    }
  }'

Keep the API key in a server-side environment variable. Do not expose a private key in frontend code distributed to browsers.

Top-level scalar fields and simple arrays can be combined with an array of objects in the same request. For example, you can request profileName, tags, and articles together; the response keeps each property at the top level of scrapingResult.

Response

A successful response uses HTTP 200, with the requested fields inside scrapingResult:

{  "scrapingResult": {    "articles": [      {        "title": "AI-powered Development — The Good, The Bad and The Ugly",        "url": "https://medium.com/@alex.streza/ai-powered-development-the-good-the-bad-and-the-ugly-3bf66f09df64",        "date": "2023.05.29",        "description": "An article about AI that’s not written by AI"      }    ]  }}

Property names match those defined in the schema. Values that cannot be found may be Data not found for individual fields. Arrays contain the items found; an array with no matching items may be [].

Optional extraction metadata

Add options.includeMetadata: true to receive per-field diagnostics without changing the contents or types of scrapingResult. Keep it omitted when you only need extracted data.

{  "url": "https://example.com/product",  "schema": {    "price": { "type": "string", "description": "product price"},    "inventory": { "type": "string", "description": "stock availability"}  },  "options": { "includeMetadata": true }}

{  "scrapingResult": {    "price": "R$ 99.90",    "inventory": "Data not found"  },  "extractionMetadata": {    "source": "static",    "method": "structured",    "fields": {      "price": {        "status": "found",        "snippet": "Special price: R$ 99.90 today"      },      "inventory": { "status": "not_found"}    }  }}

found means a value was extracted; a short snippet is included when it can be located in the processed text. not_found means no value was found in the loaded page content. unknown means extraction could not determine a valid result and includes a reason. The metadata applies to each top-level schema field. source is static, puppeteer, or unknown; method is structured, model, or unknown.

With diagnostics enabled, recognized unavailable pages return an HTTP error and mark every requested field as unknown. Page reasons include page_not_found, access_blocked, authentication_required, and upstream_error; extraction may also report no_valid_extraction_result or source_unavailable. Common HTML error pages that return HTTP 200, such as “This page doesn't exist” and “Site not found,” are detected as unavailable pages.

Access, plans, and usage limits

• The API key identifies your account and applicable plan.
• The Free plan allows up to 100 scrape requests per month.
• The Pro plan provides unlimited requests according to the current service configuration and enables browser fallback for dynamic (SPA) or protected pages.
• There is also a traffic limit of up to 10 requests per second per user. Exceeding it may result in HTTP 429; wait and try again later.
• Browser fallback does not guarantee access to every site or bypass every protection mechanism.

Dynamic pages and result coverage

Each request first retrieves the page's HTML and tries structured extraction, page-field extraction, and then model-based extraction. On Pro, requests without a recognized page issue can fall back to browser rendering when static retrieval fails or yields no usable value. When extraction metadata is requested, recognized page issues are returned directly instead of triggering browser fallback. Free-plan requests cannot use this fallback.

The API extracts content made available by the loaded page; browser rendering does not guarantee access to every site or bypass its protections. It does not follow pagination, click “Load more,” crawl every page on a site, or access content that requires login. On infinitely scrolling pages, results depend on the items present in the HTML available to the request.

For large result sets, prefer a focused array schema with precise descriptions. Extraction can be deterministic when the page has a recognized structure; otherwise, it may use the model as a fallback, which has a limited call budget. Therefore, do not assume a schema guarantees that every site or layout will produce every field.

For arrays of recognized fields, use names and descriptions that identify the data type. For example, titles and all article titles are recognized as a request for titles, while articleUrls and all article URLs identify URLs. Deterministic extraction uses the first heading in each article as its title and excludes the subtitle.

Common response codes

Errors are returned as JSON, for example:

{  "error": "Invalid schema: Schema must have at least one field."}

Best practices

1. Declare only the fields you need.
2. Use short, specific descriptions, especially for similar fields.
3. Use type: "array" for collections and items to declare the item type; omit items only when you want a list of strings.
4. Use arrays of objects to keep each article's title, URL, date, and description associated.
5. Do not confuse a list of collected links with full-site crawling: the API works on the specified page.
6. Handle Data not found, empty arrays, and HTTP error responses in your client.
7. Keep your API key secret and make requests from a backend.