Documentation

1. Generate Strategy

Provide a URL and describe what you want to extract. meter's AI analyzes the page structure and automatically creates an extraction strategy. This is a one-time setup that typically takes 10-30 seconds.

The system intelligently identifies the best way to extract your data, handling edge cases automatically.

2. Review & Refine

Review the preview data to see what was extracted. If something is missing or incorrect, provide feedback and the strategy will be refined automatically—no need to re-scrape the page.

The system automatically handles common issues like invalid selectors, missing containers, and variant class names.

3. Execute Scrapes

Once you have a strategy, you can run scrapes on-demand or set up automatic scheduling. All scrapes use the saved strategy with raw scraping—no AI costs, instant results.

How it works: meter uses an LLM once during strategy generation to analyze the page and create extraction rules. After that, all scrapes use fast, consistent extraction instead of LLM calls. This gives you the best of both worlds: AI-powered setup with the speed, consistency, and cost-efficiency of traditional scraping.

Each scrape extracts the data and automatically detects changes compared to previous runs, so you only process what's new.

4. Monitor Changes

meter automatically detects when content changes, ignoring noise like ads, timestamps, and layout updates. You only get notified about meaningful changes.

You can compare any two scrapes to see what changed. For automated workflows, set up webhooks when creating schedules to receive scrape results via HTTP POST when jobs complete.

Strategies

A strategy is a saved extraction configuration that defines how to extract data from a website. meter automatically figures out how to extract your data based on your description.

Strategies are generated once with LLMs, then reused for all future scrapes at $0 cost.

Jobs

A job is a single execution of a strategy against a URL. Jobs are processed asynchronously and include:

• Extracted results (array of items)
• Change detection information
• Item count and metadata

Schedules

A schedule defines recurring scrapes. Supports:

• Interval-based: Run every N seconds (e.g., every hour = 3600 seconds)
• Cron-based: Use cron expressions (e.g., 0 9 * * * for daily at 9 AM)
• Webhooks: Optionally receive scrape results via HTTP POST to your webhook URL when jobs complete

Schedules can be enabled/disabled and updated without deleting them. When a scheduled scrape completes, you can receive the results via webhook for real-time processing.

Change Detection

meter automatically detects when content changes by comparing each scrape to previous runs. It intelligently identifies meaningful changes while ignoring noise like ads, timestamps, and layout updates.

The API provides endpoints to compare any two jobs and see what changed.

Step 1: Go to Dashboard

Navigate to your dashboard and click "New Strategy" or "New Job".

Step 2: Enter URL and Description

URL: https://news.ycombinator.com

Description: "Extract post titles and scores"

Step 3: Generate Strategy

Click "Generate" and wait 10-30 seconds. meter's AI will analyze the page and create an extraction strategy.

Step 4: Review Preview

Review the preview data to see what was extracted. If it looks good, click "Run Job" to execute a scrape.

Step 5: See Your Results

View your results in JSON format. You've successfully completed your first scrape!

Two ways to use meter:

1. Sign up at /login using email or OAuth (Google/GitHub)
2. Click "New Strategy" or "New Job" in the dashboard
3. Paste a URL and describe what to extract (e.g., "Extract product names and prices")
4. Review results → Click "Run Job" → Done

Takes 2 minutes, no coding required.

1. Create an Account & Get API Key

Sign up at /login. API keys are automatically generated when you sign in. You can view your API key in the dashboard.

2. Initialize the SDK

from meter_sdk import MeterClient

client = MeterClient(api_key="sk_live_...")

3. Generate Your First Strategy

result = client.generate_strategy(

 url="https://example.com/products",

 description="Extract product names and prices",

 name="Product Scraper"

)

strategy_id = result["strategy_id"]

print(result["preview_data"])

4. Refine if Needed

If the preview data looks incomplete, refine the strategy with feedback:

refined = client.refine_strategy(

 strategy_id=strategy_id,

 feedback="Missing the first product on the page"

)

5. Execute a Scrape Job

Create a job to execute the scrape (no LLM costs!):

job = client.create_job(

 strategy_id=strategy_id,

 url="https://example.com/products"

)

# Wait for completion

completed = client.wait_for_job(job["job_id"])

results = completed["results"]

6. Schedule Recurring Scrapes

Set up automatic monitoring:

schedule = client.create_schedule(

 strategy_id=strategy_id,

 url="https://example.com/products",

 interval_seconds=3600, # Every hour

 webhook_url="https://your-app.com/webhooks/scrape-results" # Optional

)

Or use cron expressions for more complex schedules. See the Webhooks Deep Dive section for details on webhook payloads.

Webhook Payload Example

Your endpoint will receive a POST request with the following JSON payload:

{

 "job_id": "job_abc123",

 "strategy_id": "strat_xyz789",

 "status": "completed",

 "results": [

 {"title": "Example Post", "score": 42},

 ...

 ],

 "has_changes": true,

 "change_summary": {

  "items_added": 3,

  "items_removed": 1,

  "items_modified": 2

 },

 "timestamp": "2024-01-15T10:30:00Z"

}

Response Requirements

Your endpoint should respond with 200 OK to indicate successful receipt.

Retry Logic: meter will retry failed webhooks up to 3 times with exponential backoff. If your endpoint is down or returns an error, meter will:

• Retry after 1 minute
• Retry after 5 minutes
• Retry after 30 minutes
• Give up if all retries fail

Example Webhook Handler

Here's a simple Python Flask example:

from flask import Flask, request

app = Flask(__name__)

@app.route("/webhooks/scrape-results", methods=["POST"])

def handle_webhook():

 data = request.json

 if data["has_changes"]:

  print(f"Changes detected: {data['change_summary']}")

  process_results(data["results"])

 return "OK", 200

How It Works

1. Schedule runs → Creates individual jobs for each URL
2. Jobs complete → Content hash compared to previous run for same URL
3. Changes tracked → Jobs with changed content marked as unseen (seen=false)
4. Poll for changes → Frontend calls /changes endpoint
5. Auto-mark as seen → Changes automatically marked when fetched (like reading email)

Database Schema

Migration: 003_add_seen_flag.sql

-- Add seen flag to scrape_jobs

ALTER TABLE scrape_jobs ADD COLUMN seen BOOLEAN DEFAULT FALSE;

-- Index for fast unseen lookups

CREATE INDEX idx_scrape_jobs_schedule_unseen

ON scrape_jobs (schedule_id, seen, completed_at DESC)

WHERE seen = FALSE AND status = 'completed';

ScrapeJob Model: seen (boolean) - Whether this change has been acknowledged

API Endpoint

GET /api/schedules/{schedule_id}/changes

Query Parameters:

• mark_seen (boolean, default: true) - Auto-mark changes as seen when fetched

Response:

{

 "schedule_id": "uuid",

 "changes": [

  {

   "id": "uuid",

   "url": "https://example.com",

   "status": "completed",

   "results": [...],

   "item_count": 10,

   "completed_at": "2024-01-15T10:30:00Z",

   "seen": true

  }

 ],

 "count": 3,

 "marked_seen": true

}

Returns:

• Only jobs where content changed compared to previous run
• Only unseen changes (unless mark_seen=false)
• Results ordered by completed_at (newest first)

Usage Examples

Get and acknowledge changes (default)

curl -X GET "https://api.meter.sh/api/schedules/{schedule_id}/changes" \

-H "Authorization: Bearer YOUR_API_KEY_HERE"

Returns changes and marks them as seen

Preview changes without marking

curl -X GET "https://api.meter.sh/api/schedules/{schedule_id}/changes?mark_seen=false" \

-H "Authorization: Bearer YOUR_API_KEY_HERE"

Returns changes but keeps them unseen

Frontend polling example

// Poll every 30 seconds for changes

setInterval(async () => {

 const response = await fetch(`/api/schedules/${scheduleId}/changes`);

 const { changes, count } = await response.json();

 if (count > 0) {

  console.log(`${count} new changes detected`);

  updateUI(changes);

 }

}, 30000);

Change Detection Logic

A job is considered "changed" if:

1. First run - No previous job exists for this schedule+URL
2. Content changed - content_hash differs from previous job for same URL

Per-URL tracking means:

• Multi-URL schedules report changes independently for each URL
• 10 URLs in schedule = up to 10 individual changes
• Each URL has its own change history

Push vs Pull

Use both together:

• Webhook for real-time notifications
• Changes endpoint for dashboard UI showing unread changes

During Beta

• Unlimited strategies - Create as many extraction strategies as you need
• Unlimited scrapes - Run as many jobs as you want
• 100MB max response size per scrape
• JavaScript rendering: Limited support (static sites work best)
• Authentication: Not yet supported (must be publicly accessible URLs)

What doesn't work (yet)

• Sites requiring login or authentication
• Heavy SPAs (React/Vue without SSR) - Limited JavaScript rendering support
• Sites with aggressive anti-bot (Cloudflare Turnstile, etc.)
• Pages that require cookies or session management

Webhook Endpoint Requirements

• Must be publicly accessible (no localhost)
• Must respond with 200 OK within 30 seconds
• Must accept POST requests with JSON payloads
• If your endpoint is down, meter will retry up to 3 times with exponential backoff

Strategy Management

• client.generate_strategy(url, description, name) - Generate a new strategy
• client.refine_strategy(strategy_id, feedback) - Refine with feedback
• client.list_strategies() - List all strategies
• client.get_strategy(strategy_id) - Get strategy details

All of these can also be done from the web dashboard.

Job Execution

• client.create_job(strategy_id, url) - Create a scrape job
• client.get_job(job_id) - Get job status and results
• client.wait_for_job(job_id) - Poll until completion
• client.compare_jobs(job_id1, job_id2) - Compare two jobs

Jobs can be created and monitored from the dashboard.

Schedule Management

• client.create_schedule(strategy_id, url, interval_seconds, webhook_url) - Create a schedule (webhook optional)
• client.update_schedule(schedule_id, enabled, ...) - Update schedule
• client.list_schedules() - List all schedules

Schedules can be created and managed from the dashboard. Webhooks receive POST requests with job results when scrapes complete.

Extract all forum posts including the original post and replies:

from meter_sdk import MeterClient

client = MeterClient(api_key="sk_live_...")

result = client.generate_strategy(

 url="https://forum.example.com/thread/123",

 description="Extract all forum posts including the original post and all replies. Each post has author and content.",

 name="Forum Posts"

)

# Run a scrape

job = client.create_job(result["strategy_id"], "https://forum.example.com/thread/123")

completed = client.wait_for_job(job["job_id"])

print(f"Extracted {len(completed['results'])} posts")

The system will automatically find all posts, including the original post and all replies, even if they use different class names.

Extract product names and prices:

from meter_sdk import MeterClient

client = MeterClient(api_key="sk_live_...")

strategy = client.generate_strategy(

 url="https://shop.example.com/products",

 description="Extract product names and prices",

 name="Product Scraper"

)

# Set up daily monitoring

schedule = client.create_schedule(

 strategy_id=strategy["strategy_id"],

 url="https://shop.example.com/products",

 cron_expression="0 9 * * *", # Daily at 9 AM

 webhook_url="https://your-app.com/webhooks/products" # Optional

)

The system will automatically identify product listings and extract names and prices from each product. You can also do this entirely from the web interface.

Extract metrics with numbers and labels:

from meter_sdk import MeterClient

client = MeterClient(api_key="sk_live_...")

strategy = client.generate_strategy(

 url="https://company.com/metrics",

 description="Extract metrics with numbers like '340,000', '85,000+', '97.6%' and their labels like 'Engineering Hours Saved', 'Vulnerabilities'",

 name="Metrics Dashboard"

)

# Check for changes over time

history = client.get_strategy_history(strategy["strategy_id"])

if len(history) > 1 and history[0]["has_changes"]:

 print("Metrics have changed!")

The system will automatically find metric cards and extract the numeric values with their corresponding labels. All of this can also be done from the web dashboard.

Problem: Extraction returns 0 items

Cause: Container selector doesn't exist or item selector is wrong.

Solution: System auto-fixes invalid containers and retries. Check logs for "Auto-fixed invalid container" message.

Manual fix: Refine with feedback: "Found 0 items, try using body as container"

Problem: Missing the first/last item

Cause: First item may have a different class name than other items.

Solution: The system automatically detects variant class names. If it still misses items, refine with feedback.

Manual fix: Refine with feedback: "Missing the first item at the top"

Problem: Extracting too many items (wrong content)

Cause: Item selector too broad or wrong container.

Solution: Use a more specific container or item class.

Manual fix: Refine with feedback: "Getting navigation items, need only post content"

Problem: All items have identical content

Cause: Item selector matches parent instead of individual items.

Solution: System validates and warns if items are identical.

Manual fix: Look for a more specific selector that matches individual items.

Problem: Strategy generation fails

Possible causes:

• URL is unreachable or returns non-HTML content
• Page requires JavaScript rendering (some SPAs may not work)
• Page is behind authentication or requires cookies

Solution: Verify the URL is accessible in a browser, check if the page requires login, review error messages in the dashboard.

How Pricing Works

meter uses a cost-efficient model:

• Strategy Generation: One-time cost when you create a new strategy (~$0.02-0.06 per strategy)
• Strategy Refinement: Optional, only when you need to improve a strategy (~$0.02-0.06 per refinement)
• Production Scrapes: Free - unlimited scrapes using saved strategies cost $0

Example: Set up 100 sites for $2-6 one-time, then run daily scrapes forever at no additional cost. This is much cheaper than traditional LLM-based scraping that charges per scrape.