Step 2: Connect Airtable

Pull the company list from Airtable for batch processing.

1. Add Retrieve Airtable Records and set Operation to search.
2. Select your Airtable Base and Table (e.g., Indeed → Table 1).
3. Credential Required: Connect your airtableTokenApi credentials.
4. Connect Configure Bright Data Zone → Retrieve Airtable Records → Iterate Through Batches.

Step 3: Set Up Batch Control, Delay, and Link Validation

These nodes throttle requests and prevent invalid URLs from being scraped.

1. In Configure Bright Data Zone, set the zone assignment to web_unlocker1.
2. Place Iterate Through Batches to control how many records are processed per cycle.
3. Configure Pause Execution with Amount set to 10 seconds.
4. In Validate Link Presence, keep the condition set to String → notEmpty with Left Value {{ $json.Link }}.

Step 4: Configure Indeed Page Request and Parallel Processing

Scrape the Indeed company page and process the response in parallel as both raw text and HTML.

1. In Request Indeed Page, set URL to https://api.brightdata.com/request and Method to POST.
2. Enable Send Body and Send Headers.
3. Set body parameters:
   zone = {{ $('Configure Bright Data Zone').item.json.zone }}
   url = https://www.indeed.com/cmp/{{ encodeURI($('Retrieve Airtable Records').item.json.Link) }}?product=unlocker&method=api
   format = raw
   data_format = markdown
4. Credential Required: Connect your httpHeaderAuth credentials in Request Indeed Page.
5. Ensure Request Indeed Page outputs to both Extract Text from Markdown and Render Markdown to HTML in parallel.

Step 5: Set Up AI Extraction, Summarization, and Agent Analysis

These nodes convert markdown to clean text, summarize it, and structure the results for webhook delivery.

1. In Extract Text from Markdown, set Text to You need to analyze the below markdown and convert to textual data. {{ $json.data }}.
2. Gemini Chat Engine is connected as the language model for Extract Text from Markdown — ensure credentials are added to Gemini Chat Engine. Credential Required: Connect your googlePalmApi credentials.
3. Summarize Company Details receives the extracted text; Gemini Summary Model powers this summarization — Credential Required: Connect your googlePalmApi credentials.
4. In Indeed Analysis Agent, keep Text set to You are an Indeed Expert... {{ $('Extract Text from Markdown').item.json.text }} so it formats the summary for delivery.
5. Gemini Agent Model is connected as the language model for Indeed Analysis Agent — Credential Required: Connect your googlePalmApi credentials.
6. Send Summary Webhook is an AI tool connected to Indeed Analysis Agent; configure tool behavior here, and add any needed auth on the parent agent if required by your endpoint.

Step 6: Configure HTML Output Webhook

This branch converts the markdown into HTML and posts it to a webhook endpoint.

1. In Render Markdown to HTML, set Mode to markdownToHtml and Markdown to {{ $json.data }}.
2. Configure Post HTML Webhook with URL set to https://webhook.site/daf9d591-a130-4010-b1d3-0c66f8fcf467 and Send Body enabled.
3. Set the body parameter html_response to {{ $json.data }}.

Step 7: Test and Activate Your Workflow

Run a full test to validate scraping, AI summarization, and webhook outputs before going live.

1. Click Manual Start Trigger → Execute Workflow to run a test.
2. Confirm that Request Indeed Page returns markdown data and that Validate Link Presence passes for valid records.
3. Verify that Extract Text from Markdown and Summarize Company Details produce AI output, and that Indeed Analysis Agent sends data via Send Summary Webhook.
4. Check your webhook endpoint to see both the HTML payload from Post HTML Webhook and the structured JSON from Send Summary Webhook.
5. Once successful, toggle the workflow Active for production use.