Intranet crawler · 4 min read

Installation and configuration guide

Step-by-step guide to setting up the Intranet crawler for your organization.

This guide walks you through the complete setup process for the Intranet crawler, from initial configuration to enabling the tool for your assistants. Setup requires collaboration with your Intric technical team — there is no self-serve onboarding flow.

For a high-level overview of capabilities and security, see the article below.

Intranet crawler

Prerequisites

Before beginning setup, ensure the following:

You have identified which intranet content should be indexed
You have created or identified a service account (or browser login) with appropriate permissions on your intranet
Your intranet is accessible over HTTPS
You understand your intranet’s authentication requirements

Step 1: Reach the crawler admin panel

Navigate to https://mcp-servers.intric.ai/<customer>-intranet/admin in your browser. The panel is protected by HTTP Basic authentication — use the credentials provided during your onboarding. The panel is Swedish-first with an English toggle.

Note: The MCP endpoint at https://mcp-servers.intric.ai/<customer>-intranet/mcp has a separate IP-based access gate. If you cannot reach the panel, verify with Intric that your inbound IP addresses are on the allowlist.

Step 2: Verify server configuration

The crawler is already configured with its server URL during onboarding. The URL follows the pattern https://mcp-servers.intric.ai/<customer>-intranet/mcp where <customer> is your customer identifier (e.g., /varberg-intranet, /lidkoping, /oslo).

If you are running the crawler in your own cluster, the URL will follow your own infrastructure pattern.

Your API key — a 365-day signed token minted by an Intric engineer from the customer’s deployment secret — is already configured. You cannot regenerate it yourself; contact Intric support when it needs renewal.

Step 3: Configure crawl targets

In the crawler admin panel, navigate to the Mål (Targets) section.

You configure two settings:

URL prefix

Set url_filter_prefix to a single prefix string. This is the base path the crawler will follow. Enter the root URL of your intranet or a specific section.

Example:

url_filter_prefix: https://intranet.yourcompany.com/handbook

URL exclusions

Set exclude_url_patterns to a list of substring patterns. Any URL containing one of these substrings will be excluded. No glob syntax — these are simple substring matches.

Example:

exclude_url_patterns:
  - /archive/
  - /draft/

These settings are written to /data/crawler_config.json on the deployment.

Step 4: Configure authentication for the crawler

The crawler supports form-based login to your intranet. It handles:

Generic CMS login forms
Sitevision portlets
Multi-step Azure AD / Entra SAML flows, including “Stay signed in?” interstitials

Provide the credentials for the account the crawler should use. Some deployments crawl public sites with no login at all.

Supported file types: The crawler detects and indexes .docx, .xlsx, .pdf, and .txt files. Note that .doc and .xls files are detected but silently dropped. Document extraction has the following limits:

10 MB hardcoded download cap for documents
20 MB configurable size limit (per-deployment environment variable)
PDFs are read only to page 20
Extracted text truncates at 300,000 characters

Not supported: The crawler does not handle CAPTCHA or device-based two-factor authentication.

Step 5: Document extraction settings

In the crawler admin panel, there is a single document extraction control: an on/off checkbox. File types are hardcoded (see above). Size limits are set per-deployment via environment variables, not through the UI.

Step 6: Configure crawl schedule

The default full crawl runs once daily at 02:00 UTC.

A second scheduled job performs incremental news crawling every 30 minutes. To change either interval you must restart the pod.

For pre-crawled data mode, updates are manual — send a new data file when content changes. Scheduled crawls are skipped entirely in that mode.

Step 7: Test the crawl

The crawler admin panel provides four buttons for testing:

Full crawl — crawl all configured targets
Test crawl (one page) — crawl a single page to verify configuration
News crawl — trigger the incremental news crawl
Extra-sources crawl — crawl additional configured sources

These are the only operational controls in the panel. There is no “Validate connection” button.

Step 8: Enable the tool for your assistants

Once configuration is complete and tested:

Navigate to the Tools section in your Intric Space
Find the Intranet crawler in the list of available tools
Enable it for assistants in that Space
Configure security classification settings if required

The tool is now available for use.

Managing corrections

In the crawler admin panel, navigate to the corrections section, which displays a pending-count badge.

Options for each pending correction:

Accept — Index the correction immediately. Accepted corrections take precedence over crawled content in future answers.
Edit — Edit the correction and accept it in one action.
Deny — Decline the correction.

Troubleshooting

Issue	Solution
Cannot reach the admin panel	Verify your inbound IP addresses are on the allowlist — contact Intric support
Crawl returns no pages	Check crawl targets and URL exclusions; verify crawl account permissions
Crawling is slow	Reduce crawl depth (default 3) or restrict URL exclusions; check rate limiting
Authentication failing	Review form login configuration; the crawler does not support Basic auth to target sites
Corrections not appearing	Ensure `correct_intranet_answer` capability is enabled on the assistant

Crawl logs are written to pod output and accessed via kubectl — this is an Intric-engineer action, not a customer-facing feature. There is no log view in the UI.

Getting help

If you encounter issues during setup or need assistance, contact Intric support:

Your customer identifier (the path segment in your admin panel URL)
Description of the issue
Any error messages you see