Data Flows · 17 min read
Intranet Crawler
How your organization's intranet content is indexed and how search queries are handled securely via the MCP interface.
The Intranet Crawler is a self-hosted MCP server that indexes your organization’s intranet and makes its content searchable for AI assistants — without any intranet content leaving your own infrastructure.
The service operates as two separate processes: a background crawl that periodically fetches and indexes pages from your intranet, and a query interface (MCP server) that Intric AI calls when a user’s question requires searching the indexed content.
How content is indexed
Section titled “How content is indexed”The crawl runs entirely inside your own Kubernetes cluster on a schedule. No user data or conversation history is involved. The only data that leaves the pod during indexing is plain text chunks sent to the external embedding API to be converted into search vectors.
All data in transit is protected by TLS 1.2 or higher.
Step 1 — Scheduled crawl trigger
APScheduler triggers a crawl automatically — a full crawl every night at 02:00 UTC and an incremental update every 30 minutes.
No user data is involved. The scheduler runs inside the Kubernetes pod and communicates with the crawler via an internal asyncio event loop.
How queries are handled
Section titled “How queries are handled”When a user asks a question that requires searching the intranet, Intric AI calls the intranet crawler’s MCP server. Intric always acts as the intermediary — the language model never contacts the MCP server directly.
All data in transit is protected by TLS 1.2 or higher.
Step 1 — User interacts with Intric in the browser
The user writes a message to an assistant that has the intranet crawler MCP tool configured.
Data sent to Intric’s server:
- The user’s message
- Chat history
- Any attached files
Data sharing and privacy
Section titled “Data sharing and privacy”The intranet crawler is designed so that your organization’s intranet content stays within your own infrastructure. Below is a summary of what is sent to external services and what remains internal.
Crawl — Embedding API
Section titled “Crawl — Embedding API”The only external service that receives data during indexing is the embedding API. Only plain text chunks are sent — no URLs, page titles, or any metadata accompany the chunks.
| Sent to the Embedding API | Not sent to the Embedding API |
|---|---|
|
|
Query — MCP Server
Section titled “Query — MCP Server”The MCP server is hosted in your own Kubernetes cluster — it is not a third-party service. Only the search query and a JWT for authentication are sent from Intric to the MCP server.
| Sent to the MCP server | Not sent to the MCP server |
|---|---|
|
|
Hosting, authentication and credentials
Section titled “Hosting, authentication and credentials”Hosting
Section titled “Hosting”MCP servers hosted by Intric are hosted in Sweden using the subprocessor Glesys AB. Custom developed MCP servers deployed in a customer’s environment run in the customer’s environment.
When hosted in a customer specific instance, each MCP server runs in its own Kubernetes pod with dedicated infrastructure, databases, and a strict set of rules governing what it can and cannot access. MCP servers are logically isolated from one another — one server cannot reach another unless an explicit connection between them is established.
Authentication and credentials
Section titled “Authentication and credentials”All secrets used by the intranet crawler are stored in Kubernetes Secrets and injected as environment variables at pod startup — they are never exposed to users or the browser.
- MCP server: JWT (HS256) using a shared secret (
MCP_SERVER_JWT_SECRET). Intric AI and the MCP server share this secret to authenticate every search call. - Intranet login: If the intranet requires a login, form-based credentials are stored on a Persistent Volume encrypted with Fernet (AES-128-CBC). The encryption key is mounted as a Kubernetes Secret volume.
- Embedding API: The API key (
EMBEDDING_API_KEY) is stored as a Kubernetes Secret. - Admin interface: The web-based admin panel (
/admin/) is protected with HTTP Basic Auth. An IP allowlist (ALLOWED_IPS) restricts which addresses can reach the MCP server endpoint.
What Intric needs to set it up
Section titled “What Intric needs to set it up”One crawler instance is configured per site. Most of the setup is a short list of URLs and one account — but two decisions shape everything else, so settle those first.
Decide these two things first
Section titled “Decide these two things first”Which content should be searchable by everyone. The crawler indexes what its own account can see, and everything it indexes becomes reachable by every user of the assistant. Per-user permissions on the intranet are not carried across. Point the crawl at content that is open to the whole organisation, and control the rest through who may use the assistant. This is the single most consequential choice in the setup.
Whether the intranet requires a login. This determines whether a service account is needed at all, and it is the part that usually takes longest on your side.
Always needed
Section titled “Always needed”| What | Why |
|---|---|
| Start URL | Where the crawl begins — usually the intranet’s front page |
| URL prefix to stay within | Keeps the crawl inside the right section when the intranet shares a domain with other content. Without it, the crawl follows wherever links lead |
| Whether a login is required | Decides whether the rest of the authentication setup applies |
If the intranet requires a login
Section titled “If the intranet requires a login”| What | Notes |
|---|---|
| Login URL | The page carrying the login form, if it is not the start URL |
| Service account username and password | Stored encrypted on the volume in the cluster — see Authentication and credentials above |
The service account itself needs some care, and getting it wrong is the most common reason a first crawl returns almost nothing:
- Read-only. The crawler never writes, so the account needs no write permission anywhere.
- Its access defines the index. Give it exactly the access that matches the decision above — no more.
- Exempt from multi-factor authentication, and from any conditional-access rule that would challenge a non-interactive login. The crawler cannot complete an MFA prompt.
- A password that does not expire, or a rotation routine agreed in advance. An expired password stops the crawl silently as far as readers are concerned — search simply goes stale.
- Not subject to lockout on repeated logins, since the crawler signs in on every scheduled run.
Strongly recommended
Section titled “Strongly recommended”| What | Why |
|---|---|
| Sitemap URL | Much more reliable than link-following, and finds pages nothing links to |
| RSS feed or news listing URL | News pages are often not reachable by ordinary navigation |
| Extra start URLs | For sections that exist but nothing links to — they are added to the crawl directly |
Scope and limits
Section titled “Scope and limits”Sensible defaults exist for all of these; supply them only if your site needs something different.
- Maximum pages and crawl depth
- URL patterns to exclude — archives, print views, calendar pages that expand forever
- Query parameters to strip before de-duplication — some platforms append session or state parameters that make one page look like thousands
- Additional crawl sources — a second site, or a login-protected subsection with its own credentials, indexed into the same searchable base
Network access
Section titled “Network access”The crawler must be able to reach your intranet, and your intranet must be willing to be reached:
- Allow the crawler through any firewall, WAF or bot protection in front of the intranet. Ask Intric for the address to allow — it is a fixed outbound IP for Intric-hosted instances.
- Self-hosted instances need a network route from the pod to the intranet instead, since nothing leaves your infrastructure.
- Admin interface access — who should reach the admin panel, and from which addresses, since it is protected by both a password and an IP allowlist.
Tell us early if any of these apply
Section titled “Tell us early if any of these apply”Each changes how the crawler has to be configured, and each is much cheaper to handle before the first crawl than after:
- SharePoint. Document libraries are rendered by JavaScript with no ordinary links to follow, so discovery has to run through SharePoint’s own permission-trimmed search instead.
- A single-page application — modern React-based intranets that finish rendering after the page appears to have loaded.
- Authentication carried in the URL rather than in a session, as some document platforms do.
- A bilingual site, which is supported, but worth knowing about in advance.
Data retention and deletion
Section titled “Data retention and deletion”Conversation history where the intranet crawler MCP tool has been used follows the same deletion rules as other assistants.
The crawled content and vector index are stored on the Persistent Volume within your Kubernetes cluster. Pages that have not changed since the last crawl (detected via ETag/Last-Modified headers) are skipped. To clear the index, the ChromaDB collection can be reset via the admin interface or by removing the PVC.
Administrators can monitor how the service is used via the audit log where enabled.