Intranet crawler

Most intranets cannot be searched from the outside: they require a login, build their pages with JavaScript, and hide half their content in PDF attachments. This integration crawls them anyway and makes the content searchable for the assistant.

It is the most widely deployed integration in our fleet, with around twenty live instances.

What it is

A configurable crawler and search service for web-based intranets. It logs in via SAML or an ordinary login form, renders JavaScript-heavy pages in a real browser, extracts the text from PDF and Word documents, and indexes everything with multilingual embeddings. The assistant then searches semantically — it finds the right passage even when the user does not happen to use the same words as the document.

Configuration and re-crawling are handled in a web-based admin interface.

Use it for

• “What does our travel policy say about hotel standards?”
• “Find the procedure for reporting an incident.”
• “Who do I contact to get access to the payroll system?”
• “What applies to parental leave here — summarise and link to the source.”
• “Is there a template for business travel, and where is it?”

Tools

Function	Answers
Usage guide	How search works and how to read the citations — the assistant reads this first
Intranet search	Relevant pages and document passages for a question, with a link to the source
Correction	Lets a user report that an answer was wrong and state what is correct

A correction does not become searchable immediately. It goes into a queue and becomes searchable only once an administrator approves it — after which it outweighs the original content for similar questions. That makes corrections a way to repair knowledge gaps without anyone being able to rewrite the truth on their own.

What the crawler handles

• Login via SAML or a form, with the session held throughout the crawl.
• JavaScript-heavy sites rendered in a real browser before the text is read out.
• Documents — PDF and Word are taken apart and indexed alongside the pages.
• Several sources in one knowledge base. One instance can crawl an additional site, or a login-protected subsection with its own credentials, and place the result in the same searchable base.
• Extra seed URLs for pages that cannot be reached through ordinary link navigation.
• External links can be followed one hop beyond the site. Off by default, and always capped.
• Multilingual — the index handles bilingual intranets.

Getting started

Intric deploys one instance per site. You provide the start URL, any sitemap or RSS feed, login credentials and crawl limits in the admin interface, and can restart the crawl from there at any time.

The instance can run in Intric’s environment or entirely in your own — one of today’s deployments sits in the customer’s own cluster, with no inbound connection from Intric.

Limitations

• Permissions are at index level, not per user. The crawler indexes what the logged-in crawl user can see, and everything indexed is then searchable by everyone with access to the assistant. Point the crawl at content that is open to the whole organisation, and control the rest through who may use the assistant.
• Content is as fresh as the last crawl. Crawling runs on a schedule or on demand, not on every question.
• Images and video are not indexed — it is the text that becomes searchable.
• Sites that actively block automated access may require your IT department to let the crawler through.