Installation and configuration guide

This guide walks you through the complete setup process for the Intranet crawler, from initial configuration to enabling the tool for your assistants. Setup requires admin permissions in Intric and typically involves collaboration with your IT or intranet administration team.

For a high-level overview of capabilities and security, see the article below.

📖

Intranet crawler

Read more

Prerequisites

Before beginning setup, ensure the following:

• You have Admin permissions in your Intric tenant
• The MCP module is enabled for your tenant
• You have identified which intranet content should be indexed
• You have created or identified a service account with appropriate permissions on your intranet
• Your intranet is accessible over HTTPS
• You understand your intranet’s authentication requirements

Step 1: Create the MCP server record

Navigate to the MCP servers section in the Intric admin interface.

Click Create and provide a descriptive name for the server, such as “Company Intranet Crawler” or “Employee Handbook Search”.

The server record is created. You can now proceed to configure its connection details.

Step 2: Configure the server URL and authentication

Edit the newly created server and provide the following information:

Server URL

Enter the HTTPS endpoint of the Intranet crawler service. This URL will be provided by Intric during setup coordination.

Example: https://crawler.yourcompany.intric.app/mcp

Authentication

Choose API key as the authentication type.

Provide the API key that was generated during the crawler service provisioning. This key is used to authenticate Intric’s requests to the crawler and is encrypted at rest.

Security note: Store the API key securely. It will not be visible in the interface after saving, and only indicates whether a key is configured.

Description (optional)

Add a description to help other administrators understand the purpose and scope of this crawler, for example:

“Indexes the employee handbook, HR policies, and IT documentation from the internal intranet.”

Save the configuration. Intric will automatically discover the crawler’s available capabilities: Search Intranet and Correct Intranet.

Step 3: Configure crawl targets

Crawl targets define which pages on your intranet the crawler should visit and index.

In the MCP server configuration, navigate to the Crawl targets section.

Add URLs for each section of your intranet you want to index. For each target, specify:

• URL — The starting point for the crawl (e.g., https://intranet.yourcompany.com/handbook)
• Crawl depth (optional) — How many levels of links the crawler should follow from the starting URL
• URL patterns (optional) — Include or exclude specific URL patterns to focus the crawl

Example configuration:

URL: https://intranet.yourcompany.com/handbook
Depth: 2
Include patterns: /handbook/*, /policies/*
Exclude patterns: /archive/*, /draft/*

This configuration indexes the handbook section and all linked policy pages, while skipping archived and draft content.

Step 4: Configure authentication for the crawler

The crawler needs credentials to access your intranet. The specific configuration depends on your intranet’s authentication method.

For basic authentication or form-based login

Provide the username and password for the service account.

Important: Specify which service account should be used and ensure it has:

• Access to all content you want indexed
• No access to content that should remain restricted
• A strong, unique password
• Appropriate audit logging enabled

For multi-step authentication

If your intranet requires multi-step login (e.g., username/password followed by a security question), you’ll need to work with Intric’s technical team to configure the authentication flow.

Provide:

• Step-by-step instructions for the login process
• Any security questions and answers
• Session timeout information

Note: Authentication flows with CAPTCHA or device-based two-factor authentication are not currently supported.

For SSO or OAuth

If your intranet uses single sign-on or OAuth authentication, contact Intric’s support team for guidance on the appropriate configuration method.

Step 5: Configure document extraction

The crawler can extract and index various document types found on your intranet.

In the Document extraction settings, specify:

• Supported file types — PDF, Word documents, Excel spreadsheets, etc.
• Maximum file size — Documents larger than this limit will be skipped
• Text extraction method — How the crawler should process different document types

Recommended settings:

Supported types: PDF, DOCX, XLSX, TXT
Maximum size: 10 MB
Extract text from: Body content, document metadata, embedded text

Step 6: Set crawl schedule

Configure how often the crawler should re-index your intranet content.

The default schedule is once per day (daily at 02:00 UTC), which is appropriate for most organizations. You can adjust this based on:

• How frequently your intranet content changes
• System load considerations
• The size of your intranet

Example schedules:

• Static content (handbooks, policies): Once per week
• Regularly updated content (news, announcements): Once per day
• Frequently changing content: Consider whether the crawler is the right tool

Rate limiting: The crawler includes built-in rate limiting to avoid overwhelming your intranet server. Review the default settings and adjust if needed based on your infrastructure.

Step 7: Test the configuration

Before enabling the crawler for production use, validate the configuration:

1. Click Validate connection to verify the crawler can reach your intranet and authenticate successfully
2. Review the connection status:
   • OK — Configuration is working correctly
   • Unauthorized — Check service account credentials
   • Failed — Verify URL and network accessibility
3. Trigger a manual test crawl to verify content extraction
4. Review the crawl logs to ensure expected pages are being indexed
5. Check for any authentication errors or permission issues

Step 8: Enable the tool for your tenant

Once configuration is complete and tested:

1. Navigate to the Tools section in the admin interface
2. Find the Intranet crawler in the list of available tools
3. Enable it for your tenant
4. Configure security classification settings if required

The tool is now available for use in assistants.

Step 9: Attach the tool to assistants

To make the Intranet crawler available to users:

1. Open or create an assistant in a Space
2. Navigate to the assistant’s Tools configuration
3. Add the Intranet crawler tool
4. Save the assistant

Users can now ask the assistant to search intranet content during conversations.

Managing corrections

When users suggest corrections to indexed content:

1. Corrections appear in the Intranet corrections admin panel
2. Review each suggestion for accuracy and appropriateness
3. Options for each correction:
   • Accept — Apply the correction as submitted
   • Modify — Edit the correction before applying
   • Reject — Decline the correction with optional feedback

Accepted corrections are applied to the search index during the next crawl cycle.

Security best practices

Service account configuration

• Create a dedicated service account specifically for the crawler
• Grant only the minimum necessary permissions
• Use a strong, unique password
• Enable audit logging for the service account
• Review service account access regularly

Content scoping

• Carefully consider which intranet sections to index
• Exclude sensitive HR data, financial information, or confidential projects
• Use URL patterns to restrict crawling to approved areas
• Regularly review indexed content for appropriateness

Access control

• Configure security classification rules for the Intranet crawler tool
• Limit which assistants have access to the crawler
• Consider creating specialized assistants for sensitive content areas
• Use Space permissions to control who can use intranet-enabled assistants

Troubleshooting

Issue	Solution
”Server URL is not set”	Edit the MCP server and provide a valid HTTPS URL
”Unauthorized” during validation	Verify service account credentials and permissions
No content appearing in search results	Check crawl logs for errors; verify crawl targets are correct
Authentication keeps failing	Review multi-step authentication configuration; check for session timeouts
Crawl is too slow	Reduce crawl depth or restrict URL patterns; adjust rate limiting
Incorrect content being indexed	Refine URL include/exclude patterns; check service account permissions
Corrections not appearing	Ensure Correct Intranet capability is enabled; check user permissions

Getting help

If you encounter issues during setup or need assistance with complex authentication configurations, contact Intric support with:

• Your tenant name
• The MCP server name
• Screenshots of any error messages
• Crawl log excerpts (if applicable)
• Description of your intranet’s authentication method