# Oktavia Crawler Policy

Canonical URL: https://oktavia.app/docs/oktavia-crawler-policy.md

## Purpose

This document describes how Oktavia exposes public content to search engines and AI retrieval systems.

## Public Content

Public pages and public Markdown references are crawlable:

- home,
- pricing,
- onboarding,
- Connect developer docs,
- status page,
- public policies,
- public docs under `/docs/`,
- `llms.txt`.

## Private Content

Operational app areas are not intended for public indexing:

- `/empresa/`
- `/trabajador/`
- `/worker/`
- `/owner`
- `/admin`
- `/manager`
- `/auditor`
- `/superadmin/`

These areas are disallowed in `robots.txt` and private/auth routes also receive `noindex,nofollow` metadata from the Angular SEO service.

## AI Retrieval

Oktavia allows AI search and assistant retrieval over public content when the crawler respects `robots.txt`.

Public AI-friendly references:

- `/llms.txt`
- `/docs/oktavia-overview.md`
- `/docs/oktavia-pricing.md`
- `/docs/oktavia-onboarding.md`
- `/docs/oktavia-compliance-summary.md`
- `/docs/oktavia-connect-summary.md`

`OAI-SearchBot` and `ChatGPT-User` are explicitly allowed for public content discovery and retrieval.

`GPTBot` is explicitly disallowed because it is associated with model-training crawling rather than search/retrieval visibility. This decision can be revisited later if Oktavia decides to allow training crawlers for public marketing content.

## IndexNow

IndexNow support is prepared with a public key file:

- Key: `00e9d567deeea90d326fa6b7f624933c`
- Key location: `https://oktavia.app/00e9d567deeea90d326fa6b7f624933c.txt`

Submission command:

```bash
npm run seo:indexnow -- --all
```

Or submit specific URLs:

```bash
npm run seo:indexnow -- /pricing /onboarding
```

## Verification Tasks

Manual tasks still required:

- Add Oktavia to Google Search Console.
- Add Oktavia to Bing Webmaster Tools.
- Submit `https://oktavia.app/sitemap.xml`.
- Validate `robots.txt`.
- Validate public metadata and structured data after deployment.