feat(knowledge): 任务队列、独立 worker 与 pgvector 向量索引

知识处理改为数据库任务队列，独立 worker 做切块与向量化，失败可重试。
2026-09-30 22:05:00 +08:00
1 parent 7937de5d7d
commit c7011040c3
@@ -0,0 +1,168 @@
+# ReplyPier Enhanced Website Knowledge Base
+
+This module provides an enhanced website knowledge extraction system that significantly improves content extraction from web pages compared to the default implementation.
+
+## Key Features
+
+- **Multi-strategy content extraction**: Uses multiple strategies to extract the most relevant content from web pages.
+- **Robust error handling**: Implements retries with exponential backoff for failed requests.
+- **Text density analysis**: Can identify important content based on paragraph density when standard selectors fail.
+- **Smart content cleaning**: Automatically removes boilerplate, navigation, ads, and other non-content elements.
+- **Configurable parameters**: Allows customization of crawl depth, number of links, timeouts, etc.
+- **High-performance parallel embedding**: Embeds documents in parallel during crawling to maximize throughput.
+- **Optimized database insertion**: Preserves embeddings to avoid redundant work during vector database operations.
+
+## Usage
+
+### Basic Usage
+
+```python
+from app.knowledge.enhanced_website_kb import EnhancedWebsiteKnowledgeBase
+from agno.vectordb.pgvector import PgVector
+
+# Initialize vector database
+vector_db = PgVector(
+    table_name="your_table",
+    db_url="your_db_url",
+    schema="your_schema"
+)
+
+# Create knowledge base
+kb = EnhancedWebsiteKnowledgeBase(
+    urls=["https://example.com"],  # List of URLs to crawl
+    max_depth=3,                   # How deep to crawl (default: 3)
+    max_links=10,                  # Maximum number of links to follow (default: 10)
+    min_content_length=100,        # Minimum content length to be considered valid
+    timeout=30,                    # Request timeout in seconds
+    max_retries=3,                 # Maximum retry attempts for failed requests
+    vector_db=vector_db            # Vector database for storing extracted content
+)
+
+# Load knowledge base (crawl websites and store in vector DB)
+kb.load(
+    recreate=False,                # Whether to recreate the collection
+    upsert=True,                   # Whether to update existing documents
+    filters={"name": "example"}    # Additional filters for the documents
+)
+```
+
+### High-Performance Configuration with Optimized Vector Database
+
+For maximum performance, use the `OptimizedPgVector` class with increased parallelism settings:
+
+```python
+from app.knowledge.enhanced_website_kb import EnhancedWebsiteKnowledgeBase
+from app.knowledge.optimized_pgvector import OptimizedPgVector
+
+# Initialize optimized vector database
+vector_db = OptimizedPgVector(
+    table_name="your_table",
+    db_url="your_db_url",
+    schema="your_schema"
+)
+
+# Create knowledge base with increased parallelism
+kb = EnhancedWebsiteKnowledgeBase(
+    urls=["https://example.com"],
+    max_depth=5,                   # Deeper crawling for more content
+    max_links=25,                  # More links for comprehensive coverage
+    min_content_length=100,
+    timeout=30,
+    max_retries=3,
+    max_workers=32,                # High parallel processing for maximum throughput
+    batch_size=100,                # Larger batch size for better database performance
+    vector_db=vector_db
+)
+
+# Load knowledge base
+kb.load(
+    recreate=False,
+    upsert=True,
+    filters={"source": "website"}
+)
+```
+
+## Performance Optimization
+
+The system is designed to maximize performance at every stage:
+
+1. **Parallel Crawling**: Multiple URLs are crawled simultaneously using ThreadPoolExecutor
+2. **Parallel Embedding**: Documents from each URL are embedded in parallel batches
+3. **Final Parallel Embedding**: Any remaining unembedded documents are processed with high parallelism
+4. **Embedding Preservation**: The OptimizedPgVector class preserves embeddings during database operations
+5. **Batched Database Operations**: Documents are inserted into the database in optimized batches
+6. **Vector Database Indexing**: Automatic index optimization for faster vector searches
+
+This approach dramatically reduces the total processing time, especially for large websites with many pages.
+
+### Configuration Options
+
+The `EnhancedWebsiteReader` and `EnhancedWebsiteKnowledgeBase` classes offer several configuration options:
+
+- **max_depth**: How many levels deep to crawl from the starting URL (default: 5)
+- **max_links**: Maximum number of links to follow (default: 25)
+- **min_content_length**: Minimum text length to be considered valid content (default: 100)
+- **timeout**: HTTP request timeout in seconds (default: 30)
+- **max_retries**: Maximum number of retry attempts for failed requests (default: 3)
+- **max_workers**: Number of parallel workers for crawling and embedding (default: 10)
+- **crawl_scope**: How far link-following may wander from the starting URL (default: `host`)
+  - `host` — pages on the starting URL's own host only (`www.` and the bare host count as one host)
+  - `path` — that host, restricted to the starting URL's path prefix (e.g. only `/hc/…`)
+  - `domain` — every host on the registrable domain, so a `help.example.com` crawl also
+    walks `www.example.com`, `example.com/blog/…` and other subdomains
+  Only `http`/`https` links are ever followed — `mailto:`, `tel:` and friends are not pages.
+
+Content stored by earlier, wider crawls is cleaned up in two steps:
+
+- Pages that were never pages (mangled `https://mailto:a@example.com/…` URLs) are deleted
+  automatically by the `purge_mangled_crawl_pages_001` migration.
+- Real pages that are simply outside the new scope (a marketing site pulled in from a
+  help-center seed) are left alone; `scripts/prune_offsite_pages.py` reports them per
+  source and deletes them only with `--apply`.
+- **batch_size**: Size of document batches for database operations (default: 20)
+- **blacklist_tags**: HTML tags to remove before content extraction (scripts, styles, etc.)
+- **min_content_length**: Floor a page's extracted text must clear to count as readable (default: 100)
+
+Candidate containers and the retention threshold live in `main_content.py`
+(`CANDIDATE_TAGS`, `CONTENT_RETENTION`), shared with the help-center importer.
+
+## Content Extraction Strategy
+
+Selection happens in `main_content.select_main_node()`, shared with the help-center
+article importer: it picks the **deepest container that still holds at least
+`CONTENT_RETENTION` (90%) of the page's non-boilerplate text**, which resolves to
+`<main>` or the content `<div>` where one exists and to `<body>` otherwise.
+Navigation chrome (`nav`, `header`, `footer`, `aside`) is excluded when measuring,
+so a page whose only text is a phone number and a login link reads as empty.
+
+This replaced a walk that returned the *first* candidate clearing
+`min_content_length`. That floor is not a quality bar — a hero banner clears it —
+so a page whose first `<article>` was a 108-character search widget indexed the
+widget and discarded 4,500 characters. Choosing by how much of the page a
+container holds cannot make that mistake.
+
+If the selected node yields less than `min_content_length`, the page is thin or
+JavaScript-rendered and these fallbacks run in order:
+
+1. **Density-based extraction**: Analyzes paragraph density to identify content-rich areas.
+2. **Meaningful-element extraction**: Collects headings, paragraphs, lists and tables.
+3. **Body fallback**: The cleaned body content.
+4. **Last resort**: All text in the document.
+
+Falling through to these is also what routes a JavaScript-rendered page to the
+Crawl4AI browser fallback in `_process_url` — so boilerplate must never be
+allowed to clear the floor on its own.
+
+## Testing
+
+The system includes comprehensive unit tests to ensure proper functionality:
+
+```bash
+# Activate the Python environment
+conda activate python3128
+
+# Run the tests
+python -m pytest backend/tests/knowledge/test_enhanced_website_reader.py -v
+python -m pytest backend/tests/knowledge/test_enhanced_website_kb.py -v
+python -m pytest backend/tests/knowledge/test_optimized_pgvector.py -v
+```