fix(brand-profile): bind Wikipedia/Wikidata lookups to a site-validated entity (LLMO-6580)

README.md

@@ -5,37 +5,47 @@
 SpaceCat Task Processor is a Node.js service that processes messages from the AWS SQS queue `SPACECAT-TASK-PROCESSOR-JOBS`. Based on the `type` field in each message, it dispatches the message to the appropriate handler for processing various site-related tasks.
 
 ## Features
+
 - Receives and processes messages from SQS
 - Supports multiple task types via modular handlers
 - Built-in handlers for audit status, demo URL preparation, generic agent execution, and Slack notifications
 - Extensible and easy to add new handlers
 
 ## Handlers
+
 - **opportunity-status-processor**: Checks and reports status audits for a site
 - **demo-url-processor**: Prepares and shares a demo URL for a site
 - **agent-executor**: Runs registered AI/LLM agents (e.g., the brand-profile agent) asynchronously after onboarding flows
 - **slack-notify**: Sends Slack notifications (text or block messages) from workflows
 
 ## Setup
+
 1. Clone the repository
 2. Install dependencies:
+
    ```sh
    npm install
    ```
+
 3. Configure AWS credentials and environment variables as needed
 
 ## Usage
+
 - The service is designed to run as a serverless function or background worker.
 - It can be invoked in two ways:
   - **SQS mode:** listens to the `SPACECAT-TASK-PROCESSOR-JOBS` queue and processes messages automatically (default path for existing workflows).
   - **Direct mode:** the Lambda entrypoint auto-detects single-message payloads (e.g., from AWS Step Functions) and executes the corresponding handler synchronously. This is used by the new agent workflows to obtain immediate results before triggering follow-up actions.
 
 ## Development
+
 - To run tests:
+
   ```sh
   npm test
   ```
+
 - To run the optional brand-profile integration test (requires Azure OpenAI env variables):
+
   ```sh
   npm run test:brand-profile-it
   ```
@@ -52,13 +62,32 @@ The `agent-executor` (and the provided brand-profile agent) rely on the Azure Op
 | `AZURE_COMPLETION_DEPLOYMENT` | Deployment/model name (e.g., `gpt-4o`) |
 
 When invoking the integration test, you can also set `BRAND_PROFILE_TEST_BASE_URL` to control which site is analyzed and `BRAND_PROFILE_IT_FULL=1` to print the complete agent response (otherwise the preview is truncated for readability).
+
+#### Brand-profile entity validation (LLMO-6580)
+
+The brand-profile product and competitor-summary paths bind every Wikipedia/Wikidata lookup to an entity that is validated against the customer's site by a strong P856 (official-website host) match, so a foreign entity's catalogue can never be attached to a customer.
+
+- **Brand-name resolution** (`services/brand-resolver.js`) turns the base profile and site URL into a display name plus the site's registrable domain. It never *derives* a brand name from a bare 2-3 letter acronym or a `dev`/`www`/`store`/`support` subdomain label as high confidence (an explicit `brand_name` supplied by the base profile is trusted as given). The registrable domain is the signal the entity validation compares against.
+- **Entity binding** (`services/wikipedia.js`): `findValidatedWikidataEntity` searches Wikidata by name but keeps a candidate **only** if its official-website host (claim P856) shares the site's registrable domain — this is the sole accepted signal; there is no by-name / label / alias fallback, and a bare public-suffix registrable domain (e.g. `co.uk`) never matches. Fallback article text is fetched by the validated entity's **exact** English Wikipedia sitelink title (`fetchWikipediaExtractByTitle`), never by a decoupled `opensearch "<name> company"` query. If no candidate's P856 host matches, the pipeline produces **no** Wikipedia/Wikidata products.
+
+| Variable | Default | Purpose |
+| --- | --- | --- |
+| `BRAND_PROFILE_ENABLE_WIKI_PRODUCTS` | `false` | Kill-switch for the entire Wikipedia/Wikidata product + competitor-summary path. When `false`, `extractProducts` returns an empty result (`products_metadata.source = "disabled"`) and no validated summary is fetched; sitemap-based product extraction and the rest of the profile still run. Ship `false` for net-new runs until the P0-a scrub and P2 backfill complete, then flip to `true`. Read from Vault per-service config (`dx_mysticat/{env}/task-processor`). |
+
+`products_metadata.source` terminal values: `sitemap`, `wikidata`, `hybrid`, `wikipedia_llm`, `disabled`, `none_no_validated_entity` (no P856-validated entity), and the pre-existing `none`/`sitemap_*` states. Additive provenance fields: `source_entity_label`, `source_wikipedia_title`, and `validation` (`p856`).
+
+**Persist guard:** `persist()` never overwrites a stored brand profile whose `products_metadata.source == "manual-curated"` — the curated `products`/`products_metadata` are preserved while all other fields update. This protects the hand-curated blocks during the P2 regeneration sweep.
+
 - To lint code:
+
   ```sh
   npm run lint
   ```
 
 ## Extending
+
 To add a new handler:
+
 1. Create a new folder in `src/` for your handler.
 2. Export your handler function.
 3. Add it to the handler mapping in `src/index.js`.
@@ -67,6 +96,7 @@ To add a new handler:
 For more details, see the documentation in `src/README.md`.
 
 ## Status
+
 [![codecov](https://img.shields.io/codecov/c/github/adobe-rnd/spacecat-task-processor.svg)](https://codecov.io/gh/adobe-rnd/spacecat-task-processor)
 [![CircleCI](https://img.shields.io/circleci/project/github/adobe-rnd/spacecat-audit-worker.svg)](https://circleci.com/gh/adobe-rnd/spacecat-task-processor)
 [![GitHub license](https://img.shields.io/github/license/adobe-rnd/spacecat-task-processor.svg)](https://github.com/adobe-rnd/spacecat-task-processor/blob/master/LICENSE.txt)
@@ -77,7 +107,7 @@ For more details, see the documentation in `src/README.md`.
 ## Installation
 
 ```bash
-$ npm install @adobe/spacecat-task-processor
+npm install @adobe/spacecat-task-processor
 ```
 
 ## Usage
@@ -89,19 +119,19 @@ See the [API documentation](docs/API.md).
 ### Build
 
 ```bash
-$ npm install
+npm install
 ```
 
 ### Test
 
 ```bash
-$ npm test
+npm test
 ```
 
 ### Lint
 
 ```bash
-$ npm run lint
+npm run lint
 ```
 
 ## Message Body Formats
@@ -141,6 +171,7 @@ When the AWS Step Functions Agent Workflow invokes the Lambda directly, it sends
 ```
 
 Field descriptions:
+
 - `agentId` *(required)* – must match a registered agent (e.g., `brand-profile`).
 - `siteId` *(required)* – kept at the envelope level for logging/metrics. Agents can still read it from the message passed into `agent.persist`.
 - `context` *(required)* – forwarded to `agent.run`. At minimum it must include `baseURL`; additional agent-specific params live here.

src/agents/brand-profile/index.js

@@ -26,6 +26,7 @@ import { createCompetitorInferenceService } from './services/competitor-inferenc
 import { createPersonaInferenceService } from './services/persona-inference.js';
 import { createProductExtractorService } from './services/product-extractor.js';
 import { createWikipediaService } from './services/wikipedia.js';
+import { resolveBrandName } from './services/brand-resolver.js';
 
 /**
  * Call the model with system and user prompts.
@@ -49,40 +50,6 @@ async function callModel({
   }
 }
 
-/**
- * Extract brand name from base profile or URL.
- * @param {object} baseProfile - Base profile from initial LLM call
- * @param {string} baseURL - Site base URL
- * @returns {string} Brand name
- */
-function extractBrandName(baseProfile, baseURL) {
-  // Try to get brand name from profile
-  if (baseProfile?.main_profile?.brand_name) {
-    return baseProfile.main_profile.brand_name;
-  }
-
-  // Try competitive_context
-  if (baseProfile?.competitive_context?.brand_name) {
-    return baseProfile.competitive_context.brand_name;
-  }
-
-  // Fall back to domain extraction
-  try {
-    const url = new URL(baseURL);
-    const parts = url.hostname.split('.');
-    // Remove www and TLD
-    const domainParts = parts.filter((p) => p !== 'www' && p.length > 2);
-    if (domainParts.length > 0) {
-      return domainParts[0].charAt(0).toUpperCase() + domainParts[0].slice(1);
-    }
-  /* c8 ignore next 3 */
-  } catch {
-    // Ignore URL parse errors
-  }
-
-  return 'Unknown Brand';
-}
-
 /**
  * Extract industry from base profile.
  * @param {object} baseProfile - Base profile from initial LLM call
@@ -152,10 +119,17 @@ async function run(context, env, log) {
   }
 
   // Extract key fields from base profile for enhanced inference
-  const brandName = extractBrandName(baseProfile, baseURL);
+  const {
+    name: brandName,
+    registrableDomain,
+  } = await resolveBrandName(baseProfile, baseURL, log);
   const industry = extractIndustry(baseProfile);
   const targetAudience = extractTargetAudience(baseProfile);
 
+  // LLMO-6580 kill-switch: the entire Wikipedia/Wikidata product + competitor-summary
+  // path stays OFF unless explicitly enabled, until the P2 backfill is validated.
+  const enableWikiProducts = env.BRAND_PROFILE_ENABLE_WIKI_PRODUCTS === 'true';
+
   log.info(`brand-profile: enhancing profile for "${brandName}" in "${industry}"`);
 
   // Initialize services
@@ -200,9 +174,16 @@ async function run(context, env, log) {
     competitorsSource = 'llmo';
   } else {
     log.info('brand-profile: inferring competitors');
-    // Optionally fetch Wikipedia summary for better competitor inference
-    const wikiResult = await wikipediaService.fetchSummary(`${brandName} company`);
-    const wikiSummary = wikiResult?.summary || '';
+    // Optionally fetch a VALIDATED Wikipedia summary (entity bound to the site) for
+    // better competitor inference. Gated by the kill-switch; null degrades gracefully.
+    let wikiSummary = '';
+    if (enableWikiProducts) {
+      const wikiResult = await wikipediaService.fetchValidatedSummary({
+        brandName,
+        registrableDomain,
+      });
+      wikiSummary = wikiResult?.summary || '';
+    }
 
     const competitorResult = await competitorService.inferCompetitors({
       brandName,
@@ -232,9 +213,13 @@ async function run(context, env, log) {
     log.info(`brand-profile: using sitemap for product extraction: ${sitemapUrl}`);
     productsResult = await productService.extractFromSitemap(sitemapUrl, brandName);
   } else {
-    // Use Wikipedia/Wikidata extraction
-    const wikiText = await wikipediaService.fetchFullText(`${brandName} company`, 12000);
-    productsResult = await productService.extractProducts(brandName, wikiText);
+    // Entity-bound Wikipedia/Wikidata extraction. The fetch now happens inside
+    // extractProducts, bound to an entity validated against the site.
+    productsResult = await productService.extractProducts({
+      brandName,
+      registrableDomain,
+      enableWikiProducts,
+    });
   }
 
   // Assemble the enhanced profile
@@ -311,7 +296,18 @@ async function persist(message, context, result) {
   const baseURL = site.getBaseURL();
   const before = cfg.getBrandProfile?.() || {};
   const beforeHash = before?.contentHash || null;
-  cfg.updateBrandProfile(result);
+
+  // LLMO-6580: never overwrite a hand-curated product catalogue. Phase-1 wrote ~20
+  // `products_metadata.source == "manual-curated"` blocks in prod; the fixed pipeline
+  // and the P2 backfill MUST preserve them. Everything else still updates.
+  const curated = before?.products_metadata?.source === 'manual-curated';
+  const toPersist = curated
+    ? { ...result, products: before.products, products_metadata: before.products_metadata }
+    : result;
+  if (curated) {
+    log.info('brand-profile persist: preserving manual-curated products', { siteId });
+  }
+  cfg.updateBrandProfile(toPersist);
   const after = cfg.getBrandProfile?.() || {};
   const afterHash = after?.contentHash || null;
   const changed = beforeHash !== afterHash;