Start ml1211

[chpeu]

PR Type

Bug fix, Enhancement, Tests, Documentation

---

Description

• PostgreSQL DataLogger robustness: Enhanced data serialization with safe type conversion, improved timezone handling using timezone.utc, reduced batch intervals (batch_size: 50→10, flush_interval: 5.0→2.0s), and added comprehensive fallback chains for price and numeric value extraction

• Scalability metrics integration: Added support for recent_volume, vol5, vol15, and scalability_score across datalogger, scanner loop, and database schema with corresponding SQL migration

• Data export and management: Implemented Excel export endpoint (/api/datalogger/export/excel) and database reset endpoint (/api/datalogger/reset) with frontend UI controls

• Indicator availability improvements: Fixed analyzer to return indicators and scores in rejection scenarios (swing structure absent, spread/orderbook rejection) instead of None/reason-only responses

• Logging enhancements: Added SimplePGLogger for ultra-simple direct PostgreSQL logging without batching, periodic flush task (30s interval), and comprehensive error handling with traceback information

• Bug fixes: Fixed timezone handling in position manager, book depth validation in scanner, profit factor calculation formula in frontend, SL/TP distance calculations using absolute values

• Test updates: Corrected cursor execute call assertions to account for session initialization calls

• Documentation: Added comprehensive guides for scalable parameters, database schema analysis, and diagnostic queries

---

Diagram Walkthrough

flowchart LR
  A["Scanner & Analyzer"] -->|"Extract indicators & prices"| B["PostgreSQL DataLogger"]
  B -->|"Safe serialization & validation"| C["PostgreSQL Database"]
  C -->|"Query & export"| D["Excel Export"]
  A -->|"Scalability metrics"| B
  E["SimplePGLogger"] -->|"Direct logging"| B
  F["Frontend UI"] -->|"Export/Reset"| G["API Endpoints"]
  G -->|"Manage data"| C

Loading

File Walkthrough

	Relevant files
Bug fix	6 files postgresql_datalogger.py PostgreSQL DataLogger robustness improvements and data serialization fixes core/postgresql_datalogger.py Added _extract_numeric_value() helper function to safely extract numeric values from dicts or other types Added serialize_config_safe() function to convert non-JSON-serializable config objects (functions, datetime, Decimal, custom objects) to JSON-safe formats Reduced default batch_size from 50 to 10 and batch_flush_interval from 5.0 to 2.0 seconds for more frequent flushing Fixed timezone handling by using timezone.utc for all datetime.now() calls to ensure PostgreSQL TIMESTAMPTZ compatibility Enhanced log_scan() with multiple fallbacks to extract price from nested dicts and added validation to prevent NULL prices Added support for scalability metrics (recent_volume, vol5, vol15, scalability_score) in scan logging Improved log_opportunity() and log_trade() with robust type checking and numeric value extraction for all numeric fields Enhanced _batch_insert_scans() and _batch_insert_opportunities() with price validation and fallback resolution for scan_id Added comprehensive logging and error handling throughout with informative debug messages +454/-86 analyzer.py Improve indicator availability in rejection scenarios        core/analyzer.py Fixed return value when swing structure is absent to include indicators and scores instead of just reason Added long_score and short_score to the main indicators dictionary for fallback usage Modified spread rejection to return dict with indicators and metadata instead of None Modified orderbook rejection to return dict with indicators and metadata instead of None Changed indicator extraction logic to include rejected analyses (removed 'reason' check filter) Enhanced final rejection case to return complete analysis data with indicators for all timeframes +221/-16 position_manager.py Fix timezone handling and improve setup monitoring              core/position_manager.py Added timezone import for UTC timestamp handling Enhanced error logging for missing _last_setup with detailed bug tracking information Fixed timestamp generation to use timezone.utc for PostgreSQL TIMESTAMPTZ compatibility Added serialize_config_safe() function calls for safe configuration serialization Updated config snapshot building to use safe serialization for all config dictionaries +30/-17  scanner.py Add book depth validation to prevent slippage errors          core/scanner.py Added critical filter check for book_depth <= 0 to prevent invalid slippage calculations Updated comment to clarify spread range (0.001% to 0.02%) and bookDepth requirement Enhanced filter logic to explicitly validate book depth before score calculation +3/-2      stats.js Fix profit factor calculation formula                                        frontend/src/lib/stores/stats.js Fixed Profit Factor calculation to use sum of profits divided by absolute sum of losses Changed from incorrect formula (wins × best_trade) / (losses × worst_trade) to correct formula Updated to derive from tradeHistory instead of stats for accurate per-trade calculation Added filtering to separate profitable and losing trades before summing Returns '∞' for infinite profit factor and '0.00' for zero loss scenarios +15/-6    position.js Fix SL and TP distance calculations                                            frontend/src/lib/stores/position.js Fixed slDistance calculation to use absolute value for consistent positive distance display Fixed tpDistance calculation to use absolute value for consistent positive distance display Updated comments to clarify that distances are always positive percentages +6/-4
Enhancement	4 files main.py Enhanced PostgreSQL logging integration and data export capabilities main.py Added global _simple_logger variable for ultra-simple logging without batch mode for debugging Enhanced scan_pair_for_setup() with priority-based indicator extraction from analysis_1m and analysis_5m nested objects Added comprehensive PostgreSQL logging for scans and opportunities with multiple price fallbacks and scalability metrics Implemented periodic flush task for PostgreSQL buffers (every 30 seconds) to ensure timely data persistence Added SimplePGLogger initialization in init_instances() for simplified debugging logging Added two new API endpoints: /api/datalogger/export/excel for Excel export and /api/datalogger/reset for database reset Enhanced market data collection with scalability parameters (recent_volume, vol5, vol15, scalability_score) Improved error handling and logging throughout with detailed traceback information +830/-115 scanner_loop.py Scanner loop PostgreSQL integration and data extraction improvements core/callbacks/scanner_loop.py Modified get_pg_datalogger() to implement force initialization pattern that creates PostgreSQL DataLogger instance if not injected Added SimplePGLogger import and global instance for simplified logging Enhanced indicator extraction logic with priority-based fallbacks from analysis_1m and analysis_5m nested objects Added comprehensive price extraction with multiple fallbacks to prevent NULL values in database Integrated scalability metrics (recent_volume, vol5, vol15, scalability_score) with aliases for compatibility Added detailed logging and error handling for PostgreSQL operations with traceback information Improved _batch_insert_scans() and _batch_insert_opportunities() with robust data validation +278/-76 simple_pg_logger.py Add simple PostgreSQL logger for scan data                              core/simple_pg_logger.py New file implementing ultra-simple PostgreSQL logger without batching complexity Direct insert operations for scan logging with minimal required fields Comprehensive fallback chain for extracting RSI and score_total from various data structures Connection management with automatic reconnection on failure and rollback on errors Debug logging to trace data extraction through multiple fallback levels +241/-0  VariablesPanel.svelte Add Excel export and database reset functionality                frontend/src/lib/components/VariablesPanel.svelte Added state variables exportingExcel and resettingDB for tracking operation status Implemented exportExcel() function to download datalogger data as Excel file Implemented resetDatabase() function with double confirmation for PostgreSQL data deletion Added two new buttons: "Export Excel" and "Reset DB" with appropriate styling Added CSS styles for .btn-export and .btn-danger buttons with hover effects +126/-0
Tests	1 files test_postgresql_datalogger.py Correct test assertions for cursor execute calls                  tests/test_postgresql_datalogger.py Fixed assertion in test_log_scan_error to expect 2 cursor.execute calls instead of 1 Fixed assertion in test_log_market_context to expect 2 cursor.execute calls instead of 1 Fixed assertion in test_log_trade to expect 2 cursor.execute calls instead of 1 Added comments explaining the expected call count (1 for session, 1 for actual operation) +9/-6
Configuration changes	2 files apply_migration_scalability.ps1 Add PowerShell migration script for scalability                    database/apply_migration_scalability.ps1 New PowerShell script for applying scalability parameters migration Prompts for PostgreSQL password securely and sets PGPASSWORD environment variable Executes migration SQL file and verifies column creation with detailed output Provides visual feedback with colored console output and status indicators Includes error handling and cleanup of sensitive environment variables +86/-0    migration_add_scalability_params.sql Add scalability parameters to database schema                        database/migration_add_scalability_params.sql New SQL migration adding 4 columns to scan_logs table (recent_volume, vol5, vol15, scalability_score) Adds 7 columns to trades table for entry/exit scalability parameters Creates indexes on scalability_score columns for ML analysis performance Includes verification logic to ensure all columns were created successfully Adds detailed comments explaining each column's purpose +113/-0
Documentation	4 files PARAMETRES_VARIABLES_SCAN_SCALABLES.md Document scalable pair scan parameters                                      PARAMETRES_VARIABLES_SCAN_SCALABLES.md New comprehensive documentation of scalable pair scan parameters Detailed table of parameters with types, descriptions, and sources Formulas for volatility calculation and scalability score computation Complete structure example of a scalable pair object Usage examples in code and PostgreSQL logging Default values and update frequency information +239/-0  ANALYSE_PARAMETRES_SCAN_SCALABLES.md Analyze scalable parameters against database schema            ANALYSE_PARAMETRES_SCAN_SCALABLES.md New analysis document comparing scan parameters to SQL schema Identifies 7 present parameters and 3 missing parameters in database Provides recommendations for handling missing parameters (recentVolume, vol5, vol15, score) Suggests using existing alternatives (atr_pct, volume_1m) as proxies Includes summary table and conclusion about schema completeness +166/-0  diagnostic_opportunities.sql Add diagnostic queries for opportunities analysis                database/diagnostic_opportunities.sql New diagnostic SQL queries to investigate missing opportunities Checks scan_logs for opportunities marked as true in various time windows Verifies opportunities table entries and their statuses Identifies scans marked as opportunities but missing from opportunities table Provides queries to inspect recent opportunity creation and status distribution +67/-0    COMPARAISON_PARAMETRES_SCALABLES_SCHEMA.md Compare scalable parameters with database schema                  COMPARAISON_PARAMETRES_SCALABLES_SCHEMA.md New comparison document between scalable scan parameters and SQL schema Detailed table showing which parameters are present/missing in each table Analysis of impact for missing parameters (vol5, vol15, recentVolume, score) Recommendations for using existing alternatives as proxies Conclusion that all essential parameters are already present and logged +213/-0

---

[qodo-code-review]

You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🔴	Unauthenticated destructive endpoint Description: The DELETE /api/datalogger/reset endpoint performs a full database wipe without authentication/authorization, allowing any caller to irreversibly delete all logs and trading data; protect this endpoint with strong auth and role checks or remove in production. main.py [4650-4724] Referred Code @app.delete("/api/datalogger/reset") async def reset_datalogger_db(): """ 🔥 Reset complet de la base de données PostgreSQL du datalogger ATTENTION: Cette opération supprime TOUTES les données (scans, opportunities, trades, etc.) Returns: Message de confirmation """ try: from core.callbacks.scanner_loop import get_pg_datalogger pg_datalogger = get_pg_datalogger() if not pg_datalogger or not pg_datalogger.enabled: return JSONResponse( {"error": "PostgreSQL DataLogger non disponible"}, status_code=503 ) conn = pg_datalogger._get_connection() ... (clipped 54 lines)
⚪	Unauthenticated background tasks Description: Periodic background tasks call pg_datalogger._flush_buffers(force=True) and log_market_context without authentication/authorization or cancellation handling, which if exposed in a multi-tenant or internet-facing environment could allow abuse of resources or unintended data writes; ensure these tasks are guarded by configuration flags and not started in untrusted deployments. main.py [1815-1899] Referred Code logger.info("✅ PostgreSQL DataLogger initialisé") # Injecter dans scanner_loop from core.callbacks.scanner_loop import set_pg_datalogger set_pg_datalogger(pg_datalogger) logger.info("✅ PostgreSQL DataLogger injecté dans scanner_loop") except ImportError as e: logger.debug(f"ℹ️ PostgreSQL DataLogger non disponible: {e}") except Exception as e: logger.warning(f"⚠️ Erreur initialisation PostgreSQL DataLogger: {e}") pg_datalogger = None # 🔥 PHASE 3: Créer tâche périodique pour logging contexte marché if pg_datalogger and pg_datalogger.enabled: async def log_market_context_periodic(): """Tâche périodique pour logger le contexte marché""" while True: try: await asyncio.sleep(300) # Toutes les 5 minutes if pg_datalogger and pg_datalogger.enabled: try: # Récupérer prix BTC/ETH ... (clipped 64 lines)
	Unauthenticated data export Description: The Excel export endpoint returns large query results (up to 10k rows) from PostgreSQL without authentication and minimal input validation, enabling potential information disclosure to unauthenticated users and resource exhaustion via repeated requests; add auth and rate limiting, and validate/limit date ranges. main.py [4447-4632] Referred Code @app.get("/api/datalogger/export/excel") async def export_datalogger_excel( start_date: Optional[str] = None, end_date: Optional[str] = None ): """ 🔥 Export des données du datalogger en Excel (.xlsx) Args: start_date: Date début (YYYY-MM-DD) - optionnel end_date: Date fin (YYYY-MM-DD) - optionnel Returns: Fichier Excel (.xlsx) avec plusieurs onglets (scans, opportunities, trades) """ try: from core.callbacks.scanner_loop import get_pg_datalogger pg_datalogger = get_pg_datalogger() if not pg_datalogger or not pg_datalogger.enabled: return JSONResponse( ... (clipped 165 lines)
	Sensitive operational logging Description: Flush routine logs detailed buffer counts and errors; while helpful for debugging, in production these logs could leak operational details if logs are exposed—ensure logging level is appropriate and sensitive information (symbols, counts) is not exposed to untrusted log consumers. postgresql_datalogger.py [1377-1406] Referred Code return with self.buffer_lock: scans_count = len(self.scan_buffer) opportunities_count = len(self.opportunity_buffer) # Flush scans if self.scan_buffer: try: logger.info(f"🔄 Flush {scans_count} scan(s) vers PostgreSQL (force={force})") self._batch_insert_scans(list(self.scan_buffer)) self.scan_buffer.clear() logger.info(f"✅ {scans_count} scan(s) flushés avec succès") except Exception as e: logger.error(f"❌ Erreur flush scans: {e}") # Flush opportunities if self.opportunity_buffer: try: logger.info(f"🔄 Flush {opportunities_count} opportunité(s) vers PostgreSQL (force={force})") self._batch_insert_opportunities(list(self.opportunity_buffer)) ... (clipped 9 lines)
	Weak data correlation Description: In batch opportunity insert, when scan_id is missing it queries the latest matching scan within 5 minutes by symbol without additional constraints, which can incorrectly associate opportunities and enable data spoofing if symbol collisions occur; include session_id and stronger correlation keys to prevent mis-association. postgresql_datalogger.py [1662-1699] Referred Code tp_price = opp_data.get('tp_price') or opp_data.get('tp_suggested') sl_price = opp_data.get('sl_price') or opp_data.get('sl_suggested') tp_sl_mode = opp_data.get('tp_sl_mode', 'FIXE') setup_score = opp_data.get('setup_score') direction = opp_data.get('direction') status = opp_data.get('status', 'PENDING') # S'assurer que les prix sont des nombres, pas des dicts if isinstance(entry_price, dict): entry_price = entry_price.get('price') or entry_price.get('value') if isinstance(tp_price, dict): tp_price = tp_price.get('price') or tp_price.get('value') if isinstance(sl_price, dict): sl_price = sl_price.get('price') or sl_price.get('value') if isinstance(tp_sl_mode, dict): tp_sl_mode = tp_sl_mode.get('mode') or 'FIXE' # S'assurer que setup_score est un nombre, pas un dict if isinstance(setup_score, dict): setup_score = setup_score.get('score') or setup_score.get('value') or setup_score.get('totalScore') # S'assurer que direction est une string, pas un dict if isinstance(direction, dict): ... (clipped 17 lines)
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Missing audit context: New logging paths add many info-level messages but do not consistently include user/session identity and outcomes for critical actions, and SimplePGLogger writes directly without structured audit context. Referred Code # Vérifier que _simple_logger est défini et accessible try: logger.info(f"🔍 DEBUG Simple Logger pour {symbol}: _simple_logger existe, enabled={getattr(_simple_logger, 'enabled', 'ATTRIBUT_MANQUANT')}") except NameError: logger.error(f"❌ _simple_logger n'est pas défini pour {symbol}") _simple_logger = None except Exception as e: logger.error(f"❌ Erreur accès _simple_logger pour {symbol}: {e}") _simple_logger = None if _simple_logger and hasattr(_simple_logger, 'enabled') and _simple_logger.enabled: logger.info(f"📝 Tentative log_scan_simple pour {symbol}") result = _simple_logger.log_scan_simple(symbol, { 'market_data': {'price': analysis.get('price') if analysis else None}, 'indicators_1m': analysis.get('indicators_1m', {}) if analysis else {}, 'scores': {'score_total': analysis.get('score_total') if analysis else None}, 'is_opportunity': bool(analysis and 'direction' in analysis and ('entry' in analysis or 'price' in analysis)) if analysis else False }) logger.info(f"📝 Résultat log_scan_simple pour {symbol}: {result}") else: logger.warning(f"⚠️ Simple Logger désactivé pour {symbol}") ... (clipped 2 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Abbreviations used: The global names _pg_datalogger, _pg_datalogger_instance, and _sio introduce abbreviations that may reduce clarity though likely conventional in this codebase. Referred Code _sio = None # 🔥 MIGRATION COMPLÈTE: Gardé pour compatibilité, mais utiliser _ws_manager _ws_manager = None # 🔥 MIGRATION COMPLÈTE: WebSocket natif _scanner_lock = None _pg_datalogger = None # 🔥 PHASE 1: PostgreSQL DataLogger pour ML (injection) _pg_datalogger_instance = None # 🔥 Force Initialization: Instance créée automatiquement _simple_logger = SimplePGLogger() # 🔥 Simple Logger: Logger ultra-simple sans batch Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Partial reconnection: On DB errors SimplePGLogger logs and attempts rollback/reconnect but does not retry the failed insert or surface actionable error types, risking silent data loss. Referred Code except Exception as e: logger.error(f"❌ Erreur log_scan pour {symbol}: {e}") import traceback logger.debug(f"Traceback: {traceback.format_exc()}") # 🔥 FIX: Rollback en cas d'erreur pour éviter que la transaction reste en état d'erreur try: if hasattr(self, 'conn') and not self.conn.closed: self.conn.rollback() logger.debug(f"🔄 Rollback effectué pour {symbol}") except Exception as rollback_error: logger.error(f"❌ Erreur rollback: {rollback_error}") # Essayer de reconnecter si la connexion est fermée try: if hasattr(self, 'conn') and (self.conn.closed if hasattr(self.conn, 'closed') else False): logger.info("🔄 Tentative de reconnexion...") self.conn = psycopg2.connect( host=os.getenv('POSTGRES_HOST', 'localhost'), port=int(os.getenv('POSTGRES_PORT', '5432')), dbname=os.getenv('POSTGRES_DB', 'trade_cursor_ml'), user=os.getenv('POSTGRES_USER', 'postgres'), password=os.getenv('POSTGRES_PASSWORD', '') ... (clipped 6 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Stack in logs: Exceptions are logged with full traceback at info/error paths (e.g., Simple Logger and PostgreSQL logging sections) which could expose internal details if logs are user-visible. Referred Code except Exception as e: logger.error(f"❌ Erreur Simple Logger pour {symbol}: {e}") import traceback logger.debug(f"Traceback: {traceback.format_exc()}") Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Verbose logging: New info/debug logs include potentially sensitive operational data (symbols, prices, config serialization issues) without explicit scrubbing and with emoji noise impacting structure. Referred Code now = datetime.now(timezone.utc) time_since_flush = (now - self.last_flush_time).total_seconds() should_flush = force or ( len(self.scan_buffer) >= self.batch_size or len(self.opportunity_buffer) >= self.batch_size or time_since_flush >= self.batch_flush_interval ) if not should_flush: return with self.buffer_lock: scans_count = len(self.scan_buffer) opportunities_count = len(self.opportunity_buffer) # Flush scans if self.scan_buffer: try: logger.info(f"🔄 Flush {scans_count} scan(s) vers PostgreSQL (force={force})") self._batch_insert_scans(list(self.scan_buffer)) self.scan_buffer.clear() ... (clipped 18 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Direct DB inserts: SimplePGLogger performs direct inserts using environment-based connection without explicit input validation or structured parameterization beyond basic psycopg2 placeholders, and reconnect logic may leave inconsistent state. Referred Code self.enabled = True try: self.conn = psycopg2.connect( host=os.getenv('POSTGRES_HOST', 'localhost'), port=int(os.getenv('POSTGRES_PORT', '5432')), dbname=os.getenv('POSTGRES_DB', 'trade_cursor_ml'), user=os.getenv('POSTGRES_USER', 'postgres'), password=os.getenv('POSTGRES_PASSWORD', '') ) logger.info("✅ SimplePGLogger connecté") except Exception as e: logger.error(f"❌ Erreur connexion: {e}") self.enabled = False def log_scan_simple(self, symbol: str, scan_data: Dict[str, Any]) -> bool: Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🔴	Unauthenticated destructive endpoint Description: The database reset endpoint allows deletion of all PostgreSQL datalogger tables without authentication or CSRF protection, enabling trivial data destruction via unauthenticated DELETE requests. main.py [4650-4724] Referred Code @app.delete("/api/datalogger/reset") async def reset_datalogger_db(): """ 🔥 Reset complet de la base de données PostgreSQL du datalogger ATTENTION: Cette opération supprime TOUTES les données (scans, opportunities, trades, etc.) Returns: Message de confirmation """ try: from core.callbacks.scanner_loop import get_pg_datalogger pg_datalogger = get_pg_datalogger() if not pg_datalogger or not pg_datalogger.enabled: return JSONResponse( {"error": "PostgreSQL DataLogger non disponible"}, status_code=503 ) conn = pg_datalogger._get_connection() ... (clipped 54 lines)
⚪	Information disclosure Description: Excessive logging of operational details (buffer sizes, flush counts, and errors) at info level can disclose internal state and processing volumes to logs, potentially leaking sensitive trading activity or enabling traffic analysis if logs are exposed. postgresql_datalogger.py [1377-1406] Referred Code return with self.buffer_lock: scans_count = len(self.scan_buffer) opportunities_count = len(self.opportunity_buffer) # Flush scans if self.scan_buffer: try: logger.info(f"🔄 Flush {scans_count} scan(s) vers PostgreSQL (force={force})") self._batch_insert_scans(list(self.scan_buffer)) self.scan_buffer.clear() logger.info(f"✅ {scans_count} scan(s) flushés avec succès") except Exception as e: logger.error(f"❌ Erreur flush scans: {e}") # Flush opportunities if self.opportunity_buffer: try: logger.info(f"🔄 Flush {opportunities_count} opportunité(s) vers PostgreSQL (force={force})") self._batch_insert_opportunities(list(self.opportunity_buffer)) ... (clipped 9 lines)
	Inadequate access control Description: The Excel export endpoint streams large query results without authentication/authorization checks, enabling unauthenticated data exfiltration of scans, opportunities, and trades if the API is exposed. main.py [4447-4632] Referred Code @app.get("/api/datalogger/export/excel") async def export_datalogger_excel( start_date: Optional[str] = None, end_date: Optional[str] = None ): """ 🔥 Export des données du datalogger en Excel (.xlsx) Args: start_date: Date début (YYYY-MM-DD) - optionnel end_date: Date fin (YYYY-MM-DD) - optionnel Returns: Fichier Excel (.xlsx) avec plusieurs onglets (scans, opportunities, trades) """ try: from core.callbacks.scanner_loop import get_pg_datalogger pg_datalogger = get_pg_datalogger() if not pg_datalogger or not pg_datalogger.enabled: return JSONResponse( ... (clipped 165 lines)
	Race condition data mixup Description: Resolving scan_id by querying recent rows then using it for batch insert introduces a race condition that could associate opportunities with the wrong scan in concurrent environments, enabling data integrity corruption or confused-deputy style mistakes. postgresql_datalogger.py [1619-1643] Referred Code values = [] for opp_item in opportunities: scan_id = opp_item['scan_id'] symbol = opp_item['symbol'] # 🔥 FIX: Si scan_id = 0 (temporaire), essayer de le résoudre depuis scan_logs if scan_id == 0 or scan_id is None: # Chercher le scan_log_id correspondant dans scan_logs # en utilisant symbol et timestamp récent (dernières 5 minutes) resolve_query = """ SELECT id FROM scan_logs WHERE symbol = %s AND is_opportunity = true AND timestamp > NOW() - INTERVAL '5 minutes' ORDER BY timestamp DESC LIMIT 1 """ cursor.execute(resolve_query, (symbol,)) result = cursor.fetchone() if result: scan_id = result[0] ... (clipped 4 lines)
	Time spoofing risk Description: Using client-side timestamps via datetime.now(timezone.utc) instead of database-side NOW() for market_context can allow time spoofing if callers control the host clock, affecting auditability and temporal integrity. postgresql_datalogger.py [824-829] Referred Code RETURNING id """ # 🔥 FIX BUG #3: Utiliser timezone.utc pour PostgreSQL TIMESTAMPTZ now = datetime.now(timezone.utc) params = (
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Missing user context: New logging to PostgreSQL (e.g., log_scan, log_opportunity, SimplePGLogger inserts) lacks user/account identifiers and action attribution, making audit trails potentially incomplete. Referred Code } # Logger le scan (mode batch par défaut) logger.info(f"📝 Appel log_scan() pour {symbol}") scan_id = pg_datalogger.log_scan(symbol, scan_data, use_batch=True) logger.info(f"✅ log_scan() terminé pour {symbol} (scan_id={scan_id})") # Si c'est une opportunité, logger aussi dans opportunities # Note: En mode batch, scan_id est None, mais l'opportunité sera loggée # avec scan_id=None temporairement (sera mis à jour lors du flush) if scan_data['is_opportunity'] and analysis: opportunity_data = { 'status': 'PENDING', 'direction': analysis.get('direction'), 'setup_score': analysis.get('score_total'), 'conditions_matched': analysis.get('condition_types', []), 'entry_price': analysis.get('entry') or analysis.get('price'), 'tp_price': analysis.get('tp'), 'sl_price': analysis.get('sl'), 'size_usdt': None, # Sera calculé lors de l'ouverture 'risk_usdt': None, ... (clipped 16 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Emoji in comments: Several added comments and log messages use emojis and mixed-language notes which may reduce long-term maintainability and clarity for all contributors. Referred Code def _extract_numeric_value(value: Any) -> Optional[float]: """ 🔥 FIX: Extraire une valeur numérique depuis un dict ou autre type Args: value: Valeur à extraire (peut être dict, float, int, str, None) Returns: float ou None """ if value is None: return None # Si c'est déjà un nombre if isinstance(value, (int, float)): return float(value) # Si c'est un dict, essayer d'extraire une valeur numérique if isinstance(value, dict): # Essayer plusieurs clés communes for key in ['value', 'price', 'score', 'rsi', 'macd', 'adx', 'atr', 'volume', 'spread', 'balance', 'depth', 'vol5', 'vol15']: ... (clipped 1355 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Partial retries: SimplePGLogger handles errors with rollback and one reconnection attempt but lacks structured backoff, retry limits, and surfacing of failures to callers for robust recovery. Referred Code except Exception as e: logger.error(f"❌ Erreur log_scan pour {symbol}: {e}") import traceback logger.debug(f"Traceback: {traceback.format_exc()}") # 🔥 FIX: Rollback en cas d'erreur pour éviter que la transaction reste en état d'erreur try: if hasattr(self, 'conn') and not self.conn.closed: self.conn.rollback() logger.debug(f"🔄 Rollback effectué pour {symbol}") except Exception as rollback_error: logger.error(f"❌ Erreur rollback: {rollback_error}") # Essayer de reconnecter si la connexion est fermée try: if hasattr(self, 'conn') and (self.conn.closed if hasattr(self.conn, 'closed') else False): logger.info("🔄 Tentative de reconnexion...") self.conn = psycopg2.connect( host=os.getenv('POSTGRES_HOST', 'localhost'), port=int(os.getenv('POSTGRES_PORT', '5432')), dbname=os.getenv('POSTGRES_DB', 'trade_cursor_ml'), user=os.getenv('POSTGRES_USER', 'postgres'), password=os.getenv('POSTGRES_PASSWORD', '') ... (clipped 6 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Stack trace logs: Errors are logged with full tracebacks (e.g., logger.debug(traceback.format_exc())) which, if exposed to user-facing logs, could leak internal details. Referred Code logger.error(f"❌ Erreur logging PostgreSQL pour {symbol}: {e}") import traceback logger.debug(f"Traceback: {traceback.format_exc()}") Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Verbose logging: The SimplePGLogger emits detailed debug/info logs including parameter values that may include sensitive or proprietary trading data and should be reviewed for sanitization. Referred Code logger.debug(f"🔍 SimplePGLogger: Insert pour {symbol} avec params: (symbol={symbol}, price={price}, rsi_1m={rsi_1m}, score_total={score_total}, is_opportunity={scan_data.get('is_opportunity', False)})") cursor.execute(query, params) self.conn.commit() cursor.close() logger.info(f"✅ Scan loggé: {symbol}") return True Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Env-based creds: Database credentials are read directly from environment variables without validation or secure secret handling patterns visible in the diff. Referred Code self.conn = psycopg2.connect( host=os.getenv('POSTGRES_HOST', 'localhost'), port=int(os.getenv('POSTGRES_PORT', '5432')), dbname=os.getenv('POSTGRES_DB', 'trade_cursor_ml'), user=os.getenv('POSTGRES_USER', 'postgres'), password=os.getenv('POSTGRES_PASSWORD', '') ) Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
High-level	Refactor data handling to enforce consistency Instead of defensively cleaning inconsistent data with helpers like _extract_numeric_value, enforce a consistent data schema at the source for objects like scan_data and trade_data. This will simplify the code and improve robustness. Examples: core/postgresql_datalogger.py [30-71] def _extract_numeric_value(value: Any) -> Optional[float]: """ 🔥 FIX: Extraire une valeur numérique depuis un dict ou autre type Args: value: Valeur à extraire (peut être dict, float, int, str, None) Returns: float ou None """ ... (clipped 32 lines) core/postgresql_datalogger.py [999-1328] net_pnl_usdt_raw = trade_data.get('net_pnl_usdt', 0) net_pnl_usdt = _extract_numeric_value(net_pnl_usdt_raw) if net_pnl_usdt_raw is not None else 0 win = net_pnl_usdt > 0 if net_pnl_usdt is not None else None # Calculer tp_escalier_profits (somme des profits) tp_escalier_levels_hit = trade_data.get('tp_escalier_levels_hit', []) # 🔥 FIX: S'assurer que les profits sont des nombres, pas des dicts tp_escalier_profits = 0 if tp_escalier_levels_hit: for p in tp_escalier_levels_hit: ... (clipped 320 lines) Solution Walkthrough: Before: # In postgresql_datalogger.py def _extract_numeric_value(value): # Tries to get a float from a dict, str, int, etc. if isinstance(value, dict): for key in ['value', 'price', 'score', ...]: if key in value: # ... get nested value # ... more checks def log_trade(trade_data): # Extensive cleaning for every single value entry_price_raw = trade_data.get('entry_price') entry_price = _extract_numeric_value(entry_price_raw) entry_indicators_clean = {} for key, value in trade_data.get('entry_indicators', {}).items(): entry_indicators_clean[key] = _extract_numeric_value(value) # ... dozens of calls to _extract_numeric_value for each param params = (..., _extract_numeric_value(trade_data.get('gross_pnl_usdt')), ...) # ... After: # In a new models.py or similar from pydantic import BaseModel from typing import Optional class TradeData(BaseModel): symbol: str entry_price: float net_pnl_usdt: Optional[float] entry_indicators: Dict[str, Optional[float]] # ... all other fields with correct types # In position_manager.py (the data source) def close_position(...) -> TradeData: # ... logic to gather data raw_data = { ... } return TradeData(**raw_data) # Pydantic validates data at the source # In postgresql_datalogger.py def log_trade(trade_data: TradeData): # No more cleaning needed, data has a guaranteed structure params = ( trade_data.symbol, trade_data.entry_price, trade_data.net_pnl_usdt, # ... directly access attributes ) # ... execute query Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical architectural flaw where the PR adds complex, defensive data cleaning logic instead of enforcing data consistency at the source, which is a more robust and maintainable solution.	High
Possible issue	Fix N+1 query in batch insert Refactor the scan_id resolution logic in _batch_insert_opportunities to use a single query before the loop, fixing the N+1 query problem and improving performance. core/postgresql_datalogger.py [1624-1643] # 🔥 FIX: Si scan_id = 0 (temporaire), essayer de le résoudre depuis scan_logs -if scan_id == 0 or scan_id is None: - # Chercher le scan_log_id correspondant dans scan_logs - # en utilisant symbol et timestamp récent (dernières 5 minutes) +# This block is now outside the loop to avoid N+1 queries. +# It should be placed before the `for opp_item in opportunities:` loop. + +# Step 1: Identify opportunities needing scan_id resolution +opps_to_resolve = [opp for opp in opportunities if opp.get('scan_id') in (0, None)] +symbols_to_resolve = list(set(opp['symbol'] for opp in opps_to_resolve)) + +resolved_scan_ids = {} +if symbols_to_resolve: + # Step 2: Fetch the latest scan_id for each symbol in a single query resolve_query = """ - SELECT id FROM scan_logs - WHERE symbol = %s + SELECT DISTINCT ON (symbol) symbol, id + FROM scan_logs + WHERE symbol = ANY(%s) AND is_opportunity = true AND timestamp > NOW() - INTERVAL '5 minutes' - ORDER BY timestamp DESC - LIMIT 1 + ORDER BY symbol, timestamp DESC; """ - cursor.execute(resolve_query, (symbol,)) - result = cursor.fetchone() - if result: - scan_id = result[0] - logger.debug(f"✅ scan_id résolu pour {symbol}: {scan_id}") + cursor.execute(resolve_query, (symbols_to_resolve,)) + results = cursor.fetchall() + resolved_scan_ids = {row[0]: row[1] for row in results} + logger.info(f"✅ {len(resolved_scan_ids)} scan_id(s) résolus en une seule requête.") + +# Step 3: Update opportunities with resolved scan_ids (inside the loop) +# The following logic should replace the original block inside the loop. +if scan_id == 0 or scan_id is None: + resolved_id = resolved_scan_ids.get(symbol) + if resolved_id: + scan_id = resolved_id + logger.debug(f"✅ scan_id mis à jour pour {symbol}: {scan_id}") else: logger.warning(f"⚠️ Impossible de résoudre scan_id pour {symbol}, utilisation de 0") scan_id = 0 Apply / Chat Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical N+1 query performance issue in the _batch_insert_opportunities method and provides a robust, efficient solution using a single batch query, which significantly improves database performance.	High
	Prevent recursion error in serialization To prevent potential RecursionError in serialize_config_safe, add a mechanism to track visited objects during the recursive serialization to handle circular references. core/postgresql_datalogger.py [114-122] elif hasattr(value, '__dict__'): # Objet custom → essayer de sérialiser __dict__ récursivement - try: - serialized[key] = serialize_config_safe(value.__dict__) - logger.debug(f"🔧 Config key '{key}': objet {type(value).__name__} converti récursivement") - except Exception as e: - # Si échec, convertir en string - serialized[key] = f"<{type(value).__name__}>" - logger.warning(f"⚠️ Config key '{key}': objet {type(value).__name__} non sérialisable, converti en string: {e}") + # Pass a set of visited object IDs to prevent infinite recursion + if 'visited' not in kwargs: + kwargs['visited'] = set() + + if id(value) in kwargs['visited']: + serialized[key] = f"<CircularReference:{type(value).__name__}>" + logger.warning(f"⚠️ Config key '{key}': détection d'une référence circulaire pour l'objet {type(value).__name__}, converti en string") + else: + kwargs['visited'].add(id(value)) + try: + serialized[key] = serialize_config_safe(value.__dict__, **kwargs) + logger.debug(f"🔧 Config key '{key}': objet {type(value).__name__} converti récursivement") + except Exception as e: + # Si échec, convertir en string + serialized[key] = f"<{type(value).__name__}>" + logger.warning(f"⚠️ Config key '{key}': objet {type(value).__name__} non sérialisable, converti en string: {e}") Apply / Chat Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a potential RecursionError due to circular references in the new serialize_config_safe function and proposes a standard, effective solution, which is crucial for application stability.	Medium
	Ensure timezone-aware datetime objects Make the datetime object created from a timestamp timezone-aware by adding the tz=timezone.utc argument to datetime.fromtimestamp() to ensure timezone consistency. core/position_manager.py [1166-1168] # 🔥 FIX BUG #3: Utiliser timezone.utc pour PostgreSQL TIMESTAMPTZ -opened_at = datetime.fromtimestamp(self.active_position.start_time).isoformat() if hasattr(self.active_position, 'start_time') else self.active_position.timestamp +opened_at = datetime.fromtimestamp(self.active_position.start_time, tz=timezone.utc).isoformat() if hasattr(self.active_position, 'start_time') else self.active_position.timestamp closed_at = datetime.now(timezone.utc).isoformat() Apply / Chat Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies that datetime.fromtimestamp() creates a naive datetime object, which contradicts the PR's goal of using timezone-aware UTC datetimes for database consistency, making it a valuable correction.	Medium
	Avoid using infinity string literal Replace the string literal '∞' with a large number like '9999.00' when calculating the profit factor with zero losses to ensure a consistent numerical data type for downstream processing. frontend/src/lib/stores/stats.js [91-107] // Computed: Profit Factor (Somme profits / |Somme pertes|) export const profitFactor = derived(tradeHistory, $trades => { if ($trades.length === 0) return '0.00'; // 🔥 FIX BUG CRITIQUE: Profit Factor = Σ(profits) / |Σ(losses)| // Ancienne formule incorrecte: (wins × best_trade) / (losses × worst_trade) const grossProfit = $trades .filter(t => (t.net_pnl_usdt || t.pnl_usdt || 0) > 0) .reduce((sum, t) => sum + (t.net_pnl_usdt || t.pnl_usdt || 0), 0); const grossLoss = Math.abs($trades .filter(t => (t.net_pnl_usdt || t.pnl_usdt || 0) < 0) .reduce((sum, t) => sum + (t.net_pnl_usdt || t.pnl_usdt || 0), 0)); - if (grossLoss === 0) return grossProfit > 0 ? '∞' : '0.00'; + if (grossLoss === 0) { + return grossProfit > 0 ? '9999.00' : '0.00'; // Use a large number instead of '∞' + } return (grossProfit / grossLoss).toFixed(2); }); Apply / Chat Suggestion importance[1-10]: 5 __ Why: The suggestion correctly points out that returning the string '∞' can cause issues, and replacing it with a large number is a good practice for robustness, improving data handling in the frontend.	Low
General	Use TRUNCATE for faster database reset In the reset_datalogger_db function, replace the loop of DELETE statements with a single TRUNCATE TABLE ... RESTART IDENTITY CASCADE command for a more efficient and faster database reset. main.py [4650-4724] @app.delete("/api/datalogger/reset") async def reset_datalogger_db(): """ 🔥 Reset complet de la base de données PostgreSQL du datalogger ATTENTION: Cette opération supprime TOUTES les données (scans, opportunities, trades, etc.) Returns: Message de confirmation """ try: from core.callbacks.scanner_loop import get_pg_datalogger pg_datalogger = get_pg_datalogger() if not pg_datalogger or not pg_datalogger.enabled: return JSONResponse( {"error": "PostgreSQL DataLogger non disponible"}, status_code=503 ) conn = pg_datalogger._get_connection() if not conn: return JSONResponse( {"error": "Impossible de se connecter à PostgreSQL"}, status_code=503 ) try: cursor = conn.cursor() - # Supprimer toutes les données (dans l'ordre pour respecter les contraintes FK) - tables = [ - 'trades', + # Utiliser TRUNCATE pour une suppression rapide et efficace + # RESTART IDENTITY réinitialise les séquences (ex: IDs auto-incrémentés) + # CASCADE propage la troncature aux tables dépendantes via FK + tables_to_truncate = [ + 'trading_sessions', + 'config_snapshots', + 'market_context', + 'scan_errors', + 'scan_logs', 'opportunities', - 'scan_logs', - 'scan_errors', - 'market_context', - 'config_snapshots', - 'trading_sessions' + 'trades' ] - deleted_counts = {} - for table in tables: - cursor.execute(f"DELETE FROM {table}") - deleted_counts[table] = cursor.rowcount + # La commande TRUNCATE peut prendre plusieurs tables + truncate_query = f"TRUNCATE TABLE {', '.join(tables_to_truncate)} RESTART IDENTITY CASCADE;" + cursor.execute(truncate_query) conn.commit() cursor.close() pg_datalogger._return_connection(conn) - total_deleted = sum(deleted_counts.values()) - logger.warning(f"🗑️ Base de données PostgreSQL resetée: {total_deleted} enregistrements supprimés") + logger.warning(f"🗑️ Base de données PostgreSQL resetée avec TRUNCATE sur les tables: {', '.join(tables_to_truncate)}") return JSONResponse({ "success": True, - "message": f"Base de données resetée avec succès", - "deleted": deleted_counts, - "total_deleted": total_deleted + "message": "Base de données resetée avec succès via TRUNCATE." }) except Exception as e: conn.rollback() pg_datalogger._return_connection(conn) logger.error(f"❌ Erreur reset DB: {e}", exc_info=True) return JSONResponse( {"error": f"Erreur reset DB: {str(e)}"}, status_code=500 ) except Exception as e: logger.error(f"❌ Erreur reset DB: {e}", exc_info=True) return JSONResponse( {"error": f"Erreur reset DB: {str(e)}"}, status_code=500 ) Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion proposes using TRUNCATE instead of DELETE for better performance in the new /api/datalogger/reset endpoint, which is a valid and significant optimization for database operations.	Medium
More