Start ml1211

[chpeu]

PR Type

Enhancement, Bug fix, Tests

---

Description

• PostgreSQL ML Data Logger Infrastructure: Comprehensive new module (postgresql_datalogger.py) implementing connection pooling, batch inserts, and session management for collecting ML training data across trading scans, opportunities, and trades

• Analyzer Integration with ML Logging: Enhanced analyzer.py with indicator extraction and logging at key decision points (Point A for scans, Point B for opportunities), including all technical indicators (EMA, RSI, MACD, ADX, Bollinger Bands, Volume)

• Position Manager Trade Logging: Implemented comprehensive entry indicators capture (Point C) and trade exit logging to PostgreSQL with complete trade data, indicators, and configuration snapshots; added opened_at field for frontend countdown timer

• Scanner Loop PostgreSQL Integration: Added scan data logging with market data, indicators, and pattern information; enhanced scalability data retrieval with fallback mechanisms

• Application Startup & Configuration: Added .env loading, DataLogger initialization with proper shutdown handling, and new configuration parameters (max_slippage_pct, PostgreSQL credentials, connection pool settings)

• WebSocket Real-time Log Streaming: New WebSocketLogHandler class to send logs to frontend in real-time while preserving ANSI colors and emojis

• Frontend UI Enhancements: Added dynamic countdown timer for position duration, TP/SL info calculations, dynamic status messaging in bot controls, and debug attributes throughout components

• Bug Fixes: Fixed spread validation with NaN handling in scanner.py, improved market data robustness in analyzer/market_data.py, corrected PnL calculator fee calculation for partial TP scenarios, fixed atr_percent field naming to atr_pct

• Database Verification Tools: Multiple diagnostic scripts for schema validation, column/parameter counting, and data integrity checks

• Unit Tests: Comprehensive test suite for PostgreSQLDataLogger class with 11 test methods covering initialization, connection handling, batch mode, error logging, and buffer flushing

---

Diagram Walkthrough

flowchart LR
  A["Scanner Loop"] -->|"scan data + indicators"| B["PostgreSQL DataLogger"]
  C["Analyzer"] -->|"opportunities + conditions"| B
  D["Position Manager"] -->|"entry indicators + trades"| B
  B -->|"batch inserts"| E["PostgreSQL DB"]
  F["Logger"] -->|"real-time logs"| G["WebSocket"]
  G -->|"display"| H["Frontend UI"]
  E -->|"trade data"| H
  H -->|"countdown timer + stats"| I["Position Card"]

Loading

File Walkthrough

	Relevant files
Enhancement	11 files postgresql_datalogger.py PostgreSQL DataLogger for ML Training Data Collection        core/postgresql_datalogger.py New comprehensive PostgreSQL data logger module for ML training data collection Implements connection pooling, batch inserts, and session management for trading scans, opportunities, and trades Provides methods to log scans with full indicator data, opportunities, errors, market context, and completed trades Includes batch insert optimization using execute_values for improved performance +1382/-0 analyzer.py Enhanced Analyzer with ML Data Logging Integration              core/analyzer.py Added helper function build_indicators_dict() to construct consistent indicator dictionaries across rejection scenarios Enhanced return values to include all technical indicators (EMA, RSI, MACD, ADX, Bollinger Bands, Volume) in analysis results Integrated scan logging at Point A with indicator extraction from 1m and 5m timeframes Added opportunity logging at Point B with conditions matching and score calculation Fixed atr_percent field naming to atr_pct for consistency with database schema Improved error handling with websocket logging for orderbook imbalance rejections +541/-15 position_manager.py Position management enhancements with PostgreSQL logging and indicator capture core/position_manager.py Added opened_at field to position dictionary for frontend countdown timer functionality Fixed TP/SL mode handling to accept 'ESCALIER' as valid mode and read use_atr from TRADING_CONFIG dynamically Implemented comprehensive entry indicators capture (Point C) with fallback mechanisms for PostgreSQL logging Added _update_trailing_stop_fixe() method for fixed-distance trailing stop in FIXE mode Enhanced slippage calculation with proper USDT conversion and detailed logging Implemented trade exit logging to PostgreSQL with complete trade data, indicators, and configuration snapshots +594/-32 main.py Application startup, configuration management, and data logging infrastructure main.py Added .env loading at startup before other imports Initialized DataLogger and PostgreSQL DataLogger in startup event with proper shutdown handling Enhanced scalability data retrieval with fallback mechanisms and NaN handling for spread values Implemented comprehensive entry indicators construction from analysis data Added slippage validation and SL pre-check before position opening Extended WebSocket state with Telegram notification type configuration Added support for break_even_trigger, trailing_distance, and max_slippage_pct configuration parameters +648/-33 scanner_loop.py Scanner loop PostgreSQL integration and scan data logging core/callbacks/scanner_loop.py Added PostgreSQL DataLogger injection and retrieval functions for ML data logging Implemented comprehensive scan logging to PostgreSQL with market data, indicators, and pattern information Enhanced scalability data retrieval with fallback mechanisms for spread and depth values Added entry indicators construction and validation before setup storage Implemented scan error logging to PostgreSQL with error details and stack traces +339/-6  position_check_loop.py Position update and stats calculation improvements              core/callbacks/position_check_loop.py Added opened_at field to position update data for countdown timer functionality Extracts timestamp from start_time or position.timestamp with proper type handling Added debug logging for trade statistics calculation with sample trade details Fixed PnL rounding to 4 decimal places to preserve small values instead of 2 decimals +23/-2    logger.py WebSocket-based real-time log streaming to frontend            utils/logger.py New WebSocketLogHandler class to send logs to frontend via WebSocket in real-time Preserves ANSI color codes and emojis in log messages for frontend display Integrates with existing ColoredFormatter for consistent formatting Added optional ws_manager parameter to setup_logger function +84/-1    analytics_database.py Trade retrieval duration field compatibility fix                  core/analytics_database.py Enhanced get_trades method to ensure duration_seconds field is present in returned trades Adds fallback mapping from duration to duration_seconds if only duration exists Improves compatibility with different database schema versions +8/-1      PositionCard.svelte Position card UI enhancements with dynamic duration and TP/SL info frontend/src/lib/components/PositionCard.svelte Added dynamic countdown timer for position duration with support for days/hours/minutes/seconds format Implemented nextTpInfo and nextSlInfo calculations based on trading config and TP/SL mode Added opened_at field support for accurate position duration calculation Enhanced TP/SL display with PnL objectives and position size percentages Added comprehensive debug attributes (data-debug-name) throughout component for frontend debugging +284/-46 BotControls.svelte Bot controls dynamic status messaging                                        frontend/src/lib/components/BotControls.svelte Added dynamic status message calculation based on bot phase, position state, and scanner status Integrated botPhase store and getPhaseMessage function for contextual status display Added debug attributes to all major UI elements for frontend debugging Improved status messaging priority: active position > bot phase > scanner state > stopped +30/-12  sessions.js Global stats store PnL percentage tracking                              frontend/src/lib/stores/sessions.js Added total_pnl_percent field to globalStats store for tracking overall PnL percentage +1/-0
Tools	1 files verify_all.py PostgreSQL Database Verification and Diagnostic Tool          database/verify_all.py New verification script to validate PostgreSQL connection, schema, and data integrity Checks for required tables, important columns, and data population across scan_logs, opportunities, and trades Validates configuration settings and Python code syntax Provides detailed diagnostic output with color-coded status messages +360/-0
Miscellaneous	17 files verifier_indicators.py Database verification utility for entry indicators              database/verifier_indicators.py New utility script to verify entry indicators in PostgreSQL trades table Displays statistics on indicator coverage across all trades Shows detailed breakdown of last trade's indicators for debugging +121/-0  count_exact_final.py Database column and parameter counting diagnostic script  database/count_exact_final.py New diagnostic script to count and verify exact number of columns in INSERT statement, placeholders in VALUES clause, and parameters in params tuple Lists all 128 columns from the trades table with line-by-line documentation Calculates totals and differences between columns, placeholders, and parameters Identifies mismatches in the database logging implementation +246/-0  verify_schema_complete.py Complete PostgreSQL schema verification and validation tool database/verify_schema_complete.py New comprehensive schema verification script that connects to PostgreSQL and validates table structure Retrieves and displays all columns, indexes, and foreign keys for each table Includes specific verification for the trades table against expected columns from Python code Provides detailed reporting with emoji indicators for success/failure status +270/-0  verify_schema.py PostgreSQL schema structure verification script                    database/verify_schema.py New schema verification script that checks PostgreSQL database structure against expected tables and columns Validates presence of required tables, important columns, extensions, indexes, and foreign keys Includes statistics on table row counts and specific checks for timestamp columns in trades table Provides formatted output with status indicators +191/-0  compare_insert_values.py INSERT vs VALUES placeholder comparison diagnostic              database/compare_insert_values.py New diagnostic script comparing INSERT columns with VALUES placeholders line-by-line Counts 128 total columns across 39 lines and 111 total placeholders across 21 lines Identifies 17-placeholder mismatch between INSERT and VALUES clauses +75/-0    count_columns.py Column counting script for INSERT statement                            database/count_columns.py New script counting columns in INSERT statement line-by-line Totals 128 columns with detailed breakdown by line number Compares against expected 111 placeholders to identify 17-column discrepancy +58/-0    find_missing_placeholders.py Missing placeholder identification script                                database/find_missing_placeholders.py New diagnostic script listing all 128 column names from INSERT statement in order Identifies which columns lack corresponding placeholders in VALUES clause Provides detailed column list for manual verification and debugging +103/-0  count_all_params.py Manual parameter counting script for log_trade method        database/count_all_params.py New script performing manual line-by-line parameter counting in log_trade method Counts parameters across 24 sections from lines 916-1016 Calculates total and compares against expected 111 placeholders +93/-0    count_params_final.py Automated parameter counting with regex parsing                    database/count_params_final.py New diagnostic script using regex to parse and count parameters in log_trade method Extracts INSERT columns, VALUES placeholders, and params tuple from source file Compares counts and identifies mismatches between expected and actual parameters +82/-0    count_exact.py Exact column and parameter extraction script                          database/count_exact.py New script extracting and counting columns, placeholders, and parameters from source file Uses regex to parse INSERT, VALUES, and params sections Displays first and last 10 columns for verification +82/-0    fix_trade_logging.py Trade logging parameter verification script                            database/fix_trade_logging.py New diagnostic script to verify and identify parameter mismatches in log_trade method Parses INSERT columns and VALUES placeholders from source code Estimates parameter count and identifies discrepancies +73/-0    count_params_exact.py Line-by-line parameter counting script                                      database/count_params_exact.py New script analyzing params tuple line-by-line to count individual parameters Displays first 20 parameters for debugging and verification Compares parameter count against placeholder count +57/-0    count_params.py Placeholder and parameter counting script                                database/count_params.py New script counting placeholders in VALUES clause and parameters in params tuple Uses regex to find all %s occurrences in VALUES section Estimates parameter count by analyzing comma separators +51/-0    count_manual.py Manual parameter counting documentation                                    database/count_manual.py New script documenting manual line-by-line parameter count across 24 sections Totals 111 parameters but notes discrepancy with actual 128 parameters Identifies 17-parameter excess and notes one known doublon (gross_pnl_usdt) +38/-0    count_params_manual.py Parameter count discrepancy analysis                                          database/count_params_manual.py New script analyzing parameter count discrepancy between expected (111) and actual (128) Documents known doublon at parameter 15 (gross_pnl_usdt) Notes 16 additional unidentified extra parameters +27/-0    count_placeholders.py VALUES clause placeholder counting script                                database/count_placeholders.py New script counting placeholders in VALUES clause line-by-line across 21 lines Totals 111 placeholders and notes 17-parameter excess in params tuple +35/-0    count_placeholders_exact.py Exact placeholder counting from source file                            database/count_placeholders_exact.py New script extracting VALUES section from source file and counting %s placeholders Displays placeholder count per line for verification +24/-0
Configuration changes	1 files config.py Configuration additions for slippage validation and PostgreSQL config.py Added max_slippage_pct configuration parameter (0.03% default) for slippage validation Added PostgreSQL configuration variables for ML datalogger (host, port, database, credentials, connection pool settings) +16/-0
Bug fix	3 files scanner.py Spread validation and NaN handling improvements                    core/scanner.py Fixed spread validation to check for NaN values explicitly using math.isnan() Added minimum spread threshold of 0.001% to prevent zero slippage scenarios Improved spread range validation to be between 0.001% and 0.02% +5/-3      market_data.py Market data spread check robustness improvements                  core/analyzer/market_data.py Added null check for orderbook to prevent crashes when API returns None Safely extracts bids and asks with defensive checks for empty lists and array bounds Improved error handling with proper logging via logger.error (captured by WebSocketLogHandler) Removed redundant manual WebSocket emission since logger now handles it automatically +15/-3    pnl_calculator.py PnL calculator fee calculation bug fix                                      core/position/pnl_calculator.py Fixed fee calculation bug: fees now calculated only on remaining position size when partial TP already executed Previously calculated fees on full position size even after partial TP sold Correctly accounts for fees already paid on partial TP sold portion +7/-2
Tests	1 files test_postgresql_datalogger.py Unit tests for PostgreSQL data logger functionality            tests/test_postgresql_datalogger.py New comprehensive unit test suite for PostgreSQLDataLogger class with 11 test methods Tests initialization, connection handling, batch mode logging, error logging, and buffer flushing Includes fixtures for mocking PostgreSQL connections and connection pools Tests both successful operations and error conditions with proper assertions +321/-0
Additional files	75 files .env.example +43/-15  ANALYSE_MAIN_RESTORED.md +1/-0      CHECKLIST_ML_OPTIMIZATION.md +206/-0  CONFIG_POSTGRES.md +100/-0  DOCUMENTATION_COMPLETE.md +881/-1226 FIX_DATALOGGER.md +60/-0    PHASE3_DATALOGGER_OPTIMIZATIONS.md +100/-0  RECAPITULATIF_JOURNEE_2025-11-12.md +470/-0  ANALYSE_PARAMS.md +73/-0    ANALYSE_SCHEMA_COMPLETENESS.md +229/-0  ANALYSE_VARIABLES_MANQUANTES.md +236/-0  APPLIQUER_MIGRATION.bat +46/-0    APPLIQUER_MIGRATION.ps1 +49/-0    EXECUTE_SCHEMA.md +101/-0  FIX_MISSING_TRADES.md +58/-0    FIX_PARAMS.md +20/-0    GUIDE_APPLICATION_MIGRATIONS.md +127/-0  GUIDE_TEST_DATALOGGER.md +458/-0  INSTALLATION_COMPLETE.md +177/-0  INSTRUCTIONS_MIGRATION.md +168/-0  LISTE_COMPLETE_VARIABLES_SCHEMA.md +501/-0  MIGRATION_GUIDE.md +211/-0  QUICK_LIST_COMMANDS.md +193/-0  QUICK_QUERIES.sql +87/-0    README_POSTGRESQL.md +255/-0  SCHEMA_COMPLET_RECAPITULATIF.md +440/-0  SOLUTION_PARAMS.md +22/-0    VARIABLES_INUTILES_ANALYSE.md +121/-0  VERIFICATION_COMPLETE.md +449/-0  VERIFICATION_VARIABLES_CONFIG.md +223/-0  VERIFIER_INDICATEURS.bat +7/-0      VERIFIER_INDICATEURS.ps1 +6/-0      VERIFIER_INDICATEURS_ENTREE.md +308/-0  VERIFIER_TOUT.bat +44/-0    VIEW_LOGS_GUIDE.md +375/-0  create_all_2025_partitions.sql +82/-0    create_missing_trades_table.sql +175/-0  drop_old_schema.sql +196/-0  fix_get_global_stats.sql +50/-0    fix_immutable_error.sql +155/-0  install.ps1 +85/-0    list_existing_schema.sql +240/-0  migration_add_all_variables.sql +299/-0  migration_add_early_invalidation.sql +47/-0    migration_add_market_context_jsonb.sql +45/-0    migration_complete_all_changes.sql +352/-0  schema_postgresql_complete.sql +932/-0  update_partitions_to_3_months.sql +92/-0    verifier_indicators.sql +74/-0    verify_schema.sql +154/-0  package.json +2/-1      ConnectionStatus.svelte +3/-3      ExportPanel.svelte +61/-43  GlobalStats.svelte +58/-49  LogViewer.svelte +168/-265 NotificationSettings.svelte +343/-70 PnLChart.svelte +6/-6      PnLPercentChart.svelte +222/-0  ScannerPanel.svelte +25/-25  SessionSelector.svelte +46/-36  SettingsPanel.svelte +25/-346 StatsPanel.svelte +26/-26  Tabs.svelte +5/-4      TradeHistory.svelte +94/-43  VariablesPanel.svelte +563/-431 VolumeChart.svelte +13/-13  WinLossChart.svelte +16/-16  botPhase.js +37/-0    debug.js +25/-0    logs.js +2/-2      stats.js +59/-29  debugTooltip.js +208/-0  export.js +253/-11 +page.svelte +394/-77 requirements.txt +1/-0

---

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Credential handling Description: The PostgreSQL connection string embeds credentials directly from environment variables including POSTGRES_PASSWORD, risking sensitive information exposure if logs or error traces print the connection string or if env vars are mismanaged; consider using libpq connection URIs with separate secure secret management. postgresql_datalogger.py [73-86] Referred Code if connection_string: self.connection_string = connection_string else: # Utiliser variables d'environnement ou paramètres host = host or os.getenv('POSTGRES_HOST', 'localhost') port = port or int(os.getenv('POSTGRES_PORT', '5432')) database = database or os.getenv('POSTGRES_DB', 'trade_cursor_ml') user = user or os.getenv('POSTGRES_USER', 'postgres') password = password or os.getenv('POSTGRES_PASSWORD', '') self.connection_string = ( f"host={host} port={port} dbname={database} " f"user={user} password={password}" )
	Sensitive information exposure Description: SQL errors are logged with query previews (logger.debug(f"Query: {query[:200]}...")) which may leak sensitive structure or values if expanded elsewhere; ensure parameters are not logged and avoid logging raw queries in production. postgresql_datalogger.py [160-166] Referred Code except Exception as e: logger.error(f"❌ Erreur exécution requête: {e}") logger.debug(f"Query: {query[:200]}...") if conn: conn.rollback() self._return_connection(conn) return None
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Partial Coverage: The PR introduces extensive audit-style logging to PostgreSQL for scans, opportunities, trades, and errors, but it is unclear if all critical actions (permission changes, user login events, deletions) are applicable or covered in this trading context. Referred Code def get_or_create_session(self, session_id: Optional[str] = None) -> Optional[str]: """ Obtenir ou créer une session de trading Args: session_id: UUID de session existante (optionnel) Returns: UUID de la session """ if not self.enabled: return None # Si session_id fourni, vérifier qu'elle existe if session_id: query = "SELECT id FROM trading_sessions WHERE id = %s" result = self._execute_query(query, (session_id,), fetch=True) if result: return session_id # Créer nouvelle session ... (clipped 1194 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Debug Noise: Numerous debug logs with emojis and French text may reduce clarity and maintainability across the codebase and mixed-language teams despite generally meaningful variable names. Referred Code elif r and isinstance(r, dict): # Setup valide = a 'direction' ET 'entry' (ou 'price') if 'direction' in r and ('entry' in r or 'price' in r): # 🔥 DEBUG: Vérifier si le setup contient les indicateurs symbol_check = r.get('symbol', 'UNKNOWN') logger.info(f"🔍 DEBUG valid_setups.append({symbol_check}): contient indicators_1m: {'indicators_1m' in r}, indicators_5m: {'indicators_5m' in r}") valid_setups.append(r) else: # Rejet rejections.append(r) # else: None = aussi une erreur/skip logger.info(f"📊 Résumé: {len(valid_setups)} setups valides, {len(rejections)} rejets, {len(errors)} erreurs") # Si on a trouvé un setup valide, ouvrir une position if valid_setups and _position_manager: # Prendre le meilleur setup (premier dans la liste) best_setup = valid_setups[0] try: # BUG #7 FIX: Valider que entry est présent et valide ... (clipped 129 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Fallback Behavior: Database execution errors are caught and logged, but many methods return None without retries or structured error signaling which may mask failures in production flows. Referred Code def _execute_query(self, query: str, params: tuple = None, fetch: bool = False): """ Exécuter une requête SQL Args: query: Requête SQL params: Paramètres (tuple) fetch: Si True, retourner les résultats Returns: Résultats si fetch=True, sinon None """ if not self.enabled: return None conn = self._get_connection() if not conn: return None try: cursor = conn.cursor() ... (clipped 21 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Stack Exposure: The code captures and logs full stack traces for scan errors which is good internally, but it is unclear whether any of these logs are user-facing or exposed in non-secure channels. Referred Code if _pg_datalogger and _pg_datalogger.enabled: try: import traceback error_details = { 'error_type': type(e).__name__, 'error_message': str(e), 'stack': traceback.format_exc() } _pg_datalogger.log_scan_error( symbol=symbol, error_type='SCAN_ERROR', error_message=str(e), error_details=error_details ) except Exception as log_error: logger.warning(f"⚠️ Erreur logging erreur scan: {log_error}") Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Sensitive Fields: The logger stores detailed trade and market data including config snapshots and error stacks; while likely non-PII, absence of explicit scrubbing may risk sensitive configuration or secrets being persisted. Referred Code config_snapshot = trade_data.get('config_snapshot', {}) if config_snapshot: config_snapshot = json.dumps(config_snapshot) else: config_snapshot = None # Convertir entry_conditions en liste de strings pour PostgreSQL TEXT[] # (doit être fait avant de créer params) if isinstance(entry_conditions, dict): entry_conditions = [str(k) for k in entry_conditions.keys()] if entry_conditions else [] elif isinstance(entry_conditions, list): entry_conditions = [str(c) for c in entry_conditions if c] # Convertir en strings else: entry_conditions = [str(entry_conditions)] if entry_conditions else [] params = ( entry_timestamp, exit_timestamp, session_id, opportunity_id, scan_log_id, trade_data.get('symbol'), trade_data.get('direction'), trade_data.get('entry_price'), trade_data.get('exit_price'), ... (clipped 97 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Input Sanitization: Data from analysis and runtime dicts are inserted into SQL with parameterization (good), but external input provenance and validation (e.g., symbol strings, numeric bounds) are not explicitly enforced in the new code. Referred Code params = ( session_id, symbol, scan_data.get('scan_duration_ms'), market_data.get('price'), market_data.get('spread_pct'), market_data.get('book_depth'), market_data.get('balance_score'), market_data.get('bid_vol'), market_data.get('ask_vol'), market_data.get('orderbook_imbalance_ratio'), # 1m indicators_1m.get('ema9'), indicators_1m.get('ema21'), indicators_1m.get('ema_diff_pct'), indicators_1m.get('rsi'), indicators_1m.get('rsi_prev'), indicators_1m.get('macd'), indicators_1m.get('macd_signal'), indicators_1m.get('macd_hist'), indicators_1m.get('macd_hist_prev'), indicators_1m.get('adx'), indicators_1m.get('di_plus'), indicators_1m.get('di_minus'), indicators_1m.get('di_gap'), indicators_1m.get('atr'), indicators_1m.get('atr_pct'), indicators_1m.get('bb_upper'), indicators_1m.get('bb_middle'), indicators_1m.get('bb_lower'), indicators_1m.get('bb_width'), indicators_1m.get('bb_distance_to_lower'), indicators_1m.get('bb_distance_to_upper'), indicators_1m.get('volume'), indicators_1m.get('volume_avg'), indicators_1m.get('volume_ratio'), indicators_1m.get('volume_spike'), ... (clipped 57 lines) Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Credential exposure risk Description: The PostgreSQL connection string may include plaintext password from environment variables and gets constructed as "user=... password=..." which can be exposed via logs or exceptions; ensure secrets are not logged and use secure parameterization or lib-native dsn handling. postgresql_datalogger.py [73-87] Referred Code if connection_string: self.connection_string = connection_string else: # Utiliser variables d'environnement ou paramètres host = host or os.getenv('POSTGRES_HOST', 'localhost') port = port or int(os.getenv('POSTGRES_PORT', '5432')) database = database or os.getenv('POSTGRES_DB', 'trade_cursor_ml') user = user or os.getenv('POSTGRES_USER', 'postgres') password = password or os.getenv('POSTGRES_PASSWORD', '') self.connection_string = ( f"host={host} port={port} dbname={database} " f"user={user} password={password}" )
	Sensitive information exposure Description: SQL execution uses string queries with %s placeholders but logs parts of the query text on error; while parameters are not interpolated, ensure no sensitive data from params (e.g., config snapshots) is leaked via error messages or debug logs. postgresql_datalogger.py [146-166] Referred Code cursor = conn.cursor() cursor.execute(query, params) if fetch: results = cursor.fetchall() cursor.close() conn.commit() self._return_connection(conn) return results else: conn.commit() cursor.close() self._return_connection(conn) return True except Exception as e: logger.error(f"❌ Erreur exécution requête: {e}") logger.debug(f"Query: {query[:200]}...") if conn: conn.rollback() self._return_connection(conn) return None
	Excessive logging leakage Description: Extensive info-level debug logging for market and orderbook data (spreads, volumes, balances) can leak sensitive trading strategy details to logs, increasing operational risk if logs are accessible. scanner_loop.py [247-355] Referred Code # 🔥 DEBUG: Log pour voir ce qui est disponible logger.info(f"💹 DEBUG: Recherche scalability_data pour {symbol}") logger.info(f"💹 DEBUG: best_setup contient spread_pct={best_setup.get('spread_pct')}, keys={list(best_setup.keys())[:10]}") if _app_state and _app_state.get('top_pairs'): logger.info(f"💹 DEBUG: top_pairs contient {len(_app_state['top_pairs'])} paires") found_pair = False for pair in _app_state['top_pairs']: if pair.get('symbol') == symbol: found_pair = True # 🔥 FIX: Utiliser les bonnes clés depuis le scanner (spread, bookDepth, balanceScore, bidVol, askVol) spread_value = pair.get('spread', 0) book_depth = pair.get('bookDepth', 0) balance_score = pair.get('balanceScore', 1.0) bid_vol = pair.get('bidVol', 0) ask_vol = pair.get('askVol', 0) logger.info(f"💹 DEBUG: Données brutes depuis top_pairs: spread={spread_value}, bookDepth={book_depth}, balanceScore={balance_score}, bidVol={bid_vol}, askVol={ask_vol}") # Vérifier si spread est NaN ou invalide if isinstance(spread_value, float) and (spread_value != spread_value or spread_value == float('nan')): ... (clipped 88 lines)
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Missing User Context: New logging of scans, opportunities, errors, and trades lacks explicit user identity or actor information in log records, which may hinder reconstructing who performed actions. Referred Code def get_or_create_session(self, session_id: Optional[str] = None) -> Optional[str]: """ Obtenir ou créer une session de trading Args: session_id: UUID de session existante (optionnel) Returns: UUID de la session """ if not self.enabled: return None # Si session_id fourni, vérifier qu'elle existe if session_id: query = "SELECT id FROM trading_sessions WHERE id = %s" result = self._execute_query(query, (session_id,), fetch=True) if result: return session_id # Créer nouvelle session ... (clipped 902 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Debug Noise: Extensive DEBUG/INFO logs with emojis and French text (e.g., "💹 DEBUG") may reduce clarity and maintainability and mix concerns within business logic. Referred Code # BUG #5 FIX: Récupérer scalability_data depuis top_pairs symbol = best_setup.get('symbol') scalability_data = None # 🔥 DEBUG: Log pour voir ce qui est disponible logger.info(f"💹 DEBUG: Recherche scalability_data pour {symbol}") logger.info(f"💹 DEBUG: best_setup contient spread_pct={best_setup.get('spread_pct')}, keys={list(best_setup.keys())[:10]}") if _app_state and _app_state.get('top_pairs'): logger.info(f"💹 DEBUG: top_pairs contient {len(_app_state['top_pairs'])} paires") found_pair = False for pair in _app_state['top_pairs']: if pair.get('symbol') == symbol: found_pair = True # 🔥 FIX: Utiliser les bonnes clés depuis le scanner (spread, bookDepth, balanceScore, bidVol, askVol) spread_value = pair.get('spread', 0) book_depth = pair.get('bookDepth', 0) balance_score = pair.get('balanceScore', 1.0) bid_vol = pair.get('bidVol', 0) ask_vol = pair.get('askVol', 0) ... (clipped 92 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Swallowed Errors: Database operation failures return None after logging without propagating or signaling callers, which may cause silent functional degradation. Referred Code except Exception as e: logger.error(f"❌ Erreur exécution requête: {e}") logger.debug(f"Query: {query[:200]}...") if conn: conn.rollback() self._return_connection(conn) return None Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Query Exposure: On SQL errors the code logs the beginning of the SQL query which could expose internal schema details in user-visible logs if log configuration is not restricted. Referred Code except Exception as e: logger.error(f"❌ Erreur exécution requête: {e}") logger.debug(f"Query: {query[:200]}...") if conn: Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Sensitive Config: Configuration snapshots and detailed analysis data are serialized and logged to the database and may include sensitive parameters unless filtered, and logs include full exception stacks. Referred Code # Préparer config_snapshot complet (toutes les variables de configuration) try: from config import ( TRADING_CONFIG, RISK_CONFIG, CONDITION_WEIGHTS, TREND_BONUS_CONFIG, RETRY_CONFIG, CIRCUIT_BREAKER_CONFIG, WEBSOCKET_CONFIG ) config_snapshot_dict = {} # Copier TRADING_CONFIG if TRADING_CONFIG: config_snapshot_dict.update(TRADING_CONFIG.copy()) # Ajouter les variables définies séparément config_snapshot_dict['RISK_CONFIG'] = RISK_CONFIG config_snapshot_dict['CONDITION_WEIGHTS'] = CONDITION_WEIGHTS config_snapshot_dict['TREND_BONUS_CONFIG'] = TREND_BONUS_CONFIG config_snapshot_dict['RETRY_CONFIG'] = RETRY_CONFIG config_snapshot_dict['CIRCUIT_BREAKER_CONFIG'] = CIRCUIT_BREAKER_CONFIG config_snapshot_dict['WEBSOCKET_CONFIG'] = WEBSOCKET_CONFIG config_snapshot = json.dumps(config_snapshot_dict) except Exception as e: logger.warning(f"⚠️ Erreur préparation config_snapshot pour session: {e}") ... (clipped 2 lines) Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Input Sanitization: Data from analysis and external sources is inserted into SQL with minimal validation (relying on parameterization) and may include unexpected structures or large payloads without explicit sanitization or size checks. Referred Code # Préparer les paramètres params = ( session_id, symbol, scan_data.get('scan_duration_ms'), market_data.get('price'), market_data.get('spread_pct'), market_data.get('book_depth'), market_data.get('balance_score'), market_data.get('bid_vol'), market_data.get('ask_vol'), market_data.get('orderbook_imbalance_ratio'), # 1m indicators_1m.get('ema9'), indicators_1m.get('ema21'), indicators_1m.get('ema_diff_pct'), indicators_1m.get('rsi'), indicators_1m.get('rsi_prev'), indicators_1m.get('macd'), indicators_1m.get('macd_signal'), indicators_1m.get('macd_hist'), indicators_1m.get('macd_hist_prev'), indicators_1m.get('adx'), indicators_1m.get('di_plus'), indicators_1m.get('di_minus'), indicators_1m.get('di_gap'), indicators_1m.get('atr'), indicators_1m.get('atr_pct'), indicators_1m.get('bb_upper'), indicators_1m.get('bb_middle'), indicators_1m.get('bb_lower'), indicators_1m.get('bb_width'), indicators_1m.get('bb_distance_to_lower'), indicators_1m.get('bb_distance_to_upper'), indicators_1m.get('volume'), indicators_1m.get('volume_avg'), ... (clipped 907 lines) Learn more about managing compliance generic rules or creating your own custom rules

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
High-level	Decouple data logging from trading logic Decouple data logging from the core trading logic by using an event-based system. Trading components should emit simple events, and a separate asynchronous service should handle the complex data processing and database insertion. Examples: core/position_manager.py [1196-1374] # ======================================== # ✅ POINT D : LOG TRADE EXIT # ======================================== # 🔥 PHASE 2: Logger dans PostgreSQL si activé try: from core.callbacks.scanner_loop import get_pg_datalogger pg_datalogger = get_pg_datalogger() if pg_datalogger and pg_datalogger.enabled: try: ... (clipped 169 lines) core/analyzer.py [711-879] # ======================================== # ✅ POINT A : LOG SCAN (NON-BLOCKING) # ======================================== scan_duration_ms = (time.time() - start_time) * 1000 scan_uuid = None try: from backend.ml.data_logger import DataLogger data_logger = DataLogger() ... (clipped 159 lines) Solution Walkthrough: Before: # In core/position_manager.py def close_position(self, exit_price: float, reason: str): # ... core logic to calculate PnL ... pnl_data = self.pnl_calculator.calculate_pnl(...) result = { ... } # Create result dict # Tightly coupled data logging logic starts here try: pg_datalogger = get_pg_datalogger() if pg_datalogger and pg_datalogger.enabled: # Retrieve indicators stored on the position entry_indicators = getattr(self.active_position, '_entry_indicators', {}) # ... complex fallback logic if indicators are missing ... # Prepare a massive dictionary with dozens of fields for the database trade_data = { 'symbol': self.active_position.symbol, 'net_pnl_usdt': result['net_pnl_usdt'], # ... dozens more fields ... 'entry_indicators': entry_indicators, 'config_snapshot': get_full_config_snapshot(), } # Directly call the logger with the complex data structure pg_datalogger.log_trade(trade_data=trade_data, ...) except Exception as e: logger.warning(f"Error logging trade: {e}") # ... more logic ... return result After: # In core/position_manager.py def close_position(self, exit_price: float, reason: str): # ... core logic to calculate PnL ... pnl_data = self.pnl_calculator.calculate_pnl(...) result = { ... } # Final trade result object # Emit a simple event with the necessary context. # The complex data preparation is handled by a separate service. event_bus.emit('PositionClosed', { 'position_context': self.active_position.to_dict(), 'trade_result': result, 'close_reason': reason, 'scan_log_id': getattr(self.active_position, '_scan_log_id', None) }) # ... more logic ... return result # In a separate, asynchronous data_logging_service.py @event_bus.on('PositionClosed') async def handle_position_closed(event_data: dict): # Complex data enrichment and transformation happens here, # decoupled from the critical trading path. trade_data = await enrich_trade_data_for_db(event_data) pg_datalogger.log_trade(trade_data) Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical design flaw, as the PR tightly couples complex data preparation for logging within core trading functions like analyze_pair and close_position, harming maintainability and architectural clarity.	High
Possible issue	Correctly calculate maximum drawdown metric Correct the maximum drawdown calculation in log_trade to iterate through PnL history and find the largest drop from a running maximum, instead of just calculating the range between the highest and lowest PnL. core/postgresql_datalogger.py [889-917] if pnl_history: pnl_pcts = [p.get('pnl_pct', 0) for p in pnl_history if p.get('pnl_pct') is not None] pnl_usdts = [p.get('pnl_usdt', 0) for p in pnl_history if p.get('pnl_usdt') is not None] if pnl_pcts: - max_favorable_excursion = max(pnl_pcts) - max_adverse_excursion = min(pnl_pcts) + max_favorable_excursion = max(pnl_pcts) if pnl_pcts else 0 + max_adverse_excursion = min(pnl_pcts) if pnl_pcts else 0 # Calculer entry_to_max_profit_price_change_pct et entry_to_max_loss_price_change_pct ... # Calculer max_drawdown (drawdown depuis le profit max) - if max_favorable_excursion and max_adverse_excursion: - max_drawdown_pct = max_favorable_excursion - max_adverse_excursion if max_favorable_excursion > 0 else abs(max_adverse_excursion) + running_max_pct = 0 + max_drawdown_pct = 0 + for pnl_pct in pnl_pcts: + running_max_pct = max(running_max_pct, pnl_pct) + max_drawdown_pct = max(max_drawdown_pct, running_max_pct - pnl_pct) if pnl_usdts: - max_favorable_excursion_usdt = max(pnl_usdts) - max_adverse_excursion_usdt = min(pnl_usdts) - if max_favorable_excursion_usdt and max_adverse_excursion_usdt: - max_drawdown_usdt = max_favorable_excursion_usdt - max_adverse_excursion_usdt if max_favorable_excursion_usdt > 0 else abs(max_adverse_excursion_usdt) + max_favorable_excursion_usdt = max(pnl_usdts) if pnl_usdts else 0 + max_adverse_excursion_usdt = min(pnl_usdts) if pnl_usdts else 0 + running_max_usdt = 0 + max_drawdown_usdt = 0 + for pnl_usdt in pnl_usdts: + running_max_usdt = max(running_max_usdt, pnl_usdt) + max_drawdown_usdt = max(max_drawdown_usdt, running_max_usdt - pnl_usdt) [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a bug in the calculation of max_drawdown_pct and max_drawdown_usdt, which currently computes the range instead of the actual maximum drawdown. Fixing this is crucial for accurate performance metric logging, which is the core purpose of this new datalogger module.	Medium
	Avoid performance issues in logging To improve performance, instantiate ColoredFormatter once in the WebSocketLogHandler's init method instead of on every emit call, and reuse the instance. utils/logger.py [21-82] +def __init__(self): + super().__init__() + self.ws_manager = None + # Instantiate formatter once to avoid performance issues + self.formatter = ColoredFormatter( + '[%(asctime)s] %(levelname)s - %(message)s', + datefmt='%H:%M:%S' + ) + self.color_codes = { + 'DEBUG': '\x1b[36m', 'INFO': '\x1b[32m', 'WARNING': '\x1b[33m', + 'ERROR': '\x1b[31m', 'CRITICAL': '\x1b[35m' + } + self.reset_code = '\x1b[0m' + +def set_ws_manager(self, ws_manager): + """Définir le websocket manager""" + self.ws_manager = ws_manager + def emit(self, record): """Envoyer le log au frontend""" try: if not self.ws_manager: return + + level = record.levelname + raw_message = record.getMessage() - # Convertir le niveau de logging en string - level_map = { - logging.DEBUG: 'DEBUG', - logging.INFO: 'INFO', - logging.WARNING: 'WARNING', - logging.ERROR: 'ERROR', - logging.CRITICAL: 'CRITICAL' + # Use pre-defined color codes to create the colored message + color = self.color_codes.get(level, '') + colored_message = f"{color}{raw_message}{self.reset_code}" if color else raw_message + + entry = { + 'timestamp': datetime.fromtimestamp(record.created).strftime('%H:%M:%S'), + 'level': level, + 'message': colored_message, + 'detail': '', + 'raw_message': raw_message } - level = level_map.get(record.levelno, 'INFO') - - # 🔥 FIX: Formater le message avec le ColoredFormatter pour préserver les couleurs ANSI et emojis - # Utiliser le formatter pour obtenir les couleurs ANSI - formatter = ColoredFormatter( - '[%(asctime)s] %(levelname)s - %(message)s', - datefmt='%H:%M:%S' - ) - formatted_message = formatter.format(record) - - # Extraire le message avec couleurs (tout après le timestamp et level) - # Format: [HH:MM:SS] LEVEL - message - # On veut garder le message avec ses couleurs ANSI - message_with_colors = record.getMessage() - # Ajouter les couleurs ANSI selon le niveau - color_codes = { - 'DEBUG': '\x1b[36m', # Cyan - 'INFO': '\x1b[32m', # Vert - 'WARNING': '\x1b[33m', # Jaune - 'ERROR': '\x1b[31m', # Rouge - 'CRITICAL': '\x1b[35m', # Magenta - } - reset_code = '\x1b[0m' - color = color_codes.get(level, '') - colored_message = f"{color}{message_with_colors}{reset_code}" if color else message_with_colors - - # Créer l'entrée de log - entry = { - 'timestamp': datetime.now().strftime('%H:%M:%S'), - 'level': level, - 'message': colored_message, # Message avec couleurs ANSI et emojis préservés - 'detail': '', - 'raw_message': message_with_colors # Message sans couleur pour recherche - } - + # Envoyer via WebSocket (asynchrone, donc on crée une tâche) import asyncio try: loop = asyncio.get_running_loop() async def send_log(): await self.ws_manager.emit('log', entry) loop.create_task(send_log()) except RuntimeError: # Pas de loop en cours, ignorer pass except Exception: # Ne pas bloquer le logging si l'envoi échoue pass [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 6 __ Why: The suggestion correctly identifies a performance issue by instantiating ColoredFormatter on every call to emit and proposes a valid fix, improving code quality and efficiency.	Low
General	Simplify asynchronous task creation Simplify asynchronous task creation by using asyncio.create_task() directly, which avoids deprecated asyncio.get_event_loop() usage and potential blocking issues. core/position_manager.py [627-671] try: - loop = asyncio.get_event_loop() - if loop.is_running(): - # Créer une task non-bloquante - async def log_entry(): - trade_id = await data_logger.log_trade_entry( - ... - ) - # Stocker trade_id dans position - if trade_id: - self.active_position._trade_id = trade_id - loop.create_task(log_entry()) - else: - # Pas de loop, créer un nouveau - trade_id = loop.run_until_complete(data_logger.log_trade_entry( - ... - )) + import asyncio + # Créer une task non-bloquante + async def log_entry(): + trade_id = await data_logger.log_trade_entry( + opportunity_id=opportunity_id, + scan_log_id=scan_uuid, + symbol=symbol, + direction=direction, + entry_price=entry, + size_usdt=size, + tp_price=tp, + sl_price=sl, + tp_sl_mode=TRADING_CONFIG.get('tp_sl_mode', 'FIXE'), + entry_indicators=entry_indicators, + entry_conditions=entry_conditions, + entry_scalability=entry_scalability + ) # Stocker trade_id dans position if trade_id: self.active_position._trade_id = trade_id -except RuntimeError: - # Pas de loop disponible, ignorer - pass + + asyncio.create_task(log_entry()) +except Exception as e: + logger.debug(f"Erreur log_trade_entry (non-bloquant): {e}") + [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 6 __ Why: The suggestion correctly identifies that using asyncio.get_event_loop() is discouraged and proposes a modern, simpler, and more robust approach with asyncio.create_task().	Low
	Simplify redundant conditional checks Simplify the conditional checks for best_bid and best_ask by removing the redundant len() > 0 checks, as truthiness checks on the lists are sufficient. core/analyzer/market_data.py [47-48] -best_bid = bids[0][0] if bids and len(bids) > 0 and len(bids[0]) > 0 else 0 -best_ask = asks[0][0] if asks and len(asks) > 0 and len(asks[0]) > 0 else 0 +best_bid = bids[0][0] if bids and bids[0] else 0 +best_ask = asks[0][0] if asks and asks[0] else 0 Apply / Chat Suggestion importance[1-10]: 2 __ Why: The suggestion correctly points out a redundant check and proposes a valid simplification, which slightly improves code readability.	Low
More