chore: observability and security for HTTP gateway

[shumkov]

Issue being fixed or feature implemented

Currently, to handle HTTP requests in Evonode we use outdated version of Envoy This version contains many security vulnerabilities and is not configured properly for being an edge proxy for all incoming HTTP traffic. To control and limit the load it should provide mechanisms to limit incoming requests. We don't have any visibility on HTTP traffic in general and specific requests to Drive and DAPI.

What was done?

• Updated Envoy and configuration to latest version 1.30.1
• Renamed DAPI Envoy service to Platform Gateway since it handles requests not just for DAPI but for Drive as well
• Added configuration platform.gateway.maxConnections to limit max active connections to Gateway. If we close to the limit, we disable keep-alive, and stop accepting requests when we reach it.
• Added configuration platform.gateway.maxHeapSizeInBytes to release free memory and disable keep-alive when we close to limit and reject incoming connections when we reach it.
• Added configuration platform.gateway.upstreams.*.maxRequest to limit active requests to specific upstream (DAPI, Drive, etc)
• Added configuration platform.gateway.metrics to expose Envoy Prometheus metrics.
• Added configuration platform.gateway.admin to enable Envoy admin interface (must be enabled for metrics).
• Added configuration platform.gateway.listeners.*.http2.maxConcurrentStreams to limit max concurrent streams per incoming HTTP2 connection.
• Added configuration platform.gateway.log.level to define verbosity of Envoy application logs.
• Added configuration platform.gateway.log.accessLogs to define any number of access logs. Supported destinations are stdout, stderr and file. Supported formats are text or json. Default output templates are defined for each format and can be ovewritten for any specific access logs.
• Global HTTP request rate limiter replaced with IP-based so now it limits per IP but not for all incoming requests. Whitelisting and blacklisting are now supported as well.
• Added configuration platfrom.gateway.rateLimiter.metrics to expose Rate limiter Prometheus metrics.
• Added configuration platform.drive.abci.metrics to expose Drive ABCI Prometheus metrics.
• the dapi_tx_filter_stream docker compose service renamed to dapi_core_streams since it expose multiple streaming endpoints.
• Configured Envoy to sanitize incoming request path
• Configured Envoy to limit the connection and request max duration and idle timeouts
• Configured Envoy to limit http2 connection and stream window sizes
• Rejected external requests to getProofs since this is internal endpoint between DAPI and Drive and can be misused to make a huge load on the system.
• Reduced new connection read and write buffers to reduce memory consumption
• Forced more efficient HTTP2 connections between Envoy and upstreams
• Configured Envoy to expose Prometheus metrics with default path instead of /stats/prometheus
• Added metrics for Drive ABCI queries

How Has This Been Tested?

Multiple testing scenarios to verify each configuration on the ouzo devnet with metric visualisations in Grafana

Breaking Changes

None

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
• I have made corresponding changes to the documentation if needed

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

[QuantumExplorer]

Approving without review.