PROMEX: A Prometheus exporter for HAProxy

See also

• https://github.com/haproxy/haproxy/tree/master/contrib/prometheus-exporter

• https://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=f959d0809ead017555ef5db60ef081a7fd87ca20

• https://github.com/prometheus/haproxy_exporter

PROMEX: A Prometheus exporter for HAProxy

Prometheus is a monitoring and alerting system. More and more people use it to monitor their environment (this is written February 2019). It collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets.

For HAProxy, The Prometheus team offically supports an exporter written in :ref:Go <go_language>` (https://github.com/prometheus/haproxy_exporter).

But it requires an extra software to deploy and monitor.

PROMEX, on its side, is a built-in Prometheus exporter for HAProxy.

It was developed as a service and is directly available in HAProxy, like the stats applet.

However, PROMEX is not built by default with HAProxy. It is provided as an extra component for everyone want to use it. So you need to explicity build HAProxy with the PROMEX service, using the Makefile variable “EXTRA_OBJS”. For instance:

> make TARGET=linux2628 EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"

if HAProxy provides the PROMEX service, the following build option will be reported by the command “haproxy -vv”:

Built with the Prometheus exporter as a service

To be used, it must be enabled in the configuration with an “http-request” rule and the corresponding HTTP proxy must enable the HTX support. For instance:

frontend test
    mode http
    ...
    option http-use-htx
    http-request use-service prometheus-exporter if { path /metrics }
    ...

This service has been developed as a third-party component because it could become obsolete, depending on how much time Prometheus will remain heavily used. This is said with no ulterior motive of course. Prometheus is a great software and I hope all the well for it. But we involve in a environment moving quickly and a solution may be obvious today could be deprecated the next year. And because PROMEX is not integrated by default into the HAProxy codebase, it will need some interest to be actively supported. All contribution of any kind are welcome.

You must also be careful if you use with huge configurations. Unlike the stats applet, all metrics are not grouped by service (proxy, listener or server). With PROMEX, all lines for a given metric are provided as one single group. So instead of collecting all metrics for a proxy before moving to the next one, we must loop on all proxies for each metric. Same for the servers. Thus, it will spend much more ressources to produce the Prometheus metrics than the CSV export through the stats page. To give a comparison order, quick benchmarks shown that a PROMEX dump is 5x slower and 20x more verbose than a CSV export.

Exported metrics

• Globals metrics

Metric name

Description

haproxy_process_nbthread haproxy_process_nbproc haproxy_process_relative_process_id haproxy_process_uptime_s econds haproxy_process_max_memo ry haproxy_process_pool_all ocated_total haproxy_process_pool_use d_total haproxy_process_pool_fai lures_total haproxy_process_max_fds haproxy_process_max_sock ets haproxy_process_max_conn ections haproxy_process_hard_max _connections haproxy_process_current_ connections haproxy_process_connectio ns_total haproxy_process_requests_total haproxy_process_max_ssl_connections haproxy_process_current_ ssl_connections haproxy_process_ssl_conn ections_total haproxy_process_max_pipe s haproxy_process_pipes_us ed_total haproxy_process_pipes_fr ee_total haproxy_process_current_ connection_rate haproxy_process_limit_co nnection_rate haproxy_process_max_conn ection_rate haproxy_process_current_ session_rate haproxy_process_limit_se ssion_rate haproxy_process_max_sess ion_rate haproxy_process_current_ ssl_rate haproxy_process_limit_ss l_rate haproxy_process_max_ssl_rate haproxy_process_current_ frontend_ssl_key_rate haproxy_process_max_fron tend_ssl_key_rate haproxy_process_frontent_ssl_reuse haproxy_process_current_ backend_ssl_key_rate haproxy_process_max_back end_ssl_key_rate haproxy_process_ssl_cach e_lookups haproxy_process_ssl_cach e_misses haproxy_process_http_com p_bytes_in_total haproxy_process_http_com p_bytes_out_total haproxy_process_limit_ht tp_comp haproxy_process_current_ zlib_memory haproxy_process_max_zlib _memory haproxy_process_current_ tasks haproxy_process_current_ run_queue haproxy_process_idle_tim e_percent haproxy_process_stopping haproxy_process_jobs haproxy_process_unstoppab le_jobs haproxy_process_listeners haproxy_process_active_p eers haproxy_process_connected _peers haproxy_process_dropped_ logs_total haproxy_process_busy_pol ling_enabled

Configured number of threads. Configured number of processes. Relative process id, starting at 1. Uptime in seconds. Per-process memory limit (in MB); 0=unset. Total amount of memory allocated in pools (in MB). Total amount of memory used in pools (in MB). Total number of failed pool allocations. Maximum number of open file descriptors; 0=unset. Maximum numer of open sockets. Maximum number of concurrent connections. Initial Maximum number of concurrent connections. Number of active sessions. Total number of terminated sessions. Total number of requests (TCP or HTTP). Configured maximum number of concurrent SSL connections. Number of opened SSL connections. Total number of opened SSL connections. Configured maximum number of pipes. Number of pipes in used. Number of pipes unused. Current number of connections per second over last elapsed second. Configured maximum number of connections per second. Maximum observed number of connections per second. Current number of sessions per second over last elapsed second. Configured maximum number of sessions per second. Maximum observed number of sessions per second. Current number of SSL sessions per second over last elapsed second. Configured maximum number of SSL sessions per second. Maximum observed number of SSL sessions per second. Current frontend SSL Key computation per second over last elapsed second. Maximum observed frontend SSL Key computation per second. SSL session reuse ratio (percent). Current backend SSL Key computation per second over last elapsed second. Maximum observed backend SSL Key computation per second. Total number of SSL session cache lookups. Total number of SSL session cache misses. Number of bytes per second over last elapsed second, before http compression. Number of bytes per second over last elapsed second, after http compression. Configured maximum input compression rate in bytes. Current memory used for zlib in bytes. Configured maximum amount of memory for zlib in bytes. Current number of tasks. Current number of tasks in the run-queue. Idle to total ratio over last sample (percent). Non zero means stopping in progress. Current number of active jobs (listeners, sessions, open devices). Current number of active jobs that can’t be stopped during a soft stop. Current number of active listeners. Current number of active peers. Current number of connected peers. Total number of dropped logs. Non zero if the busy polling is enabled.

• Frontend metrics

Metric name

Description

haproxy_frontend_status haproxy_frontend_current_sessions haproxy_frontend_max_ses sions haproxy_frontend_limit_s essions haproxy_frontend_sessions _total haproxy_frontend_current_session_rate haproxy_frontend_limit_s ession_rate haproxy_frontend_max_ses sion_rate haproxy_frontend_connecti ons_rate_current haproxy_frontend_connecti ons_rate_max haproxy_frontend_connecti ons_total haproxy_frontend_bytes_i n_total haproxy_frontend_bytes_o ut_total haproxy_frontend_requests _denied_total haproxy_frontend_response s_denied_total haproxy_frontend_request_errors_total haproxy_frontend_denied_ connections_total haproxy_frontend_denied_ sessions_total haproxy_frontend_failed_ header_rewriting_total haproxy_frontend_http_re quests_rate_current haproxy_frontend_http_re quests_rate_max haproxy_frontend_http_re quests_total haproxy_frontend_http_re sponses_total haproxy_frontend_intercep ted_requests_total haproxy_frontend_http_ca che_lookups_total haproxy_frontend_http_ca che_hits_total haproxy_frontend_http_co mp_bytes_in_total haproxy_frontend_http_co mp_bytes_out_total haproxy_frontend_http_co mp_bytes_bypassed_total haproxy_frontend_http_co mp_responses_total

Current status of the service. Current number of active sessions. Maximum observed number of active sessions. Configured session limit. Total number of sessions. Current number of sessions per second over last elapsed second. Configured limit on new sessions per second. Maximum observed number of sessions per second. Current number of connections per second over the last elapsed second. Maximum observed number of connections per second. Total number of connections. Current total of incoming bytes. Current total of outgoing bytes. Total number of denied requests. Total number of denied responses. Total number of request errors. Total number of requests denied by “tcp-request connection” rules. Total number of requests denied by “tcp-request session” rules. Total number of failed header rewriting warnings. Current number of HTTP requests per second over last elapsed second. Maximum observed number of HTTP requests per second. Total number of HTTP requests received. Total number of HTTP responses. Total number of intercepted HTTP requests. Total number of HTTP cache lookups. Total number of HTTP cache hits. Total number of HTTP response bytes fed to the compressor. Total number of HTTP response bytes emitted by the compressor. Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). Total number of HTTP responses that were compressed.

• Backend metrics

Metric name

Description

haproxy_backend_status haproxy_backend_current_ses sions haproxy_backend_max_session s haproxy_backend_limit_sessi ons haproxy_backend_sessions_to tal haproxy_backend_current_ses sion_rate haproxy_backend_max_session _rate haproxy_backend_last_sessio n_seconds haproxy_backend_current_que ue haproxy_backend_max_queue haproxy_backend_connection_ attempts_total haproxy_backend_connection_ reuses_total haproxy_backend_bytes_in_t otal haproxy_backend_bytes_out_ total haproxy_backend_http_queue_time_average_seconds haproxy_backend_http_connec t_time_average_seconds haproxy_backend_http_respon se_time_average_seconds haproxy_backend_http_total_time_average_seconds haproxy_backend_requests_de nied_total haproxy_backend_responses_d enied_total haproxy_backend_connection_ errors_total haproxy_backend_response_er rors_total haproxy_backend_retry_warni ngs_total haproxy_backend_redispatch_ warnings_total haproxy_backend_failed_head er_rewriting_total haproxy_backend_client_abor ts_total haproxy_backend_server_abor ts_total haproxy_backend_weight haproxy_backend_active_serv ers haproxy_backend_backup_serv ers haproxy_backend_check_up_d own_total haproxy_backend_check_last_change_seconds haproxy_backend_downtime_se conds_total haproxy_backend_loadbalanced _total haproxy_backend_http_reques ts_total haproxy_backend_http_respon ses_total haproxy_backend_http_cache_lookups_total haproxy_backend_http_cache_hits_total haproxy_backend_http_comp_ bytes_in_total haproxy_backend_http_comp_ bytes_out_total haproxy_backend_http_comp_ bytes_bypassed_total haproxy_backend_http_comp_ responses_total

Current status of the service. Current number of active sessions. Maximum observed number of active sessions. Configured session limit. Total number of sessions. Current number of sessions per second over last elapsed second. Maximum observed number of sessions per second. Number of seconds since last session assigned to server/backend. Current number of queued requests. Maximum observed number of queued requests. Total number of connection establishment attempts. Total number of connection reuses. Current total of incoming bytes. Current total of outgoing bytes. Avg. queue time for last 1024 successful connections. Avg. connect time for last 1024 successful connections. Avg. response time for last 1024 successful connections. Avg. total time for last 1024 successful connections. Total number of denied requests. Total number of denied responses. Total number of connection errors. Total number of response errors. Total number of retry warnings. Total number of redispatch warnings. Total number of failed header rewriting warnings. Total number of data transfers aborted by the client. Total number of data transfers aborted by the server. Service weight. Current number of active servers. Current number of backup servers. Total number of UP->DOWN transitions. Number of seconds since the last UP<->DOWN transition. Total downtime (in seconds) for the service. Total number of times a service was selected. Total number of HTTP requests received. Total number of HTTP responses. Total number of HTTP cache lookups. Total number of HTTP cache hits. Total number of HTTP response bytes fed to the compressor. Total number of HTTP response bytes emitted by the compressor. Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). Total number of HTTP responses that were compressed.

• Server metrics

Metric name

Description

haproxy_server_status haproxy_server_current_ses sions haproxy_server_max_session s haproxy_server_limit_sessi ons haproxy_server_sessions_to tal haproxy_server_current_ses sion_rate haproxy_server_max_session _rate haproxy_server_last_sessio n_seconds haproxy_server_current_que ue haproxy_server_max_queue haproxy_server_queue_limit haproxy_server_bytes_in_t otal haproxy_server_bytes_out_ total haproxy_server_http_queue_time_average_seconds haproxy_server_http_connec t_time_average_seconds haproxy_server_http_respon se_time_average_seconds haproxy_server_http_total_time_average_seconds haproxy_server_connection_ attempts_total haproxy_server_connection_ reuses_total haproxy_server_responses_d enied_total haproxy_server_connection_ errors_total haproxy_server_response_er rors_total haproxy_server_retry_warni ngs_total haproxy_server_redispatch_ warnings_total haproxy_server_failed_head er_rewriting_total haproxy_server_client_abor ts_total haproxy_server_server_abor ts_total haproxy_server_weight haproxy_server_check_failu res_total haproxy_server_check_up_d own_total haproxy_server_downtime_se conds_total haproxy_server_check_last_change_seconds haproxy_server_current_thr ottle haproxy_server_loadbalanced _total haproxy_server_http_respon ses_total

Current status of the service. Current number of active sessions. Maximum observed number of active sessions. Configured session limit. Total number of sessions. Current number of sessions per second over last elapsed second. Maximum observed number of sessions per second. Number of seconds since last session assigned to server/backend. Current number of queued requests. Maximum observed number of queued requests. Configured maxqueue for the server (0 meaning no limit). Current total of incoming bytes. Current total of outgoing bytes. Avg. queue time for last 1024 successful connections. Avg. connect time for last 1024 successful connections. Avg. response time for last 1024 successful connections. Avg. total time for last 1024 successful connections. Total number of connection establishment attempts. Total number of connection reuses. Total number of denied responses. Total number of connection errors. Total number of response errors. Total number of retry warnings. Total number of redispatch warnings. Total number of failed header rewriting warnings. Total number of data transfers aborted by the client. Total number of data transfers aborted by the server. Service weight. Total number of failed check (Only when the server is up). Total number of UP->DOWN transitions. Total downtime (in seconds) for the service. Number of seconds since the last UP<->DOWN transition. Current throttle percentage for the server, when slowstart is active. Total number of times a service was selected. Total number of HTTP responses.