Prometheus๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Flask ๋งˆ์ดํฌ๋กœ์„œ๋น„์Šค ๋ชจ๋‹ˆํ„ฐ๋ง

๋ช‡ ์ค„์˜ ์ฝ”๋“œ์™€ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์ด ์ธก์ •ํ•ญ๋ชฉ์„ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค. ์™€!

prometheus_์˜ ์ž‘๋™ ๋ฐฉ์‹์„ ์ดํ•ดํ•˜๋ ค๋ฉดํ”Œ๋ผ์Šคํฌ_์ˆ˜์ถœ์—…์ž๋Š” ์ตœ์†Œํ•œ์˜ ์˜ˆ๋งŒ์œผ๋กœ ์ถฉ๋ถ„ํ•ฉ๋‹ˆ๋‹ค:

from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)

@app.route('/')
def main():
    return 'OK'

์‹œ์ž‘ํ•˜๋Š” ๋ฐ ํ•„์š”ํ•œ ์ „๋ถ€์ž…๋‹ˆ๋‹ค! ์ดˆ๊ธฐํ™”๋ฅผ ์œ„ํ•œ ๊ฐ€์ ธ์˜ค๊ธฐ ๋ฐ ๋ผ์ธ ์ถ”๊ฐ€ Prometheus๋ฉ”ํŠธ๋ฆญ์Šค, ์ธก์ •ํ•ญ๋ชฉ์„ ์–ป๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ์š”์ฒญ ๊ธฐ๊ฐ„ ะธ ์š”์ฒญ ์นด์šดํ„ฐ, ์—”๋“œํฌ์ธํŠธ์— ํ‘œ์‹œ๋จ /์ธก์ •ํ•ญ๋ชฉ ๋“ฑ๋ก๋œ Flask ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜๊ณผ ๊ธฐ๋ณธ์—์„œ ์–ป๋Š” ๋ชจ๋“  ๊ธฐ๋ณธ ์ธก์ •ํ•ญ๋ชฉ ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ํด๋ผ์ด์–ธํŠธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ.

๋‹น์‹ ์€ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค ์‚ฌ์šฉํ•˜๊ธฐ ์‰ฌ์šด ์˜ˆ ์ธ์Šคํ„ด์Šค๋ฅผ ์‹คํ–‰ํ•˜๋Š” GitHub ์ €์žฅ์†Œ์—์„œ ํ”„๋กœ ๋ฉ”ํ…Œ์šฐ์Šค ะธ ๊ทธ๋ผ ํŒŒ๋‚˜ ๋ฐ๋ชจ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜๊ณผ ํ•จ๊ป˜ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ธก์ •ํ•ญ๋ชฉ์„ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค.

Prometheus๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ Flask ๋งˆ์ดํฌ๋กœ์„œ๋น„์Šค ๋ชจ๋‹ˆํ„ฐ๋ง

๋˜ํ•œ ๋‹ค์Œ์—์„œ ์ง€ํ‘œ ๋ชฉ๋ก์„ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. README ๋Œ€์‹œ๋ณด๋“œ๋ฅผ ์ฑ„์šฐ๋Š” Prometheus ์ฟผ๋ฆฌ์™€ ํ•จ๊ป˜ ๋Œ€์‹œ๋ณด๋“œ์— ํ‘œ์‹œ๋˜๋Š” ์˜ˆ์ž…๋‹ˆ๋‹ค.

์กฐ์ •

๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์—๋Š” ๋งŽ์€ ๊ตฌ์„ฑ ์˜ต์…˜์ด ์žˆ์Šต๋‹ˆ๋‹ค. README ๊ฐ„๋‹จํ•œ ์„ค๋ช…๊ณผ ํ•จ๊ป˜ ํ”„๋กœ์ ํŠธ ์‚ฌ๋ก€๋ฅผ ์†Œ๊ฐœํ•ฉ๋‹ˆ๋‹ค.

๊ธฐ๋ณธ ๊ตฌ์„ฑ์€ ์œ„์— ๋‚˜์™€ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ƒฅ ์ธ์Šคํ„ด์Šค๋ฅผ ๋งŒ๋“ค์–ด ๋ณด์„ธ์š” Prometheus๋ฉ”ํŠธ๋ฆญ์Šค, ๋ถˆ๋Ÿฌ๋ณด์ž ํ†ต๊ณ„, ๊ทธ๋ฆฌ๊ณ  ์ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํ•จ์ˆ˜๋ฅผ ์žฅ์‹ํ•˜์—ฌ ์ˆ˜์ง‘ํ•˜๋ ค๋Š” ์ถ”๊ฐ€ ์ธก์ •ํ•ญ๋ชฉ์„ ์ •์˜ํ•ฉ๋‹ˆ๋‹ค.

  • @metrics.counter(..)

  • @metrics.gauge(..)

  • @metrics.summary(..)

  • @metrics.histogram(..)

์นด์šดํ„ฐ๋Š” ํ†ตํ™” ์ˆ˜๋ฅผ ๊ณ„์‚ฐํ•˜๊ณ  ๋‹ค๋ฅธ ์นด์šดํ„ฐ๋Š” ํ†ตํ™” ๊ธฐ๊ฐ„์„ ๊ธฐ์ค€์œผ๋กœ ์ธก์ •ํ•ญ๋ชฉ์„ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค. ์ž ์žฌ์ ์œผ๋กœ ์š”์ฒญ ๋˜๋Š” ์‘๋‹ต ์†์„ฑ์„ ์‚ฌ์šฉํ•˜์—ฌ ์ด๋“ค ๊ฐ๊ฐ์— ๋Œ€ํ•œ ๋ ˆ์ด๋ธ”์„ ์ •์˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด:

from flask import Flask, request
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)

# group by endpoint rather than path
metrics = PrometheusMetrics(app, group_by='endpoint')

@app.route('/collection/:collection_id/item/:item_id')
@metrics.counter(
    'cnt_collection', 'Number of invocations per collection', labels={
        'collection': lambda: request.view_args['collection_id'],
        'status': lambda resp: resp.status_code
    })
def get_item_from_collection(collection_id, item_id):
    pass

์œ„์˜ ์˜ˆ์—์„œ ๋์ ์„ ํด๋ฆญํ•˜๋ฉด /์ปฌ๋ ‰์…˜/10002/ํ•ญ๋ชฉ/76 ์˜ˆ๋ฅผ ๋“ค์–ด ์นด์šดํ„ฐ๊ฐ€ ์ฆ๊ฐ€ํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. cnt_collection{collection = "10002", ์ƒํƒœ = "200"}, ๋˜ํ•œ ๊ธฐ๋ณธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์—์„œ ๊ธฐ๋ณธ ์ธก์ •ํ•ญ๋ชฉ(์ด ์˜ˆ์—์„œ๋Š” ๊ฐ ์—”๋“œํฌ์ธํŠธ์— ๋Œ€ํ•œ)์„ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

  • flask_http_request_duration_seconds โ€” ๋ฉ”์„œ๋“œ, ๊ฒฝ๋กœ ๋ฐ ์ƒํƒœ๋ณ„ ๋ชจ๋“  Flask ์š”์ฒญ์— ๋Œ€ํ•œ HTTP ์š”์ฒญ ๊ธฐ๊ฐ„(์ดˆ)

  • flask_http_request_total โ€” ๋ฉ”์†Œ๋“œ ๋ฐ ์ƒํƒœ๋ณ„ ์ด HTTP ์š”์ฒญ ์ˆ˜

ํŠน์ • ์—”๋“œํฌ์ธํŠธ ์ถ”์ ์„ ๊ฑด๋„ˆ๋›ฐ๊ฑฐ๋‚˜, ์ถ”๊ฐ€ ๊ธฐ๋ณธ ์ง€ํ‘œ๋ฅผ ๊ธฐ๋กํ•˜๊ฑฐ๋‚˜, ์œ„์— ๋‚˜์—ด๋œ ์ง€ํ‘œ๋ฅผ ๊ฑด๋„ˆ๋›ฐ๊ฑฐ๋‚˜, ๋™์ผํ•œ ์‚ฌ์šฉ์ž ์ง€์ • ์ง€ํ‘œ๋ฅผ ์—ฌ๋Ÿฌ ์—”๋“œํฌ์ธํŠธ์— ์ ์šฉํ•˜๋Š” ์˜ต์…˜์ด ์žˆ์Šต๋‹ˆ๋‹ค. ํ™•์ธํ•ด ๋ณด์„ธ์š” README ๋ฌด์—‡์ด ์ด์šฉ ๊ฐ€๋Šฅํ•œ์ง€ ์•Œ์•„๋ณด๋Š” ํ”„๋กœ์ ํŠธ์ž…๋‹ˆ๋‹ค.

app = Flask(__name__)
metrics = PrometheusMetrics(app)

@app.route('/')
def main():
    pass  # requests tracked by default

@app.route('/skip')
@metrics.do_not_track()
def skip():
    pass  # default metrics are not collected

# custom metric to be applied to multiple endpoints
common_counter = metrics.counter(
    'by_endpoint_counter', 'Request count by endpoints',
    labels={'endpoint': lambda: request.endpoint}
)

@app.route('/common/one')
@common_counter
def endpoint_one():
    pass  # tracked by the custom and the default metrics

@app.route('/common/two')
@common_counter
def endpoint_two():
    pass  # also tracked by the custom and the default metrics

# register additional default metrics
metrics.register_default(
    metrics.counter(
        'by_path_counter', 'Request count by request paths',
        labels={'path': lambda: request.path}
    )
)

์ด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์—๋Š” uWSGI ๋ฐ Gunicorn๊ณผ ๊ฐ™์€ ๋„๋ฆฌ ์‚ฌ์šฉ๋˜๋Š” ๋‹ค์ค‘ ์ฒ˜๋ฆฌ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์— ๋Œ€ํ•œ ํŽธ๋ฆฌํ•œ ํ™•์žฅ ๊ธฐ๋Šฅ์ด ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์ค‘ ์ฒ˜๋ฆฌ๋ฅผ ํฌํ•จํ•˜์—ฌ ๋Œ€์ƒ ์‚ฌ์šฉ ์‚ฌ๋ก€์˜ ์ž‘์€ ์˜ˆ๋„ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ธก์ •ํ•ญ๋ชฉ ์ˆ˜์ง‘

์œ„์—์„œ ์–ธ๊ธ‰ํ–ˆ๋“ฏ์ด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ ์—”๋“œํฌ์ธํŠธ๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. /์ธก์ •ํ•ญ๋ชฉ Flask ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์—์„œ ๋Œ€์ƒ์œผ๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ๋นŒ๋”.

์œ„์˜ ๋Œ€์‹œ๋ณด๋“œ ์˜ˆ์—์„œ๋Š” ๋‹ค์Œ ๊ตฌ์„ฑ์„ ์‚ฌ์šฉํ•˜์—ฌ ๊ธฐ๋ณธ ์„ค์ •์„ ์‚ฌ์šฉํ•˜์—ฌ Prometheus๋ฅผ Flask ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์œผ๋กœ ํƒ€๊ฒŸํŒ…ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

scrape_configs:
  - job_name: 'example'

    dns_sd_configs:
      - names: ['app']
        port: 5000
        type: A
        refresh_interval: 5s

์ „์ฒด ์˜ˆ์‹œ๋Š” ๋‹ค์Œ์—์„œ ํ™•์ธํ•˜์„ธ์š”. GitHub ์ €์žฅ์†Œ. ์ด๋Š” Prometheus๊ฐ€ Flask ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ธ์Šคํ„ด์Šค๋ฅผ ์ฐพ์„ ์ˆ˜ ์žˆ๋‹ค๊ณ  ๊ฐ€์ •ํ•ฉ๋‹ˆ๋‹ค. http://app:5000/metrics, ์—ฌ๊ธฐ์„œ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋„๋ฉ”์ธ ์ด๋ฆ„์€ ์ž ์žฌ์ ์œผ๋กœ ์—ฌ๋Ÿฌ IP ์ฃผ์†Œ๋กœ ํ™•์ธ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค(์˜ˆ: ๋‹ค์Œ์—์„œ ์‹คํ–‰ ์ค‘์ธ ๊ฒฝ์šฐ). Kubernetes ๋˜๋Š” ๋„์ปค ์Šค์›œ.

์ด๋Ÿฌํ•œ ๋ฐฉ์‹์œผ๋กœ ์ธก์ •ํ•ญ๋ชฉ ๋์ ์„ ๋…ธ์ถœํ•˜๋Š” ๊ฒƒ์ด ์ ํ•ฉํ•˜์ง€ ์•Š์€ ๊ฒฝ์šฐ(์•„๋งˆ๋„ ์™ธ๋ถ€ ์•ก์„ธ์Šค๋ฅผ ํ—ˆ์šฉํ•˜๊ณ  ์‹ถ์ง€ ์•Š๊ธฐ ๋•Œ๋ฌธ์—) ๋‹ค์Œ์„ ์ „๋‹ฌํ•˜์—ฌ ์‰ฝ๊ฒŒ ๋น„ํ™œ์„ฑํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ฒฝ๋กœ=์—†์Œ ์ธ์Šคํ„ด์Šค๋ฅผ ์ƒ์„ฑํ•  ๋•Œ Prometheus๋ฉ”ํŠธ๋ฆญ์Šค.

from flask import Flask, request
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app, path=None)

...

metrics.start_http_server(5099)

๊ทธ๋Ÿฐ ๋‹ค์Œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค start_http_server(ํฌํŠธ)๋‹ค๋ฅธ HTTP ํฌํŠธ์—์„œ ์ด ๋์ ์„ ์—ด๋ ค๋ฉด 5099 ์œ„์˜ ์˜ˆ์—์„œ. ๋˜๋Š” ๋™์ผํ•œ Flask ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์— ์žˆ๋Š” ์—”๋“œํฌ์ธํŠธ์— ๋งŒ์กฑํ•˜์ง€๋งŒ ํ•ด๋‹น ๊ฒฝ๋กœ๋ฅผ ๋‹ค์Œ์—์„œ ๋ณ€๊ฒฝํ•ด์•ผ ํ•˜๋Š” ๊ฒฝ์šฐ /์ธก์ •ํ•ญ๋ชฉ, ๋‹ค๋ฅธ URI๋ฅผ ๊ฒฝ๋กœ ๋งค๊ฐœ๋ณ€์ˆ˜๋กœ ์ „๋‹ฌํ•˜๊ฑฐ๋‚˜ ๋‹ค์Œ์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ ˆ์ง€์Šคํ„ฐ_์—”๋“œํฌ์ธํŠธ(..)๋‚˜์ค‘์— ์„ค์น˜ํ•˜๋ ค๋ฉด

์ฐธ์กฐ

ํ•œ๋ฒˆ ์‹œ๋„ํ•ด ๋ณด์‹œ๊ธฐ๋กœ ๊ฒฐ์ •ํ•˜์…จ๋‹ค๋ฉด GitHub์—์„œ ๋ฌธ์ œ๋ฅผ ๊ณต๊ฐœํ•˜์‹œ๊ฑฐ๋‚˜ ๋Œ“๊ธ€, ํ”ผ๋“œ๋ฐฑ, ์ œ์•ˆ์„ ๋‚จ๊ฒจ์ฃผ์„ธ์š”!

๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!

์ถœ์ฒ˜ : habr.com

์ฝ”๋ฉ˜ํŠธ๋ฅผ ์ถ”๊ฐ€