Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 46 additions & 11 deletions docs/examples/fork-process-model/README.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
Working With Fork Process Models
================================

The `BatchSpanProcessor` is not fork-safe and doesn't work well with application servers
(Gunicorn, uWSGI) which are based on the pre-fork web server model. The `BatchSpanProcessor`
spawns a thread to run in the background to export spans to the telemetry backend. During the fork, the child
process inherits the lock which is held by the parent process and deadlock occurs. We can use fork hooks to
get around this limitation of the span processor.

Please see http://bugs.python.org/issue6721 for the problems about Python locks in (multi)threaded
context with fork.
The tracing and logging batch processors reinitialize their background worker
state after a process forks, so they can be created before application servers
such as Gunicorn and uWSGI fork their workers. Metrics aggregation state is not
safe to share across forked workers, however. Applications using metrics with a
pre-fork server should create their ``MeterProvider`` and metric readers in each
child process.

The examples below initialize tracing and metrics together in a fork hook. Only
the metrics setup requires this placement, but keeping the providers together
gives every worker a consistent resource and avoids sharing metric state. An
application that emits only traces or logs can initialize its providers before
the server forks.

Please see https://bugs.python.org/issue6721 for general problems involving
Python locks in multithreaded programs that fork.

The source code for the examples with Flask app are available :scm_web:`here <docs/examples/fork-process-model/>`.

Expand All @@ -17,8 +24,11 @@ Gunicorn post_fork hook

.. code-block:: python

from opentelemetry import trace
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
Expand All @@ -37,6 +47,17 @@ Gunicorn post_fork hook
)
trace.get_tracer_provider().add_span_processor(span_processor)

metrics.set_meter_provider(
MeterProvider(
resource=resource,
metric_readers=[
PeriodicExportingMetricReader(
OTLPMetricExporter(endpoint="http://localhost:4317")
)
],
)
)


uWSGI postfork decorator
------------------------
Expand All @@ -45,15 +66,18 @@ uWSGI postfork decorator

from uwsgidecorators import postfork

from opentelemetry import trace
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


@postfork
def init_tracing():
def init_telemetry():
resource = Resource.create(attributes={
"service.name": "api-service"
})
Expand All @@ -63,3 +87,14 @@ uWSGI postfork decorator
OTLPSpanExporter(endpoint="http://localhost:4317")
)
trace.get_tracer_provider().add_span_processor(span_processor)

metrics.set_meter_provider(
MeterProvider(
resource=resource,
metric_readers=[
PeriodicExportingMetricReader(
OTLPMetricExporter(endpoint="http://localhost:4317")
)
],
)
)
17 changes: 15 additions & 2 deletions docs/examples/fork-process-model/flask-uwsgi/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,16 @@
from flask import request
from uwsgidecorators import postfork

from opentelemetry import trace
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter,
)
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
Expand All @@ -22,7 +27,7 @@


@postfork
def init_tracing():
def init_telemetry():
resource = Resource.create(attributes={"service.name": "api-service"})

trace.set_tracer_provider(TracerProvider(resource=resource))
Expand All @@ -31,6 +36,14 @@ def init_tracing():
span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
trace.get_tracer_provider().add_span_processor(span_processor)

reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint="http://localhost:4317", insecure=True))
metrics.set_meter_provider(
MeterProvider(
resource=resource,
metric_readers=[reader],
)
)


def fib_slow(n):
if n <= 1:
Expand Down