1. Telemetry events are only captured if they’ve a handler attached
  2. You can only attach handlers if you know the telemetry events names

This situation makes finding telemetry events emitted by your application or your dependencies a bit hard. Telemetry is really thin (as it needs to be as performant as possible), and it doesn’t allow for a lot of flexibility. Currently, the developer needs to search for events in the documentation of many libraries and try to understand how they work. As a developer, I know that, at least for me, this is not the most pleasurable experience.

Disclaimer: As of the writing of this post, there is a library that will allow libraries and applications to document and register their telemetry events in a way that they are easily findable, but it is still not widely adopted. This library is Telemetry Registry and I hope that by the time of your reading it is used everywhere.

Hacking our way in

Reading :telemetry (1.0) source code, we see the following snippet:

-spec execute(EventName, Measurements, Metadata) -> ok when
      EventName :: event_name(),
      Measurements :: event_measurements() | event_value(),
      Metadata :: event_metadata().
execute(EventName, Value, Metadata) when is_number(Value) ->
    ?LOG_WARNING("Using execute/3 with a single event value is deprecated. "
                 "Use a measurement map instead.", []),
    execute(EventName, #{value => Value}, Metadata);
execute(EventName, Measurements, Metadata) when is_map(Measurements) and is_map(Metadata) ->
    Handlers = telemetry_handler_table:list_for_event(EventName),
    ApplyFun =
        fun(#handler{id=HandlerId,
                     function=HandlerFunction,
                     config=Config}) ->
            try
                HandlerFunction(EventName, Measurements, Metadata, Config)
            catch
                ?WITH_STACKTRACE(Class, Reason, Stacktrace)
                    detach(HandlerId),
                    ?LOG_ERROR("Handler ~p has failed and has been detached. "
                               "Class=~p~nReason=~p~nStacktrace=~p~n",
                               [HandlerId, Class, Reason, Stacktrace])
            end
        end,
    lists:foreach(ApplyFun, Handlers).

This is really simple: it searches in an ets table for a list of handlers, and apply each handler in sequence. If some of the handler raises, it is detached.

We can patch this in our dependencies and set a default handler instead of searching in the ets table. I’ll set it to be :default_handler:

execute(EventName, Measurements, Metadata) when is_map(Measurements) and is_map(Metadata) ->
    Handlers = telemetry_handler_table:list_for_event([default_handler]),
    ApplyFun =
        fun(#handler{id=HandlerId,
                     function=HandlerFunction,
                     config=Config}) ->
            try
                HandlerFunction(EventName, Measurements, Metadata, Config)
            catch
                ?WITH_STACKTRACE(Class, Reason, Stacktrace)
                    detach(HandlerId),
                    ?LOG_ERROR("Handler ~p has failed and has been detached. "
                               "Class=~p~nReason=~p~nStacktrace=~p~n",
                               [HandlerId, Class, Reason, Stacktrace])
            end
        end,
    lists:foreach(ApplyFun, Handlers).

After patching, it is important to recompile telemetry:

mix deps.compile telemetry

Now we have to define this handler. My handler will register all events into an ets table, so I can easily browse with iex:

defmodule TelemetryDiscovery do
  use GenServer

  require Logger

  @table_name TelemetryDiscoveryTable

  @impl true
  def init(_opts) do
    :ets.new(@table_name, [:set, :named_table, :public, read_concurrency: true])

    {:ok, nil}
  end

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts)

  def handle_event(event_name, measurements, metadata, _) do
    # Log warning instead of erroring if ets doesn't exist yet, because
    # telemetry starts before us and may fire events before we are up

    case :ets.whereis(@table_name) do
      :undefined ->
        Logger.warn("Could not save event #{inspect(event_name)} because TelemetryDiscovery is not up yet")

      _ref ->
        insert(event_name, measurements, metadata)
    end

    :ok
  end


  def get(event_name) do
    :ets.lookup(@table_name, event_name)
  end

  def list do
    @table_name
    |> :ets.tab2list()
    |> Enum.map(&elem(&1, 0))
  end

  defp insert(event_name, measurements, metadata) do
    :ets.insert(@table_name, {event_name, %{measurements: measurements, metadata: metadata}})
  end
end

Then you add it into your application supervision tree (don’t forget to attach):

  def init(_arg) do
    :telemetry.attach("default-handler", [:default_handler], &TelemetryDiscovery.handle_event/4, [])

    children = 
        [
          {TelemetryDiscovery, []},
          {:telemetry_poller, name: :base_poller, measurements: [], period: 10_000},
          {:telemetry_poller, name: :database_measurements_poller, measurements: measurements(), period: 60_000},
          {TelemetryMetricsPrometheus, [metrics: prometheus_metrics()]}
        ]
      end

    Supervisor.init(children, strategy: :one_for_one)
  end

And just restart your application. You should now be able to call the following functions:

  • TelemetryDiscovery.list(), which returns a list of all the telemetry events emitted since the application was started.
  • TelemetryDiscovery.get(event_name), which returns measurements and metadata for the last emission of a specific event.

Try to force the situations you want to get the telemetry events you need.

Limitations

Unfortunately, this allows us only to see events that were emitted since application was started. As said previously, Telemetry Registry should solve this in the future. For now, this solution may be helpful in debugging and interactively figuring out which metrics you need.

Some events, like ecto’s init one, may be really hard to get, but you can force most of them through iex.

Happy hacking and I hope this helps you :)