# Metrics Reference

This page lists every metric Panurus registers, with the **exact name Prometheus exports**, so a name
can be copied straight into a Prometheus query or a Grafana panel.

The list is not maintained by hand. `token/services/metricsdoc` instantiates every metrics constructor
in the SDK the way production wiring does, reads the resulting names back out of a Prometheus registry,
and compares them against `token/services/metricsdoc/testdata/metrics.golden` and against this page.
Adding, renaming or moving a metric fails that test until this page is updated. Because a name also
depends on *which* provider production hands the constructor, the same test pins that wiring: it
asserts that the token drivers are the only place a TMS-scoped provider is built, and that the provider
they get from the dependency-injection container reaches nothing but that wrapper. Dropping the wrapper,
or routing the plain provider past it, therefore fails too rather than silently invalidating the names
below.

See [Monitoring](./monitoring.md) for the wider monitoring setup and [Driver Metrics](../drivers/metrics.md)
for how the driver instrumentation is wired. Every metric below already has a panel in the importable
[Grafana overview dashboard](../monitoring/grafana/README.md), whose queries are checked against this
same list.

## How a metric name is built

Panurus code never writes the exported name. It supplies only the bare `Name` field — with one
stated exception, [Storage connection pool](#storage-connection-pool), which sets `Namespace` and
`Subsystem` explicitly so its name does not depend on which package happens to call the
constructor; see that section for why.

For every other metric:

```go
p.NewCounter(metrics.CounterOpts{
    Name: "endorsed_transactions",
    Help: "The number of endorsed transactions.",
    LabelNames: []string{"network", "channel", "namespace"},
})
```

FSC's Prometheus provider fills in the missing `Namespace` and `Subsystem` from **the Go package of the
caller** (`platform/view/services/metrics/prometheus/provider.go`), then joins the three parts:

1. The caller's import path is split on `/` — the last segment becomes the **subsystem**, everything
   before it, joined with `_`, becomes the **namespace**.
2. Registered replacers rewrite the namespace. Panurus registers
   `github.com_LFDT-Panurus_panurus_token` → `panurus` in `token/services/logging/init.go`, so every
   metric of this repository is prefixed `panurus_`.
3. The exported name is `<namespace>_<subsystem>_<name>`.

So `endorsed_transactions`, created from `token/services/ttx`, is exported as:

```
github.com/LFDT-Panurus/panurus/token/services/ttx  →  namespace panurus_services, subsystem ttx
                                                    →  panurus_services_ttx_endorsed_transactions
```

Two consequences are worth internalising before writing a query or a dashboard:

- **The bare name in the source is never queryable.** A dashboard built from `Name:` fields alone
  matches nothing — this is what made the first attempt at this documentation unusable.
- **Some names stutter**, because the metric name repeats its package: hence
  `panurus_services_auditor_auditor_audit_duration_seconds` and
  `panurus_services_network_fabricx_finality_queue_finality_queue_pending_events`. They are correct as
  listed; see [Metric hygiene](#metric-hygiene) for why they are not renamed here.

### The TMS-scoped provider changes the prefix

`metrics.NewTMSProvider` (`token/core/common/metrics/provider.go`) wraps a provider so that every metric
it creates is bound to fixed `network`/`channel`/`namespace` label values. Because the wrapper calls the
underlying provider on the caller's behalf, **Prometheus attributes the metric to the wrapper's package**,
`token/core/common/metrics`, and not to the package that declared it.

Every metric created through a TMS-scoped provider is therefore exported under
`panurus_core_common_metrics_`, whatever its source file. The clearest example is `cache_level`: it is
declared in `token/services/identity/idemix/cache/metrics.go`, yet exported as
`panurus_core_common_metrics_cache_level`.

The wiring that decides this is the driver setup in
`token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/driver.go`, which builds a TMS-scoped provider and
passes it to the driver services and the wallet service, and `token/sdk/db/checks.go`, which does the
same for the ledger drift checks sweep. Everything reached from the dependency-injection container
instead keeps its own package prefix.

Two rules follow for anyone adding a metric:

- Any `CounterOpts`/`GaugeOpts`/`HistogramOpts` passed to a TMS-scoped provider **must declare
  `network`, `channel`, `namespace` in `LabelNames`**, even though the caller never supplies values for
  them. Omitting them panics at runtime with "inconsistent label cardinality" the first time the metric
  is used — not at registration, so it survives a smoke test.
- Check which provider your constructor receives before predicting the name. If in doubt, add the
  constructor to `token/services/metricsdoc` and read the name out of the golden file.

## Driver services

Recorded by the decorator wrappers in `token/core/common/metrics/`, which time every driver service
call. All are TMS-scoped, so `network`, `channel` and `namespace` are always present, and `method`
carries the interface method that was invoked. See [Driver Metrics](../drivers/metrics.md) for the
method lists.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_core_common_metrics_issue_service_operations_total` | counter | `network,channel,namespace,method` | Total number of IssueService method invocations |
| `panurus_core_common_metrics_issue_service_duration_seconds` | histogram | `network,channel,namespace,method` | Duration of IssueService method calls in seconds |
| `panurus_core_common_metrics_issue_service_errors_total` | counter | `network,channel,namespace,method` | Total number of IssueService method errors |
| `panurus_core_common_metrics_transfer_service_operations_total` | counter | `network,channel,namespace,method` | Total number of TransferService method invocations |
| `panurus_core_common_metrics_transfer_service_duration_seconds` | histogram | `network,channel,namespace,method` | Duration of TransferService method calls in seconds |
| `panurus_core_common_metrics_transfer_service_errors_total` | counter | `network,channel,namespace,method` | Total number of TransferService method errors |
| `panurus_core_common_metrics_auditor_service_operations_total` | counter | `network,channel,namespace,method` | Total number of AuditorService method invocations |
| `panurus_core_common_metrics_auditor_service_duration_seconds` | histogram | `network,channel,namespace,method` | Duration of AuditorService method calls in seconds |
| `panurus_core_common_metrics_auditor_service_errors_total` | counter | `network,channel,namespace,method` | Total number of AuditorService method errors |
| `panurus_core_common_metrics_tokens_service_operations_total` | counter | `network,channel,namespace,method` | Total number of TokensService method invocations |
| `panurus_core_common_metrics_tokens_service_duration_seconds` | histogram | `network,channel,namespace,method` | Duration of TokensService method calls in seconds |
| `panurus_core_common_metrics_tokens_service_errors_total` | counter | `network,channel,namespace,method` | Total number of TokensService method errors |
| `panurus_core_common_metrics_tokens_upgrade_service_operations_total` | counter | `network,channel,namespace,method` | Total number of TokensUpgradeService method invocations |
| `panurus_core_common_metrics_tokens_upgrade_service_duration_seconds` | histogram | `network,channel,namespace,method` | Duration of TokensUpgradeService method calls in seconds |
| `panurus_core_common_metrics_tokens_upgrade_service_errors_total` | counter | `network,channel,namespace,method` | Total number of TokensUpgradeService method errors |

Source: `token/core/common/metrics/{issue,transfer,auditor,tokens,upgrade}.go`.

## Transaction lifecycle (ttx)

The `ttx` service orchestrates a token transaction from assembly to commit. These counters and
histograms mark the phases an operator cares about; they are the closest thing the SDK has to
throughput and latency of the business flow.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_ttx_endorsed_transactions` | counter | `network,channel,namespace` | The number of endorsed transactions. |
| `panurus_services_ttx_audit_approved_transactions` | counter | `network,channel,namespace` | The number of approved transactions by the auditor. |
| `panurus_services_ttx_accepted_transactions` | counter | `network,channel,namespace` | The number of accepted transactions. |
| `panurus_services_ttx_endorsement_duration_seconds` | histogram | `network,channel,namespace` | Duration of the full endorsement collection phase including signatures, audit, and chaincode approval. |
| `panurus_services_ttx_audit_approval_duration_seconds` | histogram | `network,channel,namespace` | Duration of the auditor approval phase including validation, append, and signing. |
| `panurus_services_ttx_ordering_duration_seconds` | histogram | `network,channel,namespace` | Duration of the transaction broadcast to the ordering service. |

Source: `token/services/ttx/metrics.go`. Wired in `token/sdk/dig/sdk.go`.

## Finality listener

Recorded when a transaction's ledger status resolves. `hash_mismatch_total` is the one to alert on: a
non-zero value means a committed token-request hash did not match the locally stored one, which is a
data-integrity violation rather than an ordinary failure.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_ttx_finality_finality_listener_confirmed_total` | counter | — | Total number of transactions confirmed on the ledger and successfully committed to local storage |
| `panurus_services_ttx_finality_finality_listener_deleted_total` | counter | — | Total number of transactions marked as deleted due to an invalid ledger status or token-request hash mismatch |
| `panurus_services_ttx_finality_finality_listener_hash_mismatch_total` | counter | — | Total number of transactions rejected because the committed token-request hash did not match the locally stored one |
| `panurus_services_ttx_finality_finality_listener_retry_exhausted_total` | counter | — | Total number of transactions whose finality processing was abandoned after all retries were exhausted |
| `panurus_services_ttx_finality_finality_listener_on_status_duration_seconds` | histogram | — | Histogram of total OnStatus processing time per transaction (including retries), in seconds |

Source: `token/services/ttx/finality/metrics.go`.

## Envelope sessions

Recorded by the versioned-envelope wrapper used for view-to-view messaging. `version` is the envelope
version, `type` the message type, and `error` one of `missing_version`, `version_mismatch`,
`type_mismatch`, `invalid_envelope` or `unknown` (see `classifyError` in
`token/services/utils/json/session/envelope.go`).

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_utils_json_session_ttx_envelope_sent_total` | counter | `version,type` | Total number of versioned envelopes sent |
| `panurus_services_utils_json_session_ttx_envelope_received_total` | counter | `version,type` | Total number of versioned envelopes received |
| `panurus_services_utils_json_session_ttx_envelope_errors_total` | counter | `error` | Total number of envelope validation errors |
| `panurus_services_utils_json_session_ttx_envelope_body_bytes` | histogram | `type` | Size of envelope body in bytes |

Source: `token/services/utils/json/session/metrics.go`. Wired in `token/sdk/dig/sdk.go`.

## Auditor service

Recorded by the auditor around `Audit()`, `Append()` and `Release()`. These metrics carry **no TMS
labels**, so a node auditing several TMSs reports them aggregated; see
[Metric hygiene](#metric-hygiene).

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_auditor_auditor_audit_duration_seconds` | histogram | — | Histogram of Audit() processing time per transaction (including lock acquisition), in seconds |
| `panurus_services_auditor_auditor_audit_lock_conflicts_total` | counter | — | Total number of Audit() and Append() calls that failed to acquire enrollment-ID locks |
| `panurus_services_auditor_auditor_append_duration_seconds` | histogram | — | Histogram of Append() processing time per transaction, in seconds |
| `panurus_services_auditor_auditor_append_errors_total` | counter | — | Total number of Append() calls that failed to store the audit record |
| `panurus_services_auditor_auditor_releases_total` | counter | — | Total number of Release() calls (explicit and deferred) |

Source: `token/services/auditor/metrics.go`.

## Token selection (sherdlock)

Recorded by the default token selector. `outcome` is one of `success`, `insufficient_funds`,
`locked_funds` or `error`; `fetcher_type` is `eager` or `lazy`. `selection_immediate_retries` shows how
often a selection had to retry because of concurrent lock contention. `lock_conflicts_total` and
`distinct_tokens_attempted` were added for [#2395](https://github.com/LFDT-Panurus/panurus/issues/2395)
to distinguish "one hot token retried many times" from "many tokens each contended once";
`lock_conflicts_total` is deliberately unlabeled by token id or wallet id to avoid unbounded
cardinality — per-token attribution belongs in the selector's debug-level log line (`Lost lock
race on token [...]`, visible once the sherdlock package's logger is at debug level) and in the
[`tokendiag locks`](../../cmd/tokendiag/README.md) command. `lock_store_errors_total` (also
#2395) counts the sibling case: a TryLock/TryLockBatch failure that is *not* a lock conflict (does
not wrap `driver.ErrTokenAlreadyLocked`) — a genuine store error such as a connection failure or
timeout. To the caller both currently surface identically as retried, eventually-locked-funds
contention, so this counter is what distinguishes "the store is unhealthy" from "tokens are just
contended" without changing that retry behavior.

`stale_candidates_total` (also #2395) counts the third case, which is neither: a candidate
dropped because the token was no longer spendable by the time the lock was attempted
(`driver.ErrTokenNotSpendable`). The eager fetcher serves candidates from a snapshot of the
token store, so a token spent after that snapshot was taken is still offered until the cache
refreshes; the lock is conditional on the token still being spendable, so such a candidate is
rejected rather than handed to a caller that could not load it. A non-zero rate here therefore
means the cache's freshness interval is long relative to how fast the wallet is spending, not
that tokens are contended or that the store is unhealthy.

**Both lock paths report it.** A batch-capable backend — Postgres, via `LockBatch`, currently
the only implementation — claims a whole window of candidates in one statement, and that
statement classifies each candidate it did not win: the ones it refused as no longer spendable
come back as such, and "another claimant holds it" is the remainder. So a stale candidate on the
batch path is counted here, dropped for good, and used to refresh the candidate cache, exactly
as on the single-token path — which also means a selection no longer has to wait for the eager
cache's own `freshnessInterval` to recover from one.

A backend is still allowed to win or not win a candidate without saying why; a selector treats
every unwon candidate from one of those as a lost race. On such a deployment
`stale_candidates_total` stays at zero *even during a stale-candidate episode*, and the symptom
to read is `lock_conflicts_total` rising without a corresponding rise in real contention (e.g.
with `distinct_tokens_attempted` flat and no competing senders). Neither correctness nor
recovery depends on the classification — a token the store refuses is never handed to a caller,
and a scan that comes up short refreshes the candidate cache whatever the store could say about
why (see `cache_invalidations_total` below) — the difference is purely in what is reported.

`cache_invalidations_total` (#2395) counts the forced refreshes those recoveries cost: each one
is the selector telling its candidate source that the source's snapshot is behind the store, so
the next read of it re-queries. It is raised by a stale candidate, and by a scan that ran out of
candidates before covering the request — the candidate query hides tokens locked by others, so a
lock released since the snapshot was taken cannot appear by re-reading it. The invalidation
itself issues no query; it turns the *next* read into a full refresh for the eager and mixed
strategies, and several invalidations before that read collapse into one. The first fetch of a
selection is exempt, so a request served by its first scan costs nothing here; per `Select` the
count is bounded by `selection_immediate_retries` plus one per stale candidate. A rate near that
bound on most calls means the freshness interval is long relative to how fast the wallets turn
over, and the eager strategy is paying for a snapshot it cannot keep — `fetcherStrategy: lazy`
(or a shorter `fetcherCacheRefresh`) is then the cheaper configuration.

`candidate_permutation_duration_seconds` (#2510) is the time `NewPermutation` spends copying and
shuffling a wallet's whole candidate list (the within-amount shuffle that spreads contention). It
is observed once per permutation performed and labelled `fetcher_type`: `eager` for a cache hit
(the eager strategy and the eager leg of mixed), `lazy` for a list just read from the store. A cache
miss shuffles nothing and observes nothing. In mixed, a call is normally observed under one of the
two; if the eager leg's iterator then fails its peek, the call falls through to the lazy leg and
observes under both. Read it against `selection_duration_seconds`: if its sum is a large share of the selection
time, the shuffle (not the database or lock contention) is what a large cached wallet is paying; the
cost grows with the number of candidates (see `BenchmarkNewPermutation`). If #2491 changes the
shuffle, the histogram measures whatever `NewPermutation` then does.

For `StubbornSelector`, both `selection_immediate_retries` and `distinct_tokens_attempted` are
observed once per outer `Select()` call, aggregated over every internal backoff-retry attempt it
makes — not once per attempt. They aggregate differently: `selection_immediate_retries` counts
events and so sums the per-attempt counts, while `distinct_tokens_attempted` counts distinct
tokens and so unions them, meaning a token contended across several attempts is counted once.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_selector_sherdlock_unspent_tokens_invocations` | counter | `fetcher_type` | The number of invocations |
| `panurus_services_selector_sherdlock_selection_duration_seconds` | histogram | — | Duration of a token selection call in seconds |
| `panurus_services_selector_sherdlock_selection_backoff_duration_seconds` | histogram | — | Time spent in a backoff wait between token selection rounds in seconds |
| `panurus_services_selector_sherdlock_selection_outcome_total` | counter | `outcome` | Total number of token selection outcomes by result type |
| `panurus_services_selector_sherdlock_selection_immediate_retries` | histogram | — | Distribution of immediate retry counts per token selection call |
| `panurus_services_selector_sherdlock_lock_conflicts_total` | counter | — | Total number of lost lock races (a token was already locked by another process) |
| `panurus_services_selector_sherdlock_distinct_tokens_attempted` | histogram | — | Distribution of the number of distinct tokens a lock was attempted on (won, lost, or rate-limited) per token selection call |
| `panurus_services_selector_sherdlock_lock_store_errors_total` | counter | — | Total number of TryLock/TryLockBatch failures that are not lock conflicts (a genuine store error) |
| `panurus_services_selector_sherdlock_stale_candidates_total` | counter | — | Total number of candidate tokens dropped because they were no longer spendable when the lock was attempted |
| `panurus_services_selector_sherdlock_cache_invalidations_total` | counter | — | Total number of times the candidate source was told its snapshot is behind the token store, forcing the next read to refresh it |
| `panurus_services_selector_sherdlock_candidate_permutation_duration_seconds` | histogram | `fetcher_type` | Duration in seconds of copying and shuffling a wallet's candidate token list (NewPermutation) |

**Backoff time.** `selection_duration_seconds` includes the time a `StubbornSelector` sleeps between
rounds. `selection_backoff_duration_seconds` records each such wait (the elapsed time, so a wait cut
short by context cancellation records what was actually waited), which lets you split an elevated
`Select` latency into backoff versus selection work. Zero-length waits are observed too. Sherdlock
also waits after its final failed round, so a selection that gives up records max retries + 1 waits.
The metrics are process-global, with no `tms` label.

Source: `token/services/selector/sherdlock/metrics.go`.

## Token selection (simple)

Recorded by the `simple` selector driver. Only one selector driver is wired per process
(`token.selector.driver`), so this section and the sherdlock one are never both exported.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_selector_simple_selection_backoff_duration_seconds` | histogram | — | Time spent in a backoff wait between token selection rounds in seconds |

`selection_backoff_duration_seconds` records the elapsed time of each wait between selection
attempts, including a wait cut short by context cancellation. There is no wait after the last
attempt, and none at all when the manager's retry timeout is non-positive; a jitter of exactly zero
is still observed as 0. The metric is process-global, with no `tms` label.

The simple driver exports no other selection metrics, in particular no `selection_duration_seconds`.
This histogram therefore shows how much time and how many waits backoff accounts for, but there is no
total selection latency to split it against, as there is for sherdlock.

Source: `token/services/selector/simple/metrics.go`.

## Token selection rate limiting

Recorded by the opt-in per-wallet rate limiter (`token.selector.rateLimit*`, or the `ratelimit`
options), for both the `sherdlock` and `simple` selector drivers. Nothing is registered while rate
limiting is off. `denied_total` counts selection requests rejected with `token.SelectorRateLimited`,
which is what tells a policy-driven rejection apart from real lock contention
(`lock_conflicts_total`, `selection_outcome_total{outcome="locked_funds"}`).

`denied_total` is labeled by `tms` (the TMS id the limiter was asked about) and deliberately not by
wallet id, which would be unbounded cardinality; the denial error names the wallet. `buckets_active`
is a single process-wide gauge, since one limiter serves every TMS of a selector service; it assumes one metered limiter per process (two limiters sharing a provider overwrite each other). When it sits
at `rateLimitMaxBuckets` the limiter is evicting live wallets, which resets their allowance.

The names omit the `ratelimit_` prefix the issue proposed, because the package name is already part
of the Prometheus name. The `tms` label is an intentional exception to the `network`/`channel`/`namespace`
convention: the limiter only knows the TMS id string. A pre-built limiter passed with
`ratelimit.WithLimiter` is metered only if it was built with `ratelimit.Config.Metrics`.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_selector_ratelimit_denied_total` | counter | `tms` | Total number of token selection requests denied by the per-wallet rate limiter |
| `panurus_services_selector_ratelimit_buckets_active` | gauge | — | Number of live per-wallet buckets held by the token selection rate limiter |

Source: `token/services/selector/ratelimit/metrics.go`.

## Certification (interactive)

`certified_tokens` is recorded by the certification service (server side); the remaining four are
recorded by the certification client. Note the asymmetry in labels — the client-side metrics carry
`channel` and `namespace` but not `network`.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_certifier_interactive_certified_tokens` | counter | `network,channel,namespace` | The number of tokens certified. |
| `panurus_services_certifier_interactive_certification_request_duration_seconds` | histogram | `channel,namespace` | Histogram of certification batch request durations in seconds. |
| `panurus_services_certifier_interactive_certification_errors_total` | counter | `channel,namespace` | Total number of failed certification request attempts. |
| `panurus_services_certifier_interactive_certification_pending_tokens` | gauge | `channel,namespace` | Current number of tokens waiting in the certification input buffer. |
| `panurus_services_certifier_interactive_certification_dropped_tokens_total` | counter | `channel,namespace` | Total number of tokens dropped because the certification buffer was full. |

Source: `token/services/certifier/interactive/metrics.go`.

## Identity and caches

All identity instrumentation is created through the TMS-scoped provider from the wallet service
(`token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/ws.go`), hence the
`panurus_core_common_metrics_` prefix despite living under `token/services/identity`.

`outcome` and `path` are `cache`, `routed` or `fallback`. A near-zero
`identity_signer_router_registrations_total` in a busy deployment means routing is not being populated
and every signer lookup falls back to the probing deserializer. The two `*_provision_failures_total`
counters rise when a cache cannot pre-provision in the background, so requests are served on the slow
path while the identity backend keeps failing; the corresponding log line alone is easy to miss.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_core_common_metrics_identity_signer_resolutions_total` | counter | `network,channel,namespace,outcome` | Total number of GetSigner calls by outcome (cache, routed, fallback) |
| `panurus_core_common_metrics_identity_get_signer_duration_seconds` | histogram | `network,channel,namespace,path` | Histogram of GetSigner wall-clock time in seconds, labeled by resolution path |
| `panurus_core_common_metrics_identity_signer_router_registrations_total` | counter | `network,channel,namespace` | Total number of conf_id-to-KeyManager bindings registered with the SignerRouter |
| `panurus_core_common_metrics_identity_signer_router_no_probe_errors_total` | counter | `network,channel,namespace` | Total number of errors from the SignerRouter's probe-free signer deserialization path |
| `panurus_core_common_metrics_cache_level` | gauge | `network,channel,namespace` | Level of the idemix cache |
| `panurus_core_common_metrics_cache_provision_failures_total` | counter | `network,channel,namespace` | Failed attempts to pre-provision idemix identities |
| `panurus_core_common_metrics_recipient_data_cache_level` | gauge | `network,channel,namespace` | Level of the wallet recipient data cache |
| `panurus_core_common_metrics_recipient_data_provision_failures_total` | counter | `network,channel,namespace` | Failed attempts to pre-provision wallet recipient data |

Source: `token/services/identity/metrics.go`, `token/services/identity/idemix/cache/metrics.go`,
`token/services/identity/role/metrics.go`.

## Wallet service

Recorded by the observability decorator in `token/services/observability/`, which wraps the wallet
service and per-wallet instances (`OwnerWallet`, `IssuerWallet`, `AuditorWallet`, `CertifierWallet`)
with metrics and circuit breaking. Both driver paths (FabToken and ZKAT-DLog) receive a TMS-scoped
provider, hence the `panurus_core_common_metrics_` prefix. All metrics carry `wallet_id` to prevent
metric collisions across wallets.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_core_common_metrics_wallet_requests_total` | counter | `network,channel,namespace,method,wallet_id` | Total number of wallet service requests |
| `panurus_core_common_metrics_wallet_errors_total` | counter | `network,channel,namespace,method,wallet_id` | Total number of wallet service errors |
| `panurus_core_common_metrics_wallet_request_duration_seconds` | histogram | `network,channel,namespace,method,wallet_id` | Execution duration of wallet service operations in seconds |
| `panurus_core_common_metrics_wallet_inflight_requests` | gauge | `network,channel,namespace,method,wallet_id` | Current number of in-flight wallet service requests |
| `panurus_core_common_metrics_wallet_circuit_breaker_rejections_total` | counter | `network,channel,namespace,method,wallet_id` | Total number of requests rejected by the circuit breaker |
| `panurus_core_common_metrics_wallet_circuit_breaker_state` | gauge | `network,channel,namespace,method,wallet_id` | Current circuit breaker state (0=Closed, 1=Open, 2=HalfOpen) |

Source: `token/services/observability/metrics.go`. Wired in `token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/driver.go`.

## Ledger drift checks

Recorded by the background sweep in `token/services/storage/services/checks`, wired up from
`token/sdk/db/checks.go` through the TMS-scoped provider — hence the `panurus_core_common_metrics_`
prefix despite living under `token/services/storage`. `role` is `owner` or `auditor`, naming which
store a sweep covers; `outcome` is `completed`, `not_leader`, `canceled` or `failed`. See
[Ledger Drift Checks](../services/storage/checks.md) for what the checks themselves look at.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_core_common_metrics_storage_checks_sweep_duration_seconds` | histogram | `network,channel,namespace,role` | Histogram of the wall-clock time of one ledger drift checks sweep, in seconds |
| `panurus_core_common_metrics_storage_checks_sweeps_total` | counter | `network,channel,namespace,role,outcome` | Total number of ledger drift checks sweeps by outcome (completed, not_leader, canceled, failed) |
| `panurus_core_common_metrics_storage_checks_findings_observed_total` | counter | `network,channel,namespace,role,checker,code,severity` | Total number of ledger drift findings reported, by check, code and severity |
| `panurus_core_common_metrics_storage_checks_findings_open` | gauge | `network,channel,namespace,role,severity` | Number of currently open ledger drift findings, by severity |
| `panurus_core_common_metrics_storage_checks_findings_resolved_total` | counter | `network,channel,namespace,role` | Total number of stored ledger drift findings closed because a sweep stopped reporting them |

Source: `token/services/storage/services/checks/metrics.go`.

## Fabric-X finality queue

Recorded by the Fabric-X event queue that dispatches finality notifications to workers.
`pending_events` is the backlog gauge: it counts events accepted by the queue and not yet taken by a
worker. A caller blocked in `EnqueueBlocking` on a full buffer is counted as soon as it starts waiting, so
the gauge can exceed the configured queue size (`token.finality.notification.queueSize`); a value above it
means producers are blocked waiting for space. `enqueue_drops_total` increments when the buffer was full
and an event was discarded, which means a lost finality notification.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_services_network_fabricx_finality_queue_finality_queue_pending_events` | gauge | — | Number of finality events waiting for a worker, including EnqueueBlocking callers blocked on a full buffer, so it can exceed the queue size |
| `panurus_services_network_fabricx_finality_queue_finality_queue_enqueue_drops_total` | counter | — | Total number of finality events dropped because the queue was full |
| `panurus_services_network_fabricx_finality_queue_finality_queue_processing_errors_total` | counter | — | Total number of errors returned by event.Process in worker goroutines |
| `panurus_services_network_fabricx_finality_queue_finality_queue_processing_duration_seconds` | histogram | — | Histogram of successful event processing time in worker goroutines (seconds) |

Source: `token/services/network/fabricx/finality/queue/metrics.go`.

## Storage connection pool

Recorded by a `dbpool.Sampler` that polls `(*sql.DB).Stats()` for every Postgres connection pool a
token store opens, once per second. It exists because Panurus never checks out a connection
itself — the pool lives inside fabric-smart-client's `postgres.Open` and reaches Panurus only as a
`*sql.DB` — so `Stats()` is the only available signal, and it exposes cumulative totals
(`WaitCount`/`WaitDuration`), not per-wait observations. That rules out the histogram the parent
epic ([#2492](https://github.com/LFDT-Panurus/panurus/issues/2492)) originally proposed; a gauge
plus a counter pair is the closest honest signal instead — see
[#2508](https://github.com/LFDT-Panurus/panurus/issues/2508).

`pool` identifies the underlying data source as `host:port/database`, not the store: every store
sharing a data source — including fabric-smart-client's own `kvs`/`binding`/`signerinfo`/`auditinfo`
stores, which take the same `DbProvider` singleton — shares one `*sql.DB`, so a per-store label
would register several identical series for one pool, and `in_use`/`wait_*` are therefore node-wide
occupancy, not occupancy attributable to Panurus alone. The label is parsed out of the data source
rather than redacted from it, so a credential in a field redaction would miss (query parameters,
for instance) is never read, only `host`, `port` and the database name. Postgres advisory-lock
connections (opened outside the pool, e.g. by the recovery and keystore-cleanup leaders) are not
reflected in these metrics, since they never reach `(*sql.DB).Stats()`.

`in_use` and `max_open` are deliberately **instantaneous** samples, not a max held since the
previous one. Sampling at 1s rather than something coarser is what makes a *sustained* exhaustion
episode — such as [#2474](https://github.com/LFDT-Panurus/panurus/issues/2474), where a pool of 10
was permanently exhausted by connections with a ~250ms hold time — land on most samples instead of
being missed between them; detecting a short-lived burst does not depend on the sampling rate at
all, because `wait_count_total` increments once per blocked checkout regardless of how often
anything scrapes it. Peak occupancy is a query, not a stored value:
`max_over_time(panurus_storage_db_pool_in_use[$__rate_interval])`. Do not raise the sampling
interval (`token.storage.db.poolMetricsInterval`, default `1s`) back toward something like 10s
without re-reading this — at a slower poll the gauge can sit well below `max_open` throughout an
episode that was, in fact, permanently exhausted between samples. `0` disables the sampler (and its
goroutine) entirely, for a node that does not scrape Prometheus.

| Metric | Type | Labels | Description |
|---|---|---|---|
| `panurus_storage_db_pool_in_use` | gauge | `pool` | Number of connections currently checked out of the pool |
| `panurus_storage_db_pool_max_open` | gauge | `pool` | Configured maximum number of open connections for the pool |
| `panurus_storage_db_pool_wait_count_total` | counter | `pool` | Total number of connection checkouts that had to wait because the pool was at its connection limit |
| `panurus_storage_db_pool_wait_seconds_total` | counter | `pool` | Total time, in seconds, callers spent waiting for a connection to become available |

Current saturation:

```promql
panurus_storage_db_pool_in_use / panurus_storage_db_pool_max_open
```

Peak saturation over the query window:

```promql
max_over_time(panurus_storage_db_pool_in_use[$__rate_interval]) / panurus_storage_db_pool_max_open
```

Mean connection wait:

```promql
rate(panurus_storage_db_pool_wait_seconds_total[$__rate_interval])
  / rate(panurus_storage_db_pool_wait_count_total[$__rate_interval])
```

Source: `token/services/storage/db/sql/dbpool/metrics.go`.

## Example queries

Transaction throughput per TMS:

```promql
sum by (network, channel, namespace) (rate(panurus_services_ttx_accepted_transactions[5m]))
```

95th percentile of the endorsement phase:

```promql
histogram_quantile(0.95, sum by (le, network, channel, namespace) (
  rate(panurus_services_ttx_endorsement_duration_seconds_bucket[5m])
))
```

Share of token selections that failed for lack of spendable funds:

```promql
sum(rate(panurus_services_selector_sherdlock_selection_outcome_total{outcome!="success"}[5m]))
  / sum(rate(panurus_services_selector_sherdlock_selection_outcome_total[5m]))
```

Slowest driver methods, to find the expensive cryptography:

```promql
topk(5, histogram_quantile(0.99, sum by (le, method) (
  rate(panurus_core_common_metrics_transfer_service_duration_seconds_bucket[5m])
)))
```

Fraction of signer lookups that fall back to the probing deserializer:

```promql
sum(rate(panurus_core_common_metrics_identity_signer_resolutions_total{outcome="fallback"}[5m]))
  / sum(rate(panurus_core_common_metrics_identity_signer_resolutions_total[5m]))
```

Integrity and loss alerts — any increase is worth paging on:

```promql
increase(panurus_services_ttx_finality_finality_listener_hash_mismatch_total[10m]) > 0
increase(panurus_services_network_fabricx_finality_queue_finality_queue_enqueue_drops_total[10m]) > 0
```

Finality backlog:

```promql
panurus_services_network_fabricx_finality_queue_finality_queue_pending_events
```

## Not covered here

- **FSC platform metrics** — view executions, sessions, gRPC, database and process metrics come from
  Fabric Smart Client and are documented in its [Monitoring](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring.md)
  page and [Metrics Catalog](https://github.com/hyperledger-labs/fabric-smart-client/blob/main/docs/platform/view/services/monitoring_metrics.md).
  View-level timing in particular is already instrumented by FSC, so the SDK does not duplicate it.
- **Load-generator metrics** — `integration/nwo/txgen/service/metrics/` and `integration/nwo/runner/`
  instrument the test harness, not the SDK, and are not exported by a Panurus node.
- **Traces** — spans are a separate mechanism; see [Monitoring](./monitoring.md).

## Coverage gaps

The second half of this reference: what a Panurus node currently cannot tell an operator. Ordered by
operator value. Each item states what is blind today, a concrete proposal, and why existing signals do
not already cover it.

### 1. Storage layer

Connection-pool occupancy is now covered — see [Storage connection pool](#storage-connection-pool).
What remains blind: `tokendb`, `ttxdb`, `auditdb`, `identitydb` and the keystore have **no
per-operation instrumentation**. A slow node cannot be attributed to a specific store or operation
from SDK metrics alone; the driver and ttx histograms include storage time without isolating it,
and the connection-pool metrics show whether the pool itself is the bottleneck but not which store
or query is holding it.

Proposal: `store_operations_total{store,operation,outcome}` (counter) and
`store_operation_duration_seconds{store,operation}` (histogram), recorded in the store service layer.

Why not just a database exporter: an exporter reports IO, contention and query statistics, but cannot
attribute them to `store="tokendb"`/`operation="write"` without per-query parsing rules that have to be
maintained against every schema change. Embedded backends such as SQLite have no exporter at all, so
SDK-level counters are the only available signal there. The two are complementary — the exporter for IO
and contention, these counters for application semantics.

### 2. Transaction failure counter

`panurus_services_ttx_accepted_transactions` has no failure counterpart, so a **success rate cannot be
computed from SDK metrics**. A deployment where every transaction fails looks, in these metrics, like a
deployment with no traffic.

Proposal: `panurus_services_ttx_transactions_total{network,channel,namespace,phase,outcome}` (counter)
with `phase` in `assemble|endorse|audit|order|commit` and `outcome` in `success|failure`, from which
throughput and failure rate both follow.

### 3. Commit-time rejections

Nothing counts transactions the ledger rejects, broken down by reason. Note that **double spending can
only be enforced at commit time**, so this is where a double-spend attempt becomes visible — it cannot
be detected earlier during validation.

Proposal: `panurus_services_network_commit_rejections_total{network,channel,namespace,reason}` (counter),
recorded where the commit result is processed, with `reason` distinguishing double-spend from signature,
policy and format failures.

### 4. Standard Fabric network path

The Fabric-X finality queue is instrumented; the standard Fabric network driver
(`token/services/network/fabric/`) has no equivalent, and the approval path — `EndorserService` in
`token/services/network/fabric/endorsement/provider.go` — is not timed. On Fabric deployments the
approval round trip is therefore invisible.

Proposal: `panurus_services_network_fabric_approval_duration_seconds{network,channel,namespace}`
(histogram) and `..._approval_errors_total{...,reason}` (counter) around the endorser call.

Note that the *views* that drive this path are already instrumented by FSC; the gap is the endorsement
round trip itself, not the view execution.

### 5. Auditor lock contention

`panurus_services_auditor_auditor_audit_lock_conflicts_total` counts failed lock acquisitions but says
nothing about contention short of failure: how long locks are held, how long acquisition waits, how many
holders are active.

Proposal: `panurus_services_auditor_lock_wait_duration_seconds` and `..._lock_hold_duration_seconds`
(histograms), plus `..._locks_held` (gauge).

### 6. Identity cache effectiveness

`cache_level` and `recipient_data_cache_level` report occupancy, not effectiveness. A cache that is
always full but never hit looks identical to one that serves every request.

Proposal: `identity_cache_requests_total{cache,outcome}` (counter) with `outcome` in `hit|miss`,
alongside the existing gauges. `identity_signer_resolutions_total` already does this for signers; the
identity and recipient-data caches deserve the same treatment.

### 7. Wallet resolution

Wallet lookups by identity, enrollment ID or wallet ID are not timed or counted, so a deployment with
pathological wallet resolution has no signal until it shows up as end-to-end latency.

Proposal: `panurus_services_identity_wallet_lookups_total{network,channel,namespace,role,outcome}`
(counter) and a matching duration histogram.

### Metric hygiene

Smaller consistency issues, worth fixing together since each one changes exported names:

- **Package attribution.** Metrics created through the TMS-scoped provider are exported under
  `panurus_core_common_metrics_` instead of their own package, which puts identity and cache metrics in
  a namespace that has nothing to do with them. Setting `Namespace`/`Subsystem` explicitly in the opts,
  or making the wrapper preserve the caller's package, would fix it. [Storage connection
  pool](#storage-connection-pool) already does this for its own metrics — the first family in the repo
  to do so — precisely because it is a new metric with no existing dashboard or alert to break by
  choosing a stable, explicit name up front.
- **Stuttering names.** `panurus_services_auditor_auditor_*`,
  `panurus_services_ttx_finality_finality_listener_*` and
  `panurus_services_network_fabricx_finality_queue_finality_queue_*` repeat their package in the metric
  name.
- **Missing TMS labels.** The auditor, finality-listener, sherdlock and Fabric-X queue families carry no
  `network`/`channel`/`namespace` labels, so on a node hosting several TMSs their values are aggregated
  and cannot be attributed. The certification client metrics carry `channel` and `namespace` but not
  `network`.

All three are breaking changes for existing dashboards and alerts, so they belong in a single deliberate
change with a note in the release notes — not folded into unrelated work.

### Not proposed

- **Re-instrumenting views.** FSC already records view executions; adding SDK-level equivalents would
  duplicate existing series.
- **Validation-time double-spend detection.** Not a metric gap but a protocol property: the check
  happens at commit, so item 3 is where it belongs.
