panurus

Metrics Reference

This page lists every metric Panurus registers, with the exact name Prometheus exports, so a name can be copied straight into a Prometheus query or a Grafana panel.

The list is not maintained by hand. token/services/metricsdoc instantiates every metrics constructor in the SDK the way production wiring does, reads the resulting names back out of a Prometheus registry, and compares them against token/services/metricsdoc/testdata/metrics.golden and against this page. Adding, renaming or moving a metric fails that test until this page is updated. Because a name also depends on which provider production hands the constructor, the same test pins that wiring: it asserts that the token drivers are the only place a TMS-scoped provider is built, and that the provider they get from the dependency-injection container reaches nothing but that wrapper. Dropping the wrapper, or routing the plain provider past it, therefore fails too rather than silently invalidating the names below.

See Monitoring for the wider monitoring setup and Driver Metrics for how the driver instrumentation is wired. Every metric below already has a panel in the importable Grafana overview dashboard, whose queries are checked against this same list.

How a metric name is built

Panurus code never writes the exported name. It supplies only the bare Name field — with one stated exception, Storage connection pool, which sets Namespace and Subsystem explicitly so its name does not depend on which package happens to call the constructor; see that section for why.

For every other metric:

p.NewCounter(metrics.CounterOpts{
    Name: "endorsed_transactions",
    Help: "The number of endorsed transactions.",
    LabelNames: []string{"network", "channel", "namespace"},
})

FSC’s Prometheus provider fills in the missing Namespace and Subsystem from the Go package of the caller (platform/view/services/metrics/prometheus/provider.go), then joins the three parts:

  1. The caller’s import path is split on / — the last segment becomes the subsystem, everything before it, joined with _, becomes the namespace.
  2. Registered replacers rewrite the namespace. Panurus registers github.com_LFDT-Panurus_panurus_token → panurus in token/services/logging/init.go, so every metric of this repository is prefixed panurus_.
  3. The exported name is <namespace>_<subsystem>_<name>.

So endorsed_transactions, created from token/services/ttx, is exported as:

github.com/LFDT-Panurus/panurus/token/services/ttx  →  namespace panurus_services, subsystem ttx
                                                    →  panurus_services_ttx_endorsed_transactions

Two consequences are worth internalising before writing a query or a dashboard:

The TMS-scoped provider changes the prefix

metrics.NewTMSProvider (token/core/common/metrics/provider.go) wraps a provider so that every metric it creates is bound to fixed network/channel/namespace label values. Because the wrapper calls the underlying provider on the caller’s behalf, Prometheus attributes the metric to the wrapper’s package, token/core/common/metrics, and not to the package that declared it.

Every metric created through a TMS-scoped provider is therefore exported under panurus_core_common_metrics_, whatever its source file. The clearest example is cache_level: it is declared in token/services/identity/idemix/cache/metrics.go, yet exported as panurus_core_common_metrics_cache_level.

The wiring that decides this is the driver setup in token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/driver.go, which builds a TMS-scoped provider and passes it to the driver services and the wallet service, and token/sdk/db/checks.go, which does the same for the ledger drift checks sweep. Everything reached from the dependency-injection container instead keeps its own package prefix.

Two rules follow for anyone adding a metric:

Driver services

Recorded by the decorator wrappers in token/core/common/metrics/, which time every driver service call. All are TMS-scoped, so network, channel and namespace are always present, and method carries the interface method that was invoked. See Driver Metrics for the method lists.

Metric Type Labels Description
panurus_core_common_metrics_issue_service_operations_total counter network,channel,namespace,method Total number of IssueService method invocations
panurus_core_common_metrics_issue_service_duration_seconds histogram network,channel,namespace,method Duration of IssueService method calls in seconds
panurus_core_common_metrics_issue_service_errors_total counter network,channel,namespace,method Total number of IssueService method errors
panurus_core_common_metrics_transfer_service_operations_total counter network,channel,namespace,method Total number of TransferService method invocations
panurus_core_common_metrics_transfer_service_duration_seconds histogram network,channel,namespace,method Duration of TransferService method calls in seconds
panurus_core_common_metrics_transfer_service_errors_total counter network,channel,namespace,method Total number of TransferService method errors
panurus_core_common_metrics_auditor_service_operations_total counter network,channel,namespace,method Total number of AuditorService method invocations
panurus_core_common_metrics_auditor_service_duration_seconds histogram network,channel,namespace,method Duration of AuditorService method calls in seconds
panurus_core_common_metrics_auditor_service_errors_total counter network,channel,namespace,method Total number of AuditorService method errors
panurus_core_common_metrics_tokens_service_operations_total counter network,channel,namespace,method Total number of TokensService method invocations
panurus_core_common_metrics_tokens_service_duration_seconds histogram network,channel,namespace,method Duration of TokensService method calls in seconds
panurus_core_common_metrics_tokens_service_errors_total counter network,channel,namespace,method Total number of TokensService method errors
panurus_core_common_metrics_tokens_upgrade_service_operations_total counter network,channel,namespace,method Total number of TokensUpgradeService method invocations
panurus_core_common_metrics_tokens_upgrade_service_duration_seconds histogram network,channel,namespace,method Duration of TokensUpgradeService method calls in seconds
panurus_core_common_metrics_tokens_upgrade_service_errors_total counter network,channel,namespace,method Total number of TokensUpgradeService method errors

Source: token/core/common/metrics/{issue,transfer,auditor,tokens,upgrade}.go.

Transaction lifecycle (ttx)

The ttx service orchestrates a token transaction from assembly to commit. These counters and histograms mark the phases an operator cares about; they are the closest thing the SDK has to throughput and latency of the business flow.

Metric Type Labels Description
panurus_services_ttx_endorsed_transactions counter network,channel,namespace The number of endorsed transactions.
panurus_services_ttx_audit_approved_transactions counter network,channel,namespace The number of approved transactions by the auditor.
panurus_services_ttx_accepted_transactions counter network,channel,namespace The number of accepted transactions.
panurus_services_ttx_endorsement_duration_seconds histogram network,channel,namespace Duration of the full endorsement collection phase including signatures, audit, and chaincode approval.
panurus_services_ttx_audit_approval_duration_seconds histogram network,channel,namespace Duration of the auditor approval phase including validation, append, and signing.
panurus_services_ttx_ordering_duration_seconds histogram network,channel,namespace Duration of the transaction broadcast to the ordering service.

Source: token/services/ttx/metrics.go. Wired in token/sdk/dig/sdk.go.

Finality listener

Recorded when a transaction’s ledger status resolves. hash_mismatch_total is the one to alert on: a non-zero value means a committed token-request hash did not match the locally stored one, which is a data-integrity violation rather than an ordinary failure.

Metric Type Labels Description
panurus_services_ttx_finality_finality_listener_confirmed_total counter — Total number of transactions confirmed on the ledger and successfully committed to local storage
panurus_services_ttx_finality_finality_listener_deleted_total counter — Total number of transactions marked as deleted due to an invalid ledger status or token-request hash mismatch
panurus_services_ttx_finality_finality_listener_hash_mismatch_total counter — Total number of transactions rejected because the committed token-request hash did not match the locally stored one
panurus_services_ttx_finality_finality_listener_retry_exhausted_total counter — Total number of transactions whose finality processing was abandoned after all retries were exhausted
panurus_services_ttx_finality_finality_listener_on_status_duration_seconds histogram — Histogram of total OnStatus processing time per transaction (including retries), in seconds

Source: token/services/ttx/finality/metrics.go.

Envelope sessions

Recorded by the versioned-envelope wrapper used for view-to-view messaging. version is the envelope version, type the message type, and error one of missing_version, version_mismatch, type_mismatch, invalid_envelope or unknown (see classifyError in token/services/utils/json/session/envelope.go).

Metric Type Labels Description
panurus_services_utils_json_session_ttx_envelope_sent_total counter version,type Total number of versioned envelopes sent
panurus_services_utils_json_session_ttx_envelope_received_total counter version,type Total number of versioned envelopes received
panurus_services_utils_json_session_ttx_envelope_errors_total counter error Total number of envelope validation errors
panurus_services_utils_json_session_ttx_envelope_body_bytes histogram type Size of envelope body in bytes

Source: token/services/utils/json/session/metrics.go. Wired in token/sdk/dig/sdk.go.

Auditor service

Recorded by the auditor around Audit(), Append() and Release(). These metrics carry no TMS labels, so a node auditing several TMSs reports them aggregated; see Metric hygiene.

Metric Type Labels Description
panurus_services_auditor_auditor_audit_duration_seconds histogram — Histogram of Audit() processing time per transaction (including lock acquisition), in seconds
panurus_services_auditor_auditor_audit_lock_conflicts_total counter — Total number of Audit() and Append() calls that failed to acquire enrollment-ID locks
panurus_services_auditor_auditor_append_duration_seconds histogram — Histogram of Append() processing time per transaction, in seconds
panurus_services_auditor_auditor_append_errors_total counter — Total number of Append() calls that failed to store the audit record
panurus_services_auditor_auditor_releases_total counter — Total number of Release() calls (explicit and deferred)

Source: token/services/auditor/metrics.go.

Token selection (sherdlock)

Recorded by the default token selector. outcome is one of success, insufficient_funds, locked_funds or error; fetcher_type is eager or lazy. selection_immediate_retries shows how often a selection had to retry because of concurrent lock contention. lock_conflicts_total and distinct_tokens_attempted were added for #2395 to distinguish “one hot token retried many times” from “many tokens each contended once”; lock_conflicts_total is deliberately unlabeled by token id or wallet id to avoid unbounded cardinality — per-token attribution belongs in the selector’s debug-level log line (Lost lock race on token [...], visible once the sherdlock package’s logger is at debug level) and in the tokendiag locks command. lock_store_errors_total (also #2395) counts the sibling case: a TryLock/TryLockBatch failure that is not a lock conflict (does not wrap driver.ErrTokenAlreadyLocked) — a genuine store error such as a connection failure or timeout. To the caller both currently surface identically as retried, eventually-locked-funds contention, so this counter is what distinguishes “the store is unhealthy” from “tokens are just contended” without changing that retry behavior.

stale_candidates_total (also #2395) counts the third case, which is neither: a candidate dropped because the token was no longer spendable by the time the lock was attempted (driver.ErrTokenNotSpendable). The eager fetcher serves candidates from a snapshot of the token store, so a token spent after that snapshot was taken is still offered until the cache refreshes; the lock is conditional on the token still being spendable, so such a candidate is rejected rather than handed to a caller that could not load it. A non-zero rate here therefore means the cache’s freshness interval is long relative to how fast the wallet is spending, not that tokens are contended or that the store is unhealthy.

Both lock paths report it. A batch-capable backend — Postgres, via LockBatch, currently the only implementation — claims a whole window of candidates in one statement, and that statement classifies each candidate it did not win: the ones it refused as no longer spendable come back as such, and “another claimant holds it” is the remainder. So a stale candidate on the batch path is counted here, dropped for good, and used to refresh the candidate cache, exactly as on the single-token path — which also means a selection no longer has to wait for the eager cache’s own freshnessInterval to recover from one.

A backend is still allowed to win or not win a candidate without saying why; a selector treats every unwon candidate from one of those as a lost race. On such a deployment stale_candidates_total stays at zero even during a stale-candidate episode, and the symptom to read is lock_conflicts_total rising without a corresponding rise in real contention (e.g. with distinct_tokens_attempted flat and no competing senders). Neither correctness nor recovery depends on the classification — a token the store refuses is never handed to a caller, and a scan that comes up short refreshes the candidate cache whatever the store could say about why (see cache_invalidations_total below) — the difference is purely in what is reported.

cache_invalidations_total (#2395) counts the forced refreshes those recoveries cost: each one is the selector telling its candidate source that the source’s snapshot is behind the store, so the next read of it re-queries. It is raised by a stale candidate, and by a scan that ran out of candidates before covering the request — the candidate query hides tokens locked by others, so a lock released since the snapshot was taken cannot appear by re-reading it. The invalidation itself issues no query; it turns the next read into a full refresh for the eager and mixed strategies, and several invalidations before that read collapse into one. The first fetch of a selection is exempt, so a request served by its first scan costs nothing here; per Select the count is bounded by selection_immediate_retries plus one per stale candidate. A rate near that bound on most calls means the freshness interval is long relative to how fast the wallets turn over, and the eager strategy is paying for a snapshot it cannot keep — fetcherStrategy: lazy (or a shorter fetcherCacheRefresh) is then the cheaper configuration.

candidate_permutation_duration_seconds (#2510) is the time NewPermutation spends copying and shuffling a wallet’s whole candidate list (the within-amount shuffle that spreads contention). It is observed once per permutation performed and labelled fetcher_type: eager for a cache hit (the eager strategy and the eager leg of mixed), lazy for a list just read from the store. A cache miss shuffles nothing and observes nothing. In mixed, a call is normally observed under one of the two; if the eager leg’s iterator then fails its peek, the call falls through to the lazy leg and observes under both. Read it against selection_duration_seconds: if its sum is a large share of the selection time, the shuffle (not the database or lock contention) is what a large cached wallet is paying; the cost grows with the number of candidates (see BenchmarkNewPermutation). If #2491 changes the shuffle, the histogram measures whatever NewPermutation then does.

For StubbornSelector, both selection_immediate_retries and distinct_tokens_attempted are observed once per outer Select() call, aggregated over every internal backoff-retry attempt it makes — not once per attempt. They aggregate differently: selection_immediate_retries counts events and so sums the per-attempt counts, while distinct_tokens_attempted counts distinct tokens and so unions them, meaning a token contended across several attempts is counted once.

Metric Type Labels Description
panurus_services_selector_sherdlock_unspent_tokens_invocations counter fetcher_type The number of invocations
panurus_services_selector_sherdlock_selection_duration_seconds histogram — Duration of a token selection call in seconds
panurus_services_selector_sherdlock_selection_backoff_duration_seconds histogram — Time spent in a backoff wait between token selection rounds in seconds
panurus_services_selector_sherdlock_selection_outcome_total counter outcome Total number of token selection outcomes by result type
panurus_services_selector_sherdlock_selection_immediate_retries histogram — Distribution of immediate retry counts per token selection call
panurus_services_selector_sherdlock_lock_conflicts_total counter — Total number of lost lock races (a token was already locked by another process)
panurus_services_selector_sherdlock_distinct_tokens_attempted histogram — Distribution of the number of distinct tokens a lock was attempted on (won, lost, or rate-limited) per token selection call
panurus_services_selector_sherdlock_lock_store_errors_total counter — Total number of TryLock/TryLockBatch failures that are not lock conflicts (a genuine store error)
panurus_services_selector_sherdlock_stale_candidates_total counter — Total number of candidate tokens dropped because they were no longer spendable when the lock was attempted
panurus_services_selector_sherdlock_cache_invalidations_total counter — Total number of times the candidate source was told its snapshot is behind the token store, forcing the next read to refresh it
panurus_services_selector_sherdlock_candidate_permutation_duration_seconds histogram fetcher_type Duration in seconds of copying and shuffling a wallet’s candidate token list (NewPermutation)

Backoff time. selection_duration_seconds includes the time a StubbornSelector sleeps between rounds. selection_backoff_duration_seconds records each such wait (the elapsed time, so a wait cut short by context cancellation records what was actually waited), which lets you split an elevated Select latency into backoff versus selection work. Zero-length waits are observed too. Sherdlock also waits after its final failed round, so a selection that gives up records max retries + 1 waits. The metrics are process-global, with no tms label.

Source: token/services/selector/sherdlock/metrics.go.

Token selection (simple)

Recorded by the simple selector driver. Only one selector driver is wired per process (token.selector.driver), so this section and the sherdlock one are never both exported.

Metric Type Labels Description
panurus_services_selector_simple_selection_backoff_duration_seconds histogram — Time spent in a backoff wait between token selection rounds in seconds

selection_backoff_duration_seconds records the elapsed time of each wait between selection attempts, including a wait cut short by context cancellation. There is no wait after the last attempt, and none at all when the manager’s retry timeout is non-positive; a jitter of exactly zero is still observed as 0. The metric is process-global, with no tms label.

The simple driver exports no other selection metrics, in particular no selection_duration_seconds. This histogram therefore shows how much time and how many waits backoff accounts for, but there is no total selection latency to split it against, as there is for sherdlock.

Source: token/services/selector/simple/metrics.go.

Token selection rate limiting

Recorded by the opt-in per-wallet rate limiter (token.selector.rateLimit*, or the ratelimit options), for both the sherdlock and simple selector drivers. Nothing is registered while rate limiting is off. denied_total counts selection requests rejected with token.SelectorRateLimited, which is what tells a policy-driven rejection apart from real lock contention (lock_conflicts_total, selection_outcome_total{outcome="locked_funds"}).

denied_total is labeled by tms (the TMS id the limiter was asked about) and deliberately not by wallet id, which would be unbounded cardinality; the denial error names the wallet. buckets_active is a single process-wide gauge, since one limiter serves every TMS of a selector service; it assumes one metered limiter per process (two limiters sharing a provider overwrite each other). When it sits at rateLimitMaxBuckets the limiter is evicting live wallets, which resets their allowance.

The names omit the ratelimit_ prefix the issue proposed, because the package name is already part of the Prometheus name. The tms label is an intentional exception to the network/channel/namespace convention: the limiter only knows the TMS id string. A pre-built limiter passed with ratelimit.WithLimiter is metered only if it was built with ratelimit.Config.Metrics.

Metric Type Labels Description
panurus_services_selector_ratelimit_denied_total counter tms Total number of token selection requests denied by the per-wallet rate limiter
panurus_services_selector_ratelimit_buckets_active gauge — Number of live per-wallet buckets held by the token selection rate limiter

Source: token/services/selector/ratelimit/metrics.go.

Certification (interactive)

certified_tokens is recorded by the certification service (server side); the remaining four are recorded by the certification client. Note the asymmetry in labels — the client-side metrics carry channel and namespace but not network.

Metric Type Labels Description
panurus_services_certifier_interactive_certified_tokens counter network,channel,namespace The number of tokens certified.
panurus_services_certifier_interactive_certification_request_duration_seconds histogram channel,namespace Histogram of certification batch request durations in seconds.
panurus_services_certifier_interactive_certification_errors_total counter channel,namespace Total number of failed certification request attempts.
panurus_services_certifier_interactive_certification_pending_tokens gauge channel,namespace Current number of tokens waiting in the certification input buffer.
panurus_services_certifier_interactive_certification_dropped_tokens_total counter channel,namespace Total number of tokens dropped because the certification buffer was full.

Source: token/services/certifier/interactive/metrics.go.

Identity and caches

All identity instrumentation is created through the TMS-scoped provider from the wallet service (token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/ws.go), hence the panurus_core_common_metrics_ prefix despite living under token/services/identity.

outcome and path are cache, routed or fallback. A near-zero identity_signer_router_registrations_total in a busy deployment means routing is not being populated and every signer lookup falls back to the probing deserializer. The two *_provision_failures_total counters rise when a cache cannot pre-provision in the background, so requests are served on the slow path while the identity backend keeps failing; the corresponding log line alone is easy to miss.

Metric Type Labels Description
panurus_core_common_metrics_identity_signer_resolutions_total counter network,channel,namespace,outcome Total number of GetSigner calls by outcome (cache, routed, fallback)
panurus_core_common_metrics_identity_get_signer_duration_seconds histogram network,channel,namespace,path Histogram of GetSigner wall-clock time in seconds, labeled by resolution path
panurus_core_common_metrics_identity_signer_router_registrations_total counter network,channel,namespace Total number of conf_id-to-KeyManager bindings registered with the SignerRouter
panurus_core_common_metrics_identity_signer_router_no_probe_errors_total counter network,channel,namespace Total number of errors from the SignerRouter’s probe-free signer deserialization path
panurus_core_common_metrics_cache_level gauge network,channel,namespace Level of the idemix cache
panurus_core_common_metrics_cache_provision_failures_total counter network,channel,namespace Failed attempts to pre-provision idemix identities
panurus_core_common_metrics_recipient_data_cache_level gauge network,channel,namespace Level of the wallet recipient data cache
panurus_core_common_metrics_recipient_data_provision_failures_total counter network,channel,namespace Failed attempts to pre-provision wallet recipient data

Source: token/services/identity/metrics.go, token/services/identity/idemix/cache/metrics.go, token/services/identity/role/metrics.go.

Wallet service

Recorded by the observability decorator in token/services/observability/, which wraps the wallet service and per-wallet instances (OwnerWallet, IssuerWallet, AuditorWallet, CertifierWallet) with metrics and circuit breaking. Both driver paths (FabToken and ZKAT-DLog) receive a TMS-scoped provider, hence the panurus_core_common_metrics_ prefix. All metrics carry wallet_id to prevent metric collisions across wallets.

Metric Type Labels Description
panurus_core_common_metrics_wallet_requests_total counter network,channel,namespace,method,wallet_id Total number of wallet service requests
panurus_core_common_metrics_wallet_errors_total counter network,channel,namespace,method,wallet_id Total number of wallet service errors
panurus_core_common_metrics_wallet_request_duration_seconds histogram network,channel,namespace,method,wallet_id Execution duration of wallet service operations in seconds
panurus_core_common_metrics_wallet_inflight_requests gauge network,channel,namespace,method,wallet_id Current number of in-flight wallet service requests
panurus_core_common_metrics_wallet_circuit_breaker_rejections_total counter network,channel,namespace,method,wallet_id Total number of requests rejected by the circuit breaker
panurus_core_common_metrics_wallet_circuit_breaker_state gauge network,channel,namespace,method,wallet_id Current circuit breaker state (0=Closed, 1=Open, 2=HalfOpen)

Source: token/services/observability/metrics.go. Wired in token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/driver.go.

Ledger drift checks

Recorded by the background sweep in token/services/storage/services/checks, wired up from token/sdk/db/checks.go through the TMS-scoped provider — hence the panurus_core_common_metrics_ prefix despite living under token/services/storage. role is owner or auditor, naming which store a sweep covers; outcome is completed, not_leader, canceled or failed. See Ledger Drift Checks for what the checks themselves look at.

Metric Type Labels Description
panurus_core_common_metrics_storage_checks_sweep_duration_seconds histogram network,channel,namespace,role Histogram of the wall-clock time of one ledger drift checks sweep, in seconds
panurus_core_common_metrics_storage_checks_sweeps_total counter network,channel,namespace,role,outcome Total number of ledger drift checks sweeps by outcome (completed, not_leader, canceled, failed)
panurus_core_common_metrics_storage_checks_findings_observed_total counter network,channel,namespace,role,checker,code,severity Total number of ledger drift findings reported, by check, code and severity
panurus_core_common_metrics_storage_checks_findings_open gauge network,channel,namespace,role,severity Number of currently open ledger drift findings, by severity
panurus_core_common_metrics_storage_checks_findings_resolved_total counter network,channel,namespace,role Total number of stored ledger drift findings closed because a sweep stopped reporting them

Source: token/services/storage/services/checks/metrics.go.

Fabric-X finality queue

Recorded by the Fabric-X event queue that dispatches finality notifications to workers. pending_events is the backlog gauge: it counts events accepted by the queue and not yet taken by a worker. A caller blocked in EnqueueBlocking on a full buffer is counted as soon as it starts waiting, so the gauge can exceed the configured queue size (token.finality.notification.queueSize); a value above it means producers are blocked waiting for space. enqueue_drops_total increments when the buffer was full and an event was discarded, which means a lost finality notification.

Metric Type Labels Description
panurus_services_network_fabricx_finality_queue_finality_queue_pending_events gauge — Number of finality events waiting for a worker, including EnqueueBlocking callers blocked on a full buffer, so it can exceed the queue size
panurus_services_network_fabricx_finality_queue_finality_queue_enqueue_drops_total counter — Total number of finality events dropped because the queue was full
panurus_services_network_fabricx_finality_queue_finality_queue_processing_errors_total counter — Total number of errors returned by event.Process in worker goroutines
panurus_services_network_fabricx_finality_queue_finality_queue_processing_duration_seconds histogram — Histogram of successful event processing time in worker goroutines (seconds)

Source: token/services/network/fabricx/finality/queue/metrics.go.

Storage connection pool

Recorded by a dbpool.Sampler that polls (*sql.DB).Stats() for every Postgres connection pool a token store opens, once per second. It exists because Panurus never checks out a connection itself — the pool lives inside fabric-smart-client’s postgres.Open and reaches Panurus only as a *sql.DB — so Stats() is the only available signal, and it exposes cumulative totals (WaitCount/WaitDuration), not per-wait observations. That rules out the histogram the parent epic (#2492) originally proposed; a gauge plus a counter pair is the closest honest signal instead — see #2508.

pool identifies the underlying data source as host:port/database, not the store: every store sharing a data source — including fabric-smart-client’s own kvs/binding/signerinfo/auditinfo stores, which take the same DbProvider singleton — shares one *sql.DB, so a per-store label would register several identical series for one pool, and in_use/wait_* are therefore node-wide occupancy, not occupancy attributable to Panurus alone. The label is parsed out of the data source rather than redacted from it, so a credential in a field redaction would miss (query parameters, for instance) is never read, only host, port and the database name. Postgres advisory-lock connections (opened outside the pool, e.g. by the recovery and keystore-cleanup leaders) are not reflected in these metrics, since they never reach (*sql.DB).Stats().

in_use and max_open are deliberately instantaneous samples, not a max held since the previous one. Sampling at 1s rather than something coarser is what makes a sustained exhaustion episode — such as #2474, where a pool of 10 was permanently exhausted by connections with a ~250ms hold time — land on most samples instead of being missed between them; detecting a short-lived burst does not depend on the sampling rate at all, because wait_count_total increments once per blocked checkout regardless of how often anything scrapes it. Peak occupancy is a query, not a stored value: max_over_time(panurus_storage_db_pool_in_use[$__rate_interval]). Do not raise the sampling interval (token.storage.db.poolMetricsInterval, default 1s) back toward something like 10s without re-reading this — at a slower poll the gauge can sit well below max_open throughout an episode that was, in fact, permanently exhausted between samples. 0 disables the sampler (and its goroutine) entirely, for a node that does not scrape Prometheus.

Metric Type Labels Description
panurus_storage_db_pool_in_use gauge pool Number of connections currently checked out of the pool
panurus_storage_db_pool_max_open gauge pool Configured maximum number of open connections for the pool
panurus_storage_db_pool_wait_count_total counter pool Total number of connection checkouts that had to wait because the pool was at its connection limit
panurus_storage_db_pool_wait_seconds_total counter pool Total time, in seconds, callers spent waiting for a connection to become available

Current saturation:

panurus_storage_db_pool_in_use / panurus_storage_db_pool_max_open

Peak saturation over the query window:

max_over_time(panurus_storage_db_pool_in_use[$__rate_interval]) / panurus_storage_db_pool_max_open

Mean connection wait:

rate(panurus_storage_db_pool_wait_seconds_total[$__rate_interval])
  / rate(panurus_storage_db_pool_wait_count_total[$__rate_interval])

Source: token/services/storage/db/sql/dbpool/metrics.go.

Example queries

Transaction throughput per TMS:

sum by (network, channel, namespace) (rate(panurus_services_ttx_accepted_transactions[5m]))

95th percentile of the endorsement phase:

histogram_quantile(0.95, sum by (le, network, channel, namespace) (
  rate(panurus_services_ttx_endorsement_duration_seconds_bucket[5m])
))

Share of token selections that failed for lack of spendable funds:

sum(rate(panurus_services_selector_sherdlock_selection_outcome_total{outcome!="success"}[5m]))
  / sum(rate(panurus_services_selector_sherdlock_selection_outcome_total[5m]))

Slowest driver methods, to find the expensive cryptography:

topk(5, histogram_quantile(0.99, sum by (le, method) (
  rate(panurus_core_common_metrics_transfer_service_duration_seconds_bucket[5m])
)))

Fraction of signer lookups that fall back to the probing deserializer:

sum(rate(panurus_core_common_metrics_identity_signer_resolutions_total{outcome="fallback"}[5m]))
  / sum(rate(panurus_core_common_metrics_identity_signer_resolutions_total[5m]))

Integrity and loss alerts — any increase is worth paging on:

increase(panurus_services_ttx_finality_finality_listener_hash_mismatch_total[10m]) > 0
increase(panurus_services_network_fabricx_finality_queue_finality_queue_enqueue_drops_total[10m]) > 0

Finality backlog:

panurus_services_network_fabricx_finality_queue_finality_queue_pending_events

Not covered here

Coverage gaps

The second half of this reference: what a Panurus node currently cannot tell an operator. Ordered by operator value. Each item states what is blind today, a concrete proposal, and why existing signals do not already cover it.

1. Storage layer

Connection-pool occupancy is now covered — see Storage connection pool. What remains blind: tokendb, ttxdb, auditdb, identitydb and the keystore have no per-operation instrumentation. A slow node cannot be attributed to a specific store or operation from SDK metrics alone; the driver and ttx histograms include storage time without isolating it, and the connection-pool metrics show whether the pool itself is the bottleneck but not which store or query is holding it.

Proposal: store_operations_total{store,operation,outcome} (counter) and store_operation_duration_seconds{store,operation} (histogram), recorded in the store service layer.

Why not just a database exporter: an exporter reports IO, contention and query statistics, but cannot attribute them to store="tokendb"/operation="write" without per-query parsing rules that have to be maintained against every schema change. Embedded backends such as SQLite have no exporter at all, so SDK-level counters are the only available signal there. The two are complementary — the exporter for IO and contention, these counters for application semantics.

2. Transaction failure counter

panurus_services_ttx_accepted_transactions has no failure counterpart, so a success rate cannot be computed from SDK metrics. A deployment where every transaction fails looks, in these metrics, like a deployment with no traffic.

Proposal: panurus_services_ttx_transactions_total{network,channel,namespace,phase,outcome} (counter) with phase in assemble|endorse|audit|order|commit and outcome in success|failure, from which throughput and failure rate both follow.

3. Commit-time rejections

Nothing counts transactions the ledger rejects, broken down by reason. Note that double spending can only be enforced at commit time, so this is where a double-spend attempt becomes visible — it cannot be detected earlier during validation.

Proposal: panurus_services_network_commit_rejections_total{network,channel,namespace,reason} (counter), recorded where the commit result is processed, with reason distinguishing double-spend from signature, policy and format failures.

4. Standard Fabric network path

The Fabric-X finality queue is instrumented; the standard Fabric network driver (token/services/network/fabric/) has no equivalent, and the approval path — EndorserService in token/services/network/fabric/endorsement/provider.go — is not timed. On Fabric deployments the approval round trip is therefore invisible.

Proposal: panurus_services_network_fabric_approval_duration_seconds{network,channel,namespace} (histogram) and ..._approval_errors_total{...,reason} (counter) around the endorser call.

Note that the views that drive this path are already instrumented by FSC; the gap is the endorsement round trip itself, not the view execution.

5. Auditor lock contention

panurus_services_auditor_auditor_audit_lock_conflicts_total counts failed lock acquisitions but says nothing about contention short of failure: how long locks are held, how long acquisition waits, how many holders are active.

Proposal: panurus_services_auditor_lock_wait_duration_seconds and ..._lock_hold_duration_seconds (histograms), plus ..._locks_held (gauge).

6. Identity cache effectiveness

cache_level and recipient_data_cache_level report occupancy, not effectiveness. A cache that is always full but never hit looks identical to one that serves every request.

Proposal: identity_cache_requests_total{cache,outcome} (counter) with outcome in hit|miss, alongside the existing gauges. identity_signer_resolutions_total already does this for signers; the identity and recipient-data caches deserve the same treatment.

7. Wallet resolution

Wallet lookups by identity, enrollment ID or wallet ID are not timed or counted, so a deployment with pathological wallet resolution has no signal until it shows up as end-to-end latency.

Proposal: panurus_services_identity_wallet_lookups_total{network,channel,namespace,role,outcome} (counter) and a matching duration histogram.

Metric hygiene

Smaller consistency issues, worth fixing together since each one changes exported names:

All three are breaking changes for existing dashboards and alerts, so they belong in a single deliberate change with a note in the release notes — not folded into unrelated work.

Not proposed