This page lists every metric Panurus registers, with the exact name Prometheus exports, so a name can be copied straight into a Prometheus query or a Grafana panel.
The list is not maintained by hand. token/services/metricsdoc instantiates every metrics constructor
in the SDK the way production wiring does, reads the resulting names back out of a Prometheus registry,
and compares them against token/services/metricsdoc/testdata/metrics.golden and against this page.
Adding, renaming or moving a metric fails that test until this page is updated. Because a name also
depends on which provider production hands the constructor, the same test pins that wiring: it
asserts that the token drivers are the only place a TMS-scoped provider is built, and that the provider
they get from the dependency-injection container reaches nothing but that wrapper. Dropping the wrapper,
or routing the plain provider past it, therefore fails too rather than silently invalidating the names
below.
See Monitoring for the wider monitoring setup and Driver Metrics for how the driver instrumentation is wired. Every metric below already has a panel in the importable Grafana overview dashboard, whose queries are checked against this same list.
Panurus code never writes the exported name. It supplies only the bare Name field — with one
stated exception, Storage connection pool, which sets Namespace and
Subsystem explicitly so its name does not depend on which package happens to call the
constructor; see that section for why.
For every other metric:
p.NewCounter(metrics.CounterOpts{
Name: "endorsed_transactions",
Help: "The number of endorsed transactions.",
LabelNames: []string{"network", "channel", "namespace"},
})
FSC’s Prometheus provider fills in the missing Namespace and Subsystem from the Go package of the
caller (platform/view/services/metrics/prometheus/provider.go), then joins the three parts:
/ — the last segment becomes the subsystem, everything
before it, joined with _, becomes the namespace.github.com_LFDT-Panurus_panurus_token → panurus in token/services/logging/init.go, so every
metric of this repository is prefixed panurus_.<namespace>_<subsystem>_<name>.So endorsed_transactions, created from token/services/ttx, is exported as:
github.com/LFDT-Panurus/panurus/token/services/ttx → namespace panurus_services, subsystem ttx
→ panurus_services_ttx_endorsed_transactions
Two consequences are worth internalising before writing a query or a dashboard:
Name: fields alone
matches nothing — this is what made the first attempt at this documentation unusable.panurus_services_auditor_auditor_audit_duration_seconds and
panurus_services_network_fabricx_finality_queue_finality_queue_pending_events. They are correct as
listed; see Metric hygiene for why they are not renamed here.metrics.NewTMSProvider (token/core/common/metrics/provider.go) wraps a provider so that every metric
it creates is bound to fixed network/channel/namespace label values. Because the wrapper calls the
underlying provider on the caller’s behalf, Prometheus attributes the metric to the wrapper’s package,
token/core/common/metrics, and not to the package that declared it.
Every metric created through a TMS-scoped provider is therefore exported under
panurus_core_common_metrics_, whatever its source file. The clearest example is cache_level: it is
declared in token/services/identity/idemix/cache/metrics.go, yet exported as
panurus_core_common_metrics_cache_level.
The wiring that decides this is the driver setup in
token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/driver.go, which builds a TMS-scoped provider and
passes it to the driver services and the wallet service, and token/sdk/db/checks.go, which does the
same for the ledger drift checks sweep. Everything reached from the dependency-injection container
instead keeps its own package prefix.
Two rules follow for anyone adding a metric:
CounterOpts/GaugeOpts/HistogramOpts passed to a TMS-scoped provider must declare
network, channel, namespace in LabelNames, even though the caller never supplies values for
them. Omitting them panics at runtime with “inconsistent label cardinality” the first time the metric
is used — not at registration, so it survives a smoke test.token/services/metricsdoc and read the name out of the golden file.Recorded by the decorator wrappers in token/core/common/metrics/, which time every driver service
call. All are TMS-scoped, so network, channel and namespace are always present, and method
carries the interface method that was invoked. See Driver Metrics for the
method lists.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_core_common_metrics_issue_service_operations_total |
counter | network,channel,namespace,method |
Total number of IssueService method invocations |
panurus_core_common_metrics_issue_service_duration_seconds |
histogram | network,channel,namespace,method |
Duration of IssueService method calls in seconds |
panurus_core_common_metrics_issue_service_errors_total |
counter | network,channel,namespace,method |
Total number of IssueService method errors |
panurus_core_common_metrics_transfer_service_operations_total |
counter | network,channel,namespace,method |
Total number of TransferService method invocations |
panurus_core_common_metrics_transfer_service_duration_seconds |
histogram | network,channel,namespace,method |
Duration of TransferService method calls in seconds |
panurus_core_common_metrics_transfer_service_errors_total |
counter | network,channel,namespace,method |
Total number of TransferService method errors |
panurus_core_common_metrics_auditor_service_operations_total |
counter | network,channel,namespace,method |
Total number of AuditorService method invocations |
panurus_core_common_metrics_auditor_service_duration_seconds |
histogram | network,channel,namespace,method |
Duration of AuditorService method calls in seconds |
panurus_core_common_metrics_auditor_service_errors_total |
counter | network,channel,namespace,method |
Total number of AuditorService method errors |
panurus_core_common_metrics_tokens_service_operations_total |
counter | network,channel,namespace,method |
Total number of TokensService method invocations |
panurus_core_common_metrics_tokens_service_duration_seconds |
histogram | network,channel,namespace,method |
Duration of TokensService method calls in seconds |
panurus_core_common_metrics_tokens_service_errors_total |
counter | network,channel,namespace,method |
Total number of TokensService method errors |
panurus_core_common_metrics_tokens_upgrade_service_operations_total |
counter | network,channel,namespace,method |
Total number of TokensUpgradeService method invocations |
panurus_core_common_metrics_tokens_upgrade_service_duration_seconds |
histogram | network,channel,namespace,method |
Duration of TokensUpgradeService method calls in seconds |
panurus_core_common_metrics_tokens_upgrade_service_errors_total |
counter | network,channel,namespace,method |
Total number of TokensUpgradeService method errors |
Source: token/core/common/metrics/{issue,transfer,auditor,tokens,upgrade}.go.
The ttx service orchestrates a token transaction from assembly to commit. These counters and
histograms mark the phases an operator cares about; they are the closest thing the SDK has to
throughput and latency of the business flow.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_ttx_endorsed_transactions |
counter | network,channel,namespace |
The number of endorsed transactions. |
panurus_services_ttx_audit_approved_transactions |
counter | network,channel,namespace |
The number of approved transactions by the auditor. |
panurus_services_ttx_accepted_transactions |
counter | network,channel,namespace |
The number of accepted transactions. |
panurus_services_ttx_endorsement_duration_seconds |
histogram | network,channel,namespace |
Duration of the full endorsement collection phase including signatures, audit, and chaincode approval. |
panurus_services_ttx_audit_approval_duration_seconds |
histogram | network,channel,namespace |
Duration of the auditor approval phase including validation, append, and signing. |
panurus_services_ttx_ordering_duration_seconds |
histogram | network,channel,namespace |
Duration of the transaction broadcast to the ordering service. |
Source: token/services/ttx/metrics.go. Wired in token/sdk/dig/sdk.go.
Recorded when a transaction’s ledger status resolves. hash_mismatch_total is the one to alert on: a
non-zero value means a committed token-request hash did not match the locally stored one, which is a
data-integrity violation rather than an ordinary failure.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_ttx_finality_finality_listener_confirmed_total |
counter | — | Total number of transactions confirmed on the ledger and successfully committed to local storage |
panurus_services_ttx_finality_finality_listener_deleted_total |
counter | — | Total number of transactions marked as deleted due to an invalid ledger status or token-request hash mismatch |
panurus_services_ttx_finality_finality_listener_hash_mismatch_total |
counter | — | Total number of transactions rejected because the committed token-request hash did not match the locally stored one |
panurus_services_ttx_finality_finality_listener_retry_exhausted_total |
counter | — | Total number of transactions whose finality processing was abandoned after all retries were exhausted |
panurus_services_ttx_finality_finality_listener_on_status_duration_seconds |
histogram | — | Histogram of total OnStatus processing time per transaction (including retries), in seconds |
Source: token/services/ttx/finality/metrics.go.
Recorded by the versioned-envelope wrapper used for view-to-view messaging. version is the envelope
version, type the message type, and error one of missing_version, version_mismatch,
type_mismatch, invalid_envelope or unknown (see classifyError in
token/services/utils/json/session/envelope.go).
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_utils_json_session_ttx_envelope_sent_total |
counter | version,type |
Total number of versioned envelopes sent |
panurus_services_utils_json_session_ttx_envelope_received_total |
counter | version,type |
Total number of versioned envelopes received |
panurus_services_utils_json_session_ttx_envelope_errors_total |
counter | error |
Total number of envelope validation errors |
panurus_services_utils_json_session_ttx_envelope_body_bytes |
histogram | type |
Size of envelope body in bytes |
Source: token/services/utils/json/session/metrics.go. Wired in token/sdk/dig/sdk.go.
Recorded by the auditor around Audit(), Append() and Release(). These metrics carry no TMS
labels, so a node auditing several TMSs reports them aggregated; see
Metric hygiene.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_auditor_auditor_audit_duration_seconds |
histogram | — | Histogram of Audit() processing time per transaction (including lock acquisition), in seconds |
panurus_services_auditor_auditor_audit_lock_conflicts_total |
counter | — | Total number of Audit() and Append() calls that failed to acquire enrollment-ID locks |
panurus_services_auditor_auditor_append_duration_seconds |
histogram | — | Histogram of Append() processing time per transaction, in seconds |
panurus_services_auditor_auditor_append_errors_total |
counter | — | Total number of Append() calls that failed to store the audit record |
panurus_services_auditor_auditor_releases_total |
counter | — | Total number of Release() calls (explicit and deferred) |
Source: token/services/auditor/metrics.go.
Recorded by the default token selector. outcome is one of success, insufficient_funds,
locked_funds or error; fetcher_type is eager or lazy. selection_immediate_retries shows how
often a selection had to retry because of concurrent lock contention. lock_conflicts_total and
distinct_tokens_attempted were added for #2395
to distinguish “one hot token retried many times” from “many tokens each contended once”;
lock_conflicts_total is deliberately unlabeled by token id or wallet id to avoid unbounded
cardinality — per-token attribution belongs in the selector’s debug-level log line (Lost lock
race on token [...], visible once the sherdlock package’s logger is at debug level) and in the
tokendiag locks command. lock_store_errors_total (also
#2395) counts the sibling case: a TryLock/TryLockBatch failure that is not a lock conflict (does
not wrap driver.ErrTokenAlreadyLocked) — a genuine store error such as a connection failure or
timeout. To the caller both currently surface identically as retried, eventually-locked-funds
contention, so this counter is what distinguishes “the store is unhealthy” from “tokens are just
contended” without changing that retry behavior.
stale_candidates_total (also #2395) counts the third case, which is neither: a candidate
dropped because the token was no longer spendable by the time the lock was attempted
(driver.ErrTokenNotSpendable). The eager fetcher serves candidates from a snapshot of the
token store, so a token spent after that snapshot was taken is still offered until the cache
refreshes; the lock is conditional on the token still being spendable, so such a candidate is
rejected rather than handed to a caller that could not load it. A non-zero rate here therefore
means the cache’s freshness interval is long relative to how fast the wallet is spending, not
that tokens are contended or that the store is unhealthy.
Both lock paths report it. A batch-capable backend — Postgres, via LockBatch, currently
the only implementation — claims a whole window of candidates in one statement, and that
statement classifies each candidate it did not win: the ones it refused as no longer spendable
come back as such, and “another claimant holds it” is the remainder. So a stale candidate on the
batch path is counted here, dropped for good, and used to refresh the candidate cache, exactly
as on the single-token path — which also means a selection no longer has to wait for the eager
cache’s own freshnessInterval to recover from one.
A backend is still allowed to win or not win a candidate without saying why; a selector treats
every unwon candidate from one of those as a lost race. On such a deployment
stale_candidates_total stays at zero even during a stale-candidate episode, and the symptom
to read is lock_conflicts_total rising without a corresponding rise in real contention (e.g.
with distinct_tokens_attempted flat and no competing senders). Neither correctness nor
recovery depends on the classification — a token the store refuses is never handed to a caller,
and a scan that comes up short refreshes the candidate cache whatever the store could say about
why (see cache_invalidations_total below) — the difference is purely in what is reported.
cache_invalidations_total (#2395) counts the forced refreshes those recoveries cost: each one
is the selector telling its candidate source that the source’s snapshot is behind the store, so
the next read of it re-queries. It is raised by a stale candidate, and by a scan that ran out of
candidates before covering the request — the candidate query hides tokens locked by others, so a
lock released since the snapshot was taken cannot appear by re-reading it. The invalidation
itself issues no query; it turns the next read into a full refresh for the eager and mixed
strategies, and several invalidations before that read collapse into one. The first fetch of a
selection is exempt, so a request served by its first scan costs nothing here; per Select the
count is bounded by selection_immediate_retries plus one per stale candidate. A rate near that
bound on most calls means the freshness interval is long relative to how fast the wallets turn
over, and the eager strategy is paying for a snapshot it cannot keep — fetcherStrategy: lazy
(or a shorter fetcherCacheRefresh) is then the cheaper configuration.
candidate_permutation_duration_seconds (#2510) is the time NewPermutation spends copying and
shuffling a wallet’s whole candidate list (the within-amount shuffle that spreads contention). It
is observed once per permutation performed and labelled fetcher_type: eager for a cache hit
(the eager strategy and the eager leg of mixed), lazy for a list just read from the store. A cache
miss shuffles nothing and observes nothing. In mixed, a call is normally observed under one of the
two; if the eager leg’s iterator then fails its peek, the call falls through to the lazy leg and
observes under both. Read it against selection_duration_seconds: if its sum is a large share of the selection
time, the shuffle (not the database or lock contention) is what a large cached wallet is paying; the
cost grows with the number of candidates (see BenchmarkNewPermutation). If #2491 changes the
shuffle, the histogram measures whatever NewPermutation then does.
For StubbornSelector, both selection_immediate_retries and distinct_tokens_attempted are
observed once per outer Select() call, aggregated over every internal backoff-retry attempt it
makes — not once per attempt. They aggregate differently: selection_immediate_retries counts
events and so sums the per-attempt counts, while distinct_tokens_attempted counts distinct
tokens and so unions them, meaning a token contended across several attempts is counted once.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_selector_sherdlock_unspent_tokens_invocations |
counter | fetcher_type |
The number of invocations |
panurus_services_selector_sherdlock_selection_duration_seconds |
histogram | — | Duration of a token selection call in seconds |
panurus_services_selector_sherdlock_selection_backoff_duration_seconds |
histogram | — | Time spent in a backoff wait between token selection rounds in seconds |
panurus_services_selector_sherdlock_selection_outcome_total |
counter | outcome |
Total number of token selection outcomes by result type |
panurus_services_selector_sherdlock_selection_immediate_retries |
histogram | — | Distribution of immediate retry counts per token selection call |
panurus_services_selector_sherdlock_lock_conflicts_total |
counter | — | Total number of lost lock races (a token was already locked by another process) |
panurus_services_selector_sherdlock_distinct_tokens_attempted |
histogram | — | Distribution of the number of distinct tokens a lock was attempted on (won, lost, or rate-limited) per token selection call |
panurus_services_selector_sherdlock_lock_store_errors_total |
counter | — | Total number of TryLock/TryLockBatch failures that are not lock conflicts (a genuine store error) |
panurus_services_selector_sherdlock_stale_candidates_total |
counter | — | Total number of candidate tokens dropped because they were no longer spendable when the lock was attempted |
panurus_services_selector_sherdlock_cache_invalidations_total |
counter | — | Total number of times the candidate source was told its snapshot is behind the token store, forcing the next read to refresh it |
panurus_services_selector_sherdlock_candidate_permutation_duration_seconds |
histogram | fetcher_type |
Duration in seconds of copying and shuffling a wallet’s candidate token list (NewPermutation) |
Backoff time. selection_duration_seconds includes the time a StubbornSelector sleeps between
rounds. selection_backoff_duration_seconds records each such wait (the elapsed time, so a wait cut
short by context cancellation records what was actually waited), which lets you split an elevated
Select latency into backoff versus selection work. Zero-length waits are observed too. Sherdlock
also waits after its final failed round, so a selection that gives up records max retries + 1 waits.
The metrics are process-global, with no tms label.
Source: token/services/selector/sherdlock/metrics.go.
Recorded by the simple selector driver. Only one selector driver is wired per process
(token.selector.driver), so this section and the sherdlock one are never both exported.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_selector_simple_selection_backoff_duration_seconds |
histogram | — | Time spent in a backoff wait between token selection rounds in seconds |
selection_backoff_duration_seconds records the elapsed time of each wait between selection
attempts, including a wait cut short by context cancellation. There is no wait after the last
attempt, and none at all when the manager’s retry timeout is non-positive; a jitter of exactly zero
is still observed as 0. The metric is process-global, with no tms label.
The simple driver exports no other selection metrics, in particular no selection_duration_seconds.
This histogram therefore shows how much time and how many waits backoff accounts for, but there is no
total selection latency to split it against, as there is for sherdlock.
Source: token/services/selector/simple/metrics.go.
Recorded by the opt-in per-wallet rate limiter (token.selector.rateLimit*, or the ratelimit
options), for both the sherdlock and simple selector drivers. Nothing is registered while rate
limiting is off. denied_total counts selection requests rejected with token.SelectorRateLimited,
which is what tells a policy-driven rejection apart from real lock contention
(lock_conflicts_total, selection_outcome_total{outcome="locked_funds"}).
denied_total is labeled by tms (the TMS id the limiter was asked about) and deliberately not by
wallet id, which would be unbounded cardinality; the denial error names the wallet. buckets_active
is a single process-wide gauge, since one limiter serves every TMS of a selector service; it assumes one metered limiter per process (two limiters sharing a provider overwrite each other). When it sits
at rateLimitMaxBuckets the limiter is evicting live wallets, which resets their allowance.
The names omit the ratelimit_ prefix the issue proposed, because the package name is already part
of the Prometheus name. The tms label is an intentional exception to the network/channel/namespace
convention: the limiter only knows the TMS id string. A pre-built limiter passed with
ratelimit.WithLimiter is metered only if it was built with ratelimit.Config.Metrics.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_selector_ratelimit_denied_total |
counter | tms |
Total number of token selection requests denied by the per-wallet rate limiter |
panurus_services_selector_ratelimit_buckets_active |
gauge | — | Number of live per-wallet buckets held by the token selection rate limiter |
Source: token/services/selector/ratelimit/metrics.go.
certified_tokens is recorded by the certification service (server side); the remaining four are
recorded by the certification client. Note the asymmetry in labels — the client-side metrics carry
channel and namespace but not network.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_certifier_interactive_certified_tokens |
counter | network,channel,namespace |
The number of tokens certified. |
panurus_services_certifier_interactive_certification_request_duration_seconds |
histogram | channel,namespace |
Histogram of certification batch request durations in seconds. |
panurus_services_certifier_interactive_certification_errors_total |
counter | channel,namespace |
Total number of failed certification request attempts. |
panurus_services_certifier_interactive_certification_pending_tokens |
gauge | channel,namespace |
Current number of tokens waiting in the certification input buffer. |
panurus_services_certifier_interactive_certification_dropped_tokens_total |
counter | channel,namespace |
Total number of tokens dropped because the certification buffer was full. |
Source: token/services/certifier/interactive/metrics.go.
All identity instrumentation is created through the TMS-scoped provider from the wallet service
(token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/ws.go), hence the
panurus_core_common_metrics_ prefix despite living under token/services/identity.
outcome and path are cache, routed or fallback. A near-zero
identity_signer_router_registrations_total in a busy deployment means routing is not being populated
and every signer lookup falls back to the probing deserializer. The two *_provision_failures_total
counters rise when a cache cannot pre-provision in the background, so requests are served on the slow
path while the identity backend keeps failing; the corresponding log line alone is easy to miss.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_core_common_metrics_identity_signer_resolutions_total |
counter | network,channel,namespace,outcome |
Total number of GetSigner calls by outcome (cache, routed, fallback) |
panurus_core_common_metrics_identity_get_signer_duration_seconds |
histogram | network,channel,namespace,path |
Histogram of GetSigner wall-clock time in seconds, labeled by resolution path |
panurus_core_common_metrics_identity_signer_router_registrations_total |
counter | network,channel,namespace |
Total number of conf_id-to-KeyManager bindings registered with the SignerRouter |
panurus_core_common_metrics_identity_signer_router_no_probe_errors_total |
counter | network,channel,namespace |
Total number of errors from the SignerRouter’s probe-free signer deserialization path |
panurus_core_common_metrics_cache_level |
gauge | network,channel,namespace |
Level of the idemix cache |
panurus_core_common_metrics_cache_provision_failures_total |
counter | network,channel,namespace |
Failed attempts to pre-provision idemix identities |
panurus_core_common_metrics_recipient_data_cache_level |
gauge | network,channel,namespace |
Level of the wallet recipient data cache |
panurus_core_common_metrics_recipient_data_provision_failures_total |
counter | network,channel,namespace |
Failed attempts to pre-provision wallet recipient data |
Source: token/services/identity/metrics.go, token/services/identity/idemix/cache/metrics.go,
token/services/identity/role/metrics.go.
Recorded by the observability decorator in token/services/observability/, which wraps the wallet
service and per-wallet instances (OwnerWallet, IssuerWallet, AuditorWallet, CertifierWallet)
with metrics and circuit breaking. Both driver paths (FabToken and ZKAT-DLog) receive a TMS-scoped
provider, hence the panurus_core_common_metrics_ prefix. All metrics carry wallet_id to prevent
metric collisions across wallets.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_core_common_metrics_wallet_requests_total |
counter | network,channel,namespace,method,wallet_id |
Total number of wallet service requests |
panurus_core_common_metrics_wallet_errors_total |
counter | network,channel,namespace,method,wallet_id |
Total number of wallet service errors |
panurus_core_common_metrics_wallet_request_duration_seconds |
histogram | network,channel,namespace,method,wallet_id |
Execution duration of wallet service operations in seconds |
panurus_core_common_metrics_wallet_inflight_requests |
gauge | network,channel,namespace,method,wallet_id |
Current number of in-flight wallet service requests |
panurus_core_common_metrics_wallet_circuit_breaker_rejections_total |
counter | network,channel,namespace,method,wallet_id |
Total number of requests rejected by the circuit breaker |
panurus_core_common_metrics_wallet_circuit_breaker_state |
gauge | network,channel,namespace,method,wallet_id |
Current circuit breaker state (0=Closed, 1=Open, 2=HalfOpen) |
Source: token/services/observability/metrics.go. Wired in token/core/{fabtoken/v1,zkatdlog/nogh/v1}/driver/driver.go.
Recorded by the background sweep in token/services/storage/services/checks, wired up from
token/sdk/db/checks.go through the TMS-scoped provider — hence the panurus_core_common_metrics_
prefix despite living under token/services/storage. role is owner or auditor, naming which
store a sweep covers; outcome is completed, not_leader, canceled or failed. See
Ledger Drift Checks for what the checks themselves look at.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_core_common_metrics_storage_checks_sweep_duration_seconds |
histogram | network,channel,namespace,role |
Histogram of the wall-clock time of one ledger drift checks sweep, in seconds |
panurus_core_common_metrics_storage_checks_sweeps_total |
counter | network,channel,namespace,role,outcome |
Total number of ledger drift checks sweeps by outcome (completed, not_leader, canceled, failed) |
panurus_core_common_metrics_storage_checks_findings_observed_total |
counter | network,channel,namespace,role,checker,code,severity |
Total number of ledger drift findings reported, by check, code and severity |
panurus_core_common_metrics_storage_checks_findings_open |
gauge | network,channel,namespace,role,severity |
Number of currently open ledger drift findings, by severity |
panurus_core_common_metrics_storage_checks_findings_resolved_total |
counter | network,channel,namespace,role |
Total number of stored ledger drift findings closed because a sweep stopped reporting them |
Source: token/services/storage/services/checks/metrics.go.
Recorded by the Fabric-X event queue that dispatches finality notifications to workers.
pending_events is the backlog gauge: it counts events accepted by the queue and not yet taken by a
worker. A caller blocked in EnqueueBlocking on a full buffer is counted as soon as it starts waiting, so
the gauge can exceed the configured queue size (token.finality.notification.queueSize); a value above it
means producers are blocked waiting for space. enqueue_drops_total increments when the buffer was full
and an event was discarded, which means a lost finality notification.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_services_network_fabricx_finality_queue_finality_queue_pending_events |
gauge | — | Number of finality events waiting for a worker, including EnqueueBlocking callers blocked on a full buffer, so it can exceed the queue size |
panurus_services_network_fabricx_finality_queue_finality_queue_enqueue_drops_total |
counter | — | Total number of finality events dropped because the queue was full |
panurus_services_network_fabricx_finality_queue_finality_queue_processing_errors_total |
counter | — | Total number of errors returned by event.Process in worker goroutines |
panurus_services_network_fabricx_finality_queue_finality_queue_processing_duration_seconds |
histogram | — | Histogram of successful event processing time in worker goroutines (seconds) |
Source: token/services/network/fabricx/finality/queue/metrics.go.
Recorded by a dbpool.Sampler that polls (*sql.DB).Stats() for every Postgres connection pool a
token store opens, once per second. It exists because Panurus never checks out a connection
itself — the pool lives inside fabric-smart-client’s postgres.Open and reaches Panurus only as a
*sql.DB — so Stats() is the only available signal, and it exposes cumulative totals
(WaitCount/WaitDuration), not per-wait observations. That rules out the histogram the parent
epic (#2492) originally proposed; a gauge
plus a counter pair is the closest honest signal instead — see
#2508.
pool identifies the underlying data source as host:port/database, not the store: every store
sharing a data source — including fabric-smart-client’s own kvs/binding/signerinfo/auditinfo
stores, which take the same DbProvider singleton — shares one *sql.DB, so a per-store label
would register several identical series for one pool, and in_use/wait_* are therefore node-wide
occupancy, not occupancy attributable to Panurus alone. The label is parsed out of the data source
rather than redacted from it, so a credential in a field redaction would miss (query parameters,
for instance) is never read, only host, port and the database name. Postgres advisory-lock
connections (opened outside the pool, e.g. by the recovery and keystore-cleanup leaders) are not
reflected in these metrics, since they never reach (*sql.DB).Stats().
in_use and max_open are deliberately instantaneous samples, not a max held since the
previous one. Sampling at 1s rather than something coarser is what makes a sustained exhaustion
episode — such as #2474, where a pool of 10
was permanently exhausted by connections with a ~250ms hold time — land on most samples instead of
being missed between them; detecting a short-lived burst does not depend on the sampling rate at
all, because wait_count_total increments once per blocked checkout regardless of how often
anything scrapes it. Peak occupancy is a query, not a stored value:
max_over_time(panurus_storage_db_pool_in_use[$__rate_interval]). Do not raise the sampling
interval (token.storage.db.poolMetricsInterval, default 1s) back toward something like 10s
without re-reading this — at a slower poll the gauge can sit well below max_open throughout an
episode that was, in fact, permanently exhausted between samples. 0 disables the sampler (and its
goroutine) entirely, for a node that does not scrape Prometheus.
| Metric | Type | Labels | Description |
|---|---|---|---|
panurus_storage_db_pool_in_use |
gauge | pool |
Number of connections currently checked out of the pool |
panurus_storage_db_pool_max_open |
gauge | pool |
Configured maximum number of open connections for the pool |
panurus_storage_db_pool_wait_count_total |
counter | pool |
Total number of connection checkouts that had to wait because the pool was at its connection limit |
panurus_storage_db_pool_wait_seconds_total |
counter | pool |
Total time, in seconds, callers spent waiting for a connection to become available |
Current saturation:
panurus_storage_db_pool_in_use / panurus_storage_db_pool_max_open
Peak saturation over the query window:
max_over_time(panurus_storage_db_pool_in_use[$__rate_interval]) / panurus_storage_db_pool_max_open
Mean connection wait:
rate(panurus_storage_db_pool_wait_seconds_total[$__rate_interval])
/ rate(panurus_storage_db_pool_wait_count_total[$__rate_interval])
Source: token/services/storage/db/sql/dbpool/metrics.go.
Transaction throughput per TMS:
sum by (network, channel, namespace) (rate(panurus_services_ttx_accepted_transactions[5m]))
95th percentile of the endorsement phase:
histogram_quantile(0.95, sum by (le, network, channel, namespace) (
rate(panurus_services_ttx_endorsement_duration_seconds_bucket[5m])
))
Share of token selections that failed for lack of spendable funds:
sum(rate(panurus_services_selector_sherdlock_selection_outcome_total{outcome!="success"}[5m]))
/ sum(rate(panurus_services_selector_sherdlock_selection_outcome_total[5m]))
Slowest driver methods, to find the expensive cryptography:
topk(5, histogram_quantile(0.99, sum by (le, method) (
rate(panurus_core_common_metrics_transfer_service_duration_seconds_bucket[5m])
)))
Fraction of signer lookups that fall back to the probing deserializer:
sum(rate(panurus_core_common_metrics_identity_signer_resolutions_total{outcome="fallback"}[5m]))
/ sum(rate(panurus_core_common_metrics_identity_signer_resolutions_total[5m]))
Integrity and loss alerts — any increase is worth paging on:
increase(panurus_services_ttx_finality_finality_listener_hash_mismatch_total[10m]) > 0
increase(panurus_services_network_fabricx_finality_queue_finality_queue_enqueue_drops_total[10m]) > 0
Finality backlog:
panurus_services_network_fabricx_finality_queue_finality_queue_pending_events
integration/nwo/txgen/service/metrics/ and integration/nwo/runner/
instrument the test harness, not the SDK, and are not exported by a Panurus node.The second half of this reference: what a Panurus node currently cannot tell an operator. Ordered by operator value. Each item states what is blind today, a concrete proposal, and why existing signals do not already cover it.
Connection-pool occupancy is now covered — see Storage connection pool.
What remains blind: tokendb, ttxdb, auditdb, identitydb and the keystore have no
per-operation instrumentation. A slow node cannot be attributed to a specific store or operation
from SDK metrics alone; the driver and ttx histograms include storage time without isolating it,
and the connection-pool metrics show whether the pool itself is the bottleneck but not which store
or query is holding it.
Proposal: store_operations_total{store,operation,outcome} (counter) and
store_operation_duration_seconds{store,operation} (histogram), recorded in the store service layer.
Why not just a database exporter: an exporter reports IO, contention and query statistics, but cannot
attribute them to store="tokendb"/operation="write" without per-query parsing rules that have to be
maintained against every schema change. Embedded backends such as SQLite have no exporter at all, so
SDK-level counters are the only available signal there. The two are complementary — the exporter for IO
and contention, these counters for application semantics.
panurus_services_ttx_accepted_transactions has no failure counterpart, so a success rate cannot be
computed from SDK metrics. A deployment where every transaction fails looks, in these metrics, like a
deployment with no traffic.
Proposal: panurus_services_ttx_transactions_total{network,channel,namespace,phase,outcome} (counter)
with phase in assemble|endorse|audit|order|commit and outcome in success|failure, from which
throughput and failure rate both follow.
Nothing counts transactions the ledger rejects, broken down by reason. Note that double spending can only be enforced at commit time, so this is where a double-spend attempt becomes visible — it cannot be detected earlier during validation.
Proposal: panurus_services_network_commit_rejections_total{network,channel,namespace,reason} (counter),
recorded where the commit result is processed, with reason distinguishing double-spend from signature,
policy and format failures.
The Fabric-X finality queue is instrumented; the standard Fabric network driver
(token/services/network/fabric/) has no equivalent, and the approval path — EndorserService in
token/services/network/fabric/endorsement/provider.go — is not timed. On Fabric deployments the
approval round trip is therefore invisible.
Proposal: panurus_services_network_fabric_approval_duration_seconds{network,channel,namespace}
(histogram) and ..._approval_errors_total{...,reason} (counter) around the endorser call.
Note that the views that drive this path are already instrumented by FSC; the gap is the endorsement round trip itself, not the view execution.
panurus_services_auditor_auditor_audit_lock_conflicts_total counts failed lock acquisitions but says
nothing about contention short of failure: how long locks are held, how long acquisition waits, how many
holders are active.
Proposal: panurus_services_auditor_lock_wait_duration_seconds and ..._lock_hold_duration_seconds
(histograms), plus ..._locks_held (gauge).
cache_level and recipient_data_cache_level report occupancy, not effectiveness. A cache that is
always full but never hit looks identical to one that serves every request.
Proposal: identity_cache_requests_total{cache,outcome} (counter) with outcome in hit|miss,
alongside the existing gauges. identity_signer_resolutions_total already does this for signers; the
identity and recipient-data caches deserve the same treatment.
Wallet lookups by identity, enrollment ID or wallet ID are not timed or counted, so a deployment with pathological wallet resolution has no signal until it shows up as end-to-end latency.
Proposal: panurus_services_identity_wallet_lookups_total{network,channel,namespace,role,outcome}
(counter) and a matching duration histogram.
Smaller consistency issues, worth fixing together since each one changes exported names:
panurus_core_common_metrics_ instead of their own package, which puts identity and cache metrics in
a namespace that has nothing to do with them. Setting Namespace/Subsystem explicitly in the opts,
or making the wrapper preserve the caller’s package, would fix it. Storage connection
pool already does this for its own metrics — the first family in the repo
to do so — precisely because it is a new metric with no existing dashboard or alert to break by
choosing a stable, explicit name up front.panurus_services_auditor_auditor_*,
panurus_services_ttx_finality_finality_listener_* and
panurus_services_network_fabricx_finality_queue_finality_queue_* repeat their package in the metric
name.network/channel/namespace labels, so on a node hosting several TMSs their values are aggregated
and cannot be attributed. The certification client metrics carry channel and namespace but not
network.All three are breaking changes for existing dashboards and alerts, so they belong in a single deliberate change with a note in the release notes — not folded into unrelated work.