-
-
Notifications
You must be signed in to change notification settings - Fork 312
Description
Hi,
We are using prometheus-net 8.2.0 and we are observing occasionally on all our services (and all the envs: staging, preprod, prod) those kind of errors:
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext()
at Prometheus.CollectorRegistry.UpdateRegistryMetrics()
at Prometheus.CollectorRegistry.CollectAndSerializeAsync(IMetricsSerializer serializer, CancellationToken cancel)
at Prometheus.MetricServerMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.InvokeCore(HttpContext context, PathString matchedPath, PathString remainingPath)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
It seems to happen very rarely per instance (between 1 and 3 times), an instance can live between a few hours and a few days.
I'm unclear if this is a recent regression or we never noticed it before. I would think it could come from recent 8.2.0 performance improvements as it touches this path:
9f07523#diff-92eed7c273df734f3fe9010cb841f28a3a855a8933857f646478c2d9c5949fe2
I'm not sure there is anything custom with our code, we have a pretty standard and simple usage of the lib. We rely on default Metrics.DefaultRegistry
and have maybe ~ 30 metrics per service, most of them being counter or histograms with some labels.
Thank you!