Skip to content

"Collection was modified" in CollectAndSerializeAsync() #464

@gboucher90

Description

@gboucher90

Hi,

We are using prometheus-net 8.2.0 and we are observing occasionally on all our services (and all the envs: staging, preprod, prod) those kind of errors:

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext()
   at Prometheus.CollectorRegistry.UpdateRegistryMetrics()
   at Prometheus.CollectorRegistry.CollectAndSerializeAsync(IMetricsSerializer serializer, CancellationToken cancel)
   at Prometheus.MetricServerMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.InvokeCore(HttpContext context, PathString matchedPath, PathString remainingPath)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

It seems to happen very rarely per instance (between 1 and 3 times), an instance can live between a few hours and a few days.

I'm unclear if this is a recent regression or we never noticed it before. I would think it could come from recent 8.2.0 performance improvements as it touches this path:
9f07523#diff-92eed7c273df734f3fe9010cb841f28a3a855a8933857f646478c2d9c5949fe2

I'm not sure there is anything custom with our code, we have a pretty standard and simple usage of the lib. We rely on default Metrics.DefaultRegistry and have maybe ~ 30 metrics per service, most of them being counter or histograms with some labels.

Thank you!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions