Unfortunately, this doesn't appear to have fixed the helgrind issues I've observed. I'm attaching helgrind output for our moose/test/tests/geomsearch/nearest_node_locator/nearest_node_locator.i input file. Note that this is not an urgent issue for us!

helgrind-nearest-node-locator.log

Good, since if it had been urgent, I'd have screwed you over pretty badly by this point, no? :-P

That log seems to suggest that it's actually the values of _array_is_present, _values_read_only, _first, _last, and the _read_only_values pointer that are under contention, not the value of actual array entries! Which is good, since that means my careful work to keep reads after the corresponding writes might actually be working, but also bad, since this pushes the credence of "Roy understands the cool tricks @friedmud used in #856 and #1056" from "probably" down to "probably not".

Just looking at that first "Possible data race" entry, it seems like both threads 26 and 27 are in the same PetscVector::get -> _get_array stack, but the whole point of the _array_is_present tricks there is to make sure that we can call get() simultaneously from multiple threads without a data race.

Is it possible that helgrind produces a false positive when encountering a properly-fenced double-checked lock idiom? That idiom actually was supposed to be buggy before C++11 memory fences, and text search through the Helgrind manual doesn't find the word "fence", and I don't see anything obviously relevant in the "happens-before" discussion there. What does look possibly relevant is "Do not roll your own threading primitives (mutexes, etc) from ... atomic counters": if our std::atomic<bool> idiom is just too awesome for Helgrind to understand, then our only option for clean output from it may be to add helgrind.h support and use ANNOTATE_HAPPENS_BEFORE etc. to hold it's hand.

I'm going to be darkly amused if it actually turns out that we caught a major bug by tracking down a false positive bug report.

Interesting stuff!

This failure still makes me think that we have an issue. When this simulation runs to completion, there are four time steps, but in that CIVET target we crash between time steps 3 and 4.

I've verified that there was a true race condition to MOOSE console output during adaptivity. I should know the answer to this, but could a console output race condition cause a crash?