[Finder] Using wildcard in in() leads to exceeding opened file descriptors limit #35508
Description
Symfony version(s) affected: 5.0.3, 4.3.0
Description
Using Symfony Finder with a wildcard as a documented feature of in method throws a fatal error if a number of directories that match the wildcard bigger than open file descriptors limit.
The current implementation resolves a wildcard to the list of directories regardless of their number and creates an iterator for each of them bringing the risk to exceed the limit in the system.
How to reproduce
composer init
composer require symfony/finder
touch find.php
<?php
include 'vendor/autoload.php';
use Symfony\Component\Finder\Finder;
mkdir('pool');
for ($i = 0; $i < 100; ++$i) {
$folder = 'pool/dir' . $i;
mkdir($folder);
file_put_contents($folder . '/target.txt', '');
}
$finder = new Finder();
$finder
->files('target.txt')
->in('pool/*')
->depth('== 0');
$cc = 0;
foreach ($finder as $file) {
$cc++;
}
echo 'Found: ' . $cc;bash -c "ulimit -n 50 && php find.php"Actual result: fatal error
Fatal error: Uncaught UnexpectedValueException: RecursiveDirectoryIterator::__construct(pool/dir50): failed to open dir: Too many open files in /private/tmp/finder/vendor/symfony/finder/Iterator/RecursiveDirectoryIterator.php:48
Stack trace:
#0 /private/tmp/finder/vendor/symfony/finder/Iterator/RecursiveDirectoryIterator.php(48): RecursiveDirectoryIterator->__construct('pool/dir50', 4096)
#1 /private/tmp/finder/vendor/symfony/finder/Finder.php(730): Symfony\Component\Finder\Iterator\RecursiveDirectoryIterator->__construct('pool/dir50', 4096, false)
#2 /private/tmp/finder/vendor/symfony/finder/Finder.php(619): Symfony\Component\Finder\Finder->searchInDirectory('pool/dir50')
#3 /private/tmp/finder/find.php(22): Symfony\Component\Finder\Finder->getIterator()
#4 {main}
thrown in /private/tmp/finder/vendor/symfony/finder/Iterator/RecursiveDirectoryIterator.php on line 48
Expected result: No error. Low open file descriptors usage.
Possible Solution
Avoiding creating X iterators.
Wildcard can bring millions of directories, so setting ulimit would not help.
Additional context
$dir = 'pool';
$directories = [];
foreach ($spryker as $dir) {
$directories[] = new \RecursiveDirectoryIterator($dir);
}
var_dump($directories);Running the code above within low ulimit -n leads to the same error.
It seems RecursiveDirectoryIterator opens a file descriptor on a creation and keeps it until destroyed.