Skip to content

Fix the problem with writing a file using streams and not using RAM #2558

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Fix the problem with writing a file using streams and not using RAM #2558

wants to merge 11 commits into from

Conversation

zlooun
Copy link

@zlooun zlooun commented Oct 17, 2023

Summary

This PR fixes the problem with writing a file using streams and not using RAM. It solves the problem of writing large excel files.

Test plan

import * as fs from 'fs';
import { stream } from 'exceljs';

const output_file_name = "/test.xlsx";

const writeStream = fs.createWriteStream(output_file_name, { flags: 'w' });
const wb = new stream.xlsx.WorkbookWriter({ stream: writeStream });
const worksheet = wb.addWorksheet("test");

const headers = Array.from({length: 256}, (_, i) => i + 1).map((i) => 'test' + i);

for (let i = 0; i < 100000; i++) {
  const row = headers.map((header) => header + '|' + i);
  await worksheet.addRow(row).commit(); // This raw will be immediately written to disk and will not clog RAM.
}

await worksheet.commit(); // This is not necessary because await wb.commit() is used, but you can also write to disk not raw by raw, but worksheet by worksheet.
await wb.commit();

Related to source code (for typings update)

https://github.com/zlooun/exceljs
https://www.npmjs.com/package/@zlooun/exceljs

This was referenced Oct 17, 2023
Open
Open
@Siemienik Siemienik self-requested a review October 17, 2023 22:23
@Siemienik
Copy link
Member

Thank you for the contribution; I believe it enhances the handling of large file writing 🎉. Please bear with me, as this change breaks backward-compatibility and will need to wait for the next major release (5.0).

@shuntagami shuntagami mentioned this pull request Dec 18, 2023
Merged
@sneko sneko mentioned this pull request Mar 25, 2024
Open
@AmberWilkie
Copy link

Hello just hoping to bump some attention here. This PR would be very helpful for my work. I'd offer to help get it in but honestly I learned everything I know about streams and buffers trying to understand why await workbook.commit() was hanging forever. In my case, it's because my S3 upload releases the stream, which I expected to do after committing the workbook. So effectively that await will be forever if I put it before the S3 upload terminates.

(Or then again, I could be off base here - like I said, new to streams).

@julesblancphin
Copy link

Are there any plans to merge this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Siemienik Siemienik Awaiting requested review from Siemienik

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
hacktoberfest-accepted
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zlooun @Siemienik @AmberWilkie @julesblancphin