javascript - How to create a ZIP file with Gulp that contains a lot of files? -


i have gulp task add lots of files (more 2700 in 1 case, can several thousands in others cases) in zip file. code follow:

const fs = require('fs'); const archiver = require('archiver')('zip');  let zip = fs.createwritestream('my-archive.zip'); return gulp.src('app/**/*')   .pipe(through.obj((file, encoding, cb) => {     let pathinzip = '...';     if (!isadirectory(file.path)) { // not zip directory       archiver.append(fs.createreadstream(file.path), {         name: pathinzip,         mode: fs.statsync(file.path)       });     }     cb(null, file);   }, cb => {     // create zip file!     archiver.pipe(zip);     archiver.finalize();     cb();   }));

this code works on small projects, when deals more 2000 files, following error:

events.js:154 throw er; // unhandled 'error' event ^  error: emfile: many open files, open 'd:\dev\app\some\file' @ error (native)

so understand having 2000+ files opened @ same time before writing them in zip not idea.

how can ask zip file written without having need open files?

thanks.

for information: node 5.5.0 / npm 3.8.5 / archiver 1.0.0 / windows

gulp takes care of lot of things you're trying do:

  • gulp.src() reads file contents , makes fs.stat() call each file. stores both file.contents , file.stat on vinyl-file objects emits.
  • it using graceful-fs package, automatically backs off in case of emfile error , retries when file has closed. prevents "too many open files" problem you're experiencing.

unfortunately you're not taking advantage of of because:

  • you're making explicit calls fs.statsync() , fs.createreadstream(). there's no need since gulp has done you. you're reading each file twice (and creating twice number of file descriptors in process).
  • you're circumventing gulp's built-in protection against emfile making direct use of fs module not have guards against "too many open files" problem.

i've rewritten code take advantage of gulp's features. i've tried make little more gulp-idiomatic, e.g. using gulp-filter rid of directories:

const gulp = require('gulp'); const fs = require('graceful-fs'); const archiver = require('archiver')('zip'); const through = require('through2'); const filter = require('gulp-filter');  gulp.task('default', () => {   var zip = fs.createwritestream('my-archive.zip');   archiver.pipe(zip);   return gulp.src('app/**/*')     .pipe(filter((file) => !file.stat.isdirectory()))     .pipe(through.obj((file, encoding, cb) => {       var pathinzip = '...';       archiver.append(file.contents, {         name: pathinzip,         mode: file.stat       });       cb(null, file);     }, cb => {       zip.on('finish', cb);       archiver.finalize();     })); });

Comments

Post a Comment

Popular posts from this blog

jOOQ update returning clause with Oracle -

according http://www.jooq.org/doc/3.8/manual/sql-building/sql-statements/update-statement/ jooq doesn't support returning clause in update statements databases except firebird , postgres. does know, still correct oracle jooq? according oracle's documentation, oracle db supports returning clase delete, insert, or update statements. incorrect manual you're right - manual wrong , should fixed: https://github.com/jooq/jooq/issues/5470 single row update returning jooq 3.8 added support single row update .. returning statements. internally, uses callablestatement , pl/sql block of form: begin update "my_table" set "my_table"."column" = 'xyz' "my_table"."id" = 1 returning "my_table"."id", "my_table"."column" ?, ?; ? := sql%rowcount; end; the relevant issue is: https://github.com/jooq/jooq/issues/5190 multi row update returning a...
Read more

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

i have entity inherits other. on other hand, i'm using lombok project reduce boilerplate code, put @data annotation. annotation @data inheritance produces next warning: generating equals/hashcode implementation without call superclass, though class not extend java.lang.object. if intentional, add '@equalsandhashcode(callsuper=false)' type. is advisable add annotation @equalsandhashcode (callsuper = true) or @equalsandhashcode (callsuper = false) ? if not added, 1 callsuper=false or callsuper=true ? the default value false . 1 if don't specify , ignore warning. yes, recommended add @equalsandhashcode annotation on @data annotated classes extend else object. cannot tell if need true or false , depends on class hierarchy, , need examined on case-by-case basis. however, project or package, can configure in lombok.config call super methods if not direct subclass of object. lombok.equalsandhashcode.callsuper = call for configuration system ...
Read more

java - BasicPathUsageException: Cannot join to attribute of basic type -

i using java 8 jpa (hibernate 5.2.1). have clause works perfectly, until introduce clause make use foreign key table values too. i getting following error: basicpathusageexception: cannot join attribute of basic type i think problem related fact have join table, , not sure how create join using join table. employee - employee_category - category model (employee.java): @manytomany(cascade = cascadetype.all, fetch=fetchtype.eager) @jointable ( name="employee_category", joincolumns={ @joincolumn(name="emp_id", referencedcolumnname="id") }, inversejoincolumns={ @joincolumn(name="cat_id", referencedcolumnname="id") } ) private set<category> categories; model (category.java): @id private string id; jpa when introduce following code, fails error above (i think problem new piece of code): join<t, category> category = from.join("id"); predicates.add(criteriabuilder.like(category....
Read more