Upload Zip File to S3 and Then Expand

Excerpt Zip Files From and Dorsum to the S3 Saucepan using Node.js

A lambda role to unzip files from an S3 bucket.

I have been wondering how to upload multiple files from the client-side of my web application for a couple of days now. And I constitute a similar question on StackOverflow but the accepted respond was implemented using the AWS CLI. I tin can certainly implement this using the CLI through child processes using Node.js in the backend. But surely at that place exist a better and faster approach than this.

Another article was talking virtually uploading using promise.all only that is certainly not an option on the customer-side. I also found some implementation using python and boto3 only my app was written in Node.js and then I can't afford to add another language to my dependency list. Then I searched for more on this topic and I establish this article written by Akmal Sharipov. Surely, his article was helpful but I faced some issues using this library (might be my error how I implemented it).

After some digging, I establish that the structure of the nix file has its Central Directory located at the last of the file, and there exist local headers that are a copy of the Central Directory simply are highly unreliable. And the read methods of most other streaming libraries buffers the entire null file in the retention thus, defeating the entire purpose of streaming it in the first place. And so here'south another algorithm I came up with using the yauzl library which he mentioned in his article too.

1. The user uploads lots of files through my app.
2. My app zips those files using the yazl library and uploads it in an S3 bucket on the client-side.
3. An S3 'put' event triggers the lambda role.
4. Lambda function pulls the whole object (goose egg file)into its retentivity buffer.
5. Information technology reads ane entry and uploads it dorsum to S3.
6. When the upload finishes, information technology gain to the next entry and repeats step 5.

This algorithm never hits the RAM limit of the lambda role. The max memory usage was under 500MB for extracting a 254MB nothing file containing 2.24GB of files.

Create a Lambda function and add together an S3 trigger

Before creating the lambda function, create an IAM function with full S3 and lambda permissions. At present go to the lambda services section and and so click on the 'Create Function' button on the superlative right corner. On the next page, select 'Author from scratch' and give a proper noun for the office. In the Permissions section, select 'use an existing role' and choose a office that y'all defined earlier — you can always change the permission subsequently. Then click on the 'Create Function' push button on the lesser right corner of the page.

Configure the Lambda function such that information technology'll be triggered whenever a zip file is uploaded to the S3 bucket. Click on the 'add together trigger' button on the Function overview section and select an S3 event from the dropdown. So choose your bucket and select 'PUT' every bit the issue type and also don't forget to add '.cypher' in the suffix field or it'll self invoke the function in a loop. Then click on 'add' to add the trigger on the lambda office.

Lambda function to fetch objects from the S3 bucket

We will apply the AWS SDK to become the file from the S3 saucepan. Inside the event object of the handler function, we have the Records assortment that contains the S3 saucepan name and the key of the object that triggered this lambda function. Nosotros volition use the Records assortment to become the bucket and the central for the S3.getObject(params, [callback]) method to fetch the zip file.

Lambda function to get an object from the S3 bucket that triggered it.

Upload object to S3 bucket from a stream

This is the master piece of the code that was a lilliputian tricky. Apparently, passing the readable stream directly to the Body property of the params parameter doesn't piece of work at least in the lambda environment. So I searched in GOOGLE — as I do commonly — and I establish this StackOverflow question which fits my requirement perfectly. Here's the code.

To use this function nosotros need to call the function first and then pipe the readable stream to writeStream belongings returned by it.

How to utilise the uploadStream part?

Unzip the object and upload each entry back to S3

Yauzl library is the well-nigh popular library in the npm registry. This library is likewise very reliable and I don't want my app to crash for some silly library issue. The documentation is very user-friendly too. I'll utilize the fromBuffer method as the naught file will be available to us via the buffer only. Let's dive into the code for unzipping the file.

This function will read one entry from the null file so upload it back to the S3 saucepan using the uploadStream part we implemented before. After the upload of ane entry/file has finished, then but it will read the next entry and upload it and the loop goes on until the stop of the zip file is encountered. It resolves the promise with the 'end' message when at that place's zip left in the zero file.

Putting information technology all together

The unabridged code is available in the higher up repository. You just demand to clone the repository and and then run npm install followed by npm run nil. So upload the lambda.zilch file in the lambda's code section of the panel.

Don't worry if you don't want to clone my repository, you can paste all the functions I shared inside the index.js file and replace panel.log(object) at line 15 of the 'exports.handler' with this line const upshot = await extractZip(Bucket, object.Body);

.

Last Words

I just implemented the algorithm a little differently than the other implementation I discussed earlier. My approach was to non stream the zilch file simply to stream the entries one at a fourth dimension. Simply all the other algorithms are great also.

If you find this article helpful, make sure to get out a handclapping for me. And here's some other commodity y'all might like.

Happy hacking!

More content at plainenglish.io

bogartthichise.blogspot.com

Source: https://aws.plainenglish.io/extract-zip-files-from-and-back-to-the-s3-bucket-using-node-js-f19f009ace22

Share this post