How to deal with Async

Yoann Gendrey -

Hi,

That's a pleasure to be back . I am working as a Backend Developer at Pops and I am focused on the Print part of it. I have designed, built and written an automated way to treat images and videos. I am doing various kind of processing and those are ASYNCHRONOUSLY process. In order to solve this I am mainly working with:

**ASYNC**

I already talked about async in this article Flexible and reusable functions. I'll try to dig deeper into few of sub functions or just illustrate how I used them into my workflow on a daily basis.

You can get async like this:
yarn add async or npm install async

NPM: https://www.npmjs.com/package/async
Github: https://github.com/caolan/async
Documentation: http://caolan.github.io/async/docs.html

Once you get it you are ready to go.

I am mostly using the following methods depending on my inputs and usage:

  • concat
  • detect
  • each
  • eachOf
  • every
  • map
  • doWhilst
  • parallel
  • series
  • retry
  • forever

As I said in my previous post, when I was talking about reusable functions, various Async functions has their own series version, meaning operations through the iterator will be completed synchronously rather than in parallel for their default version. Most of the time I invite you to keep on eye on the Async doc, you will get exactly how the Async sub function is working and also few examples in order to understand how they are working. To be honest I did misread the doc from time to time, but since I am using it on a daily basis, I guess I am now aware about callback rules and async behaviors, at least for those I am using.

In my daily programming I am using Async for two main things:

  • Dealing with asynchronous operations
  • Flow control

Dealing with asynchronous operations means I need to complete a set of operations on a bunch of data and I want to execute another operation after all those operations with (err, results) final callback generally. Obviously this is wrapped with promise .

Flow control is about that according to Node.js' own design see event-loop you are aware that between asynchronous operations you never know when all of your async operations are done. You can design various implementation to track this down but at the end its start to become tricky to know what is really going on behind the scene. Async is definitely here to sort those thing out and makes things easier.

Concat

see concat doc
This one will typically take as input an array, the iterator will concat all returned data to the final array, then you'll get your final array.

eg.

const _ = require('lodash')
const numbers = _.range(4) // [0, 1, 2, 3]

const iterator = (element, callback) => {
...doing something asynchronous stuff or not with element
callback(null, concatenatedValue)
}

async.concat(numbers, iterator,(err,concatenatedArray) => {
...All async operations are done
...get your concatenated Array here
console.log(concatenatedArray)
})

Because of working on AWS, I used this to parse event with Serverless. AWS is sending 'records' and you may receive one or multiple event at once through multiple events, but you never know how much events you will receive and also how much events you will process at once. In order to process all items once I gathered them into a single array and then returned back to the function that are dealing with my 'S3 records' for example.

detect

see concat doc
I recently used this one, with his 'series' version, detectSeries. I was going through an Array and I wanted to get the index of items depending on a condition. Basically I was doing image analysis and I wanted to know where was situated the first image that was matching my criteria. I did not use the default version (parallel) for two reasons.

  1. Doing so in parallel might generate extra work while analyzing more images than needed
  2. It was not accurate in my case since my analyzed array of images was ordered, and doing so in paralel even if I was able to find a matching image Async was not able to guarantee that the returned element was the first one among all items and I was actually looking for this.

For detect the doc is saying:
Returns the first value in coll that passes an async truth test.

each/eachOf

see each doc
see eachOf doc

I liked each for a while it is almost working like a native JavaScript forEach loop. As always you'll pass an async function as iterator. Final optional callback is called once either you went through all items or an error happened. According to me each is good, really good, but most of the time in my working flow it was not appropriate. Since even if one async operation failed I still wanted to run through all my elements and make operations. Actually it depend on what you are looking for if you want to be sure that all items have been processed the same way without any error just check the err parameter in the final callback. eachOf is working exactly the same way but you will have an extra parameter for the iterator an index value it should be helpful in some cases.

every

Every looks similar to each but it not the same. Each let you apply a function on elements whereas every will run a test on all elements.

map

Map is working like the native one (array as input, modified array as result), in addition you'll have the async wrapper with its final callback. Note that with map all operations are done in parallel, but the transformed and returned array is supposed to keep the initial order. I read this in the doc:

Note, that since this function applies the iteratee to each item in parallel, there is no guarantee that the iteratee functions will complete in order. However, the results array will be in the same order as the original coll.

It means that you have to use the mapSeries function only if the order of processing each item in the input array matters to you otherwise there might be no reason of using it.

doWhilst

A simple doWhile async function. I used this in a deterministic algorithm of repartition at work. I needed to put items into another object. I was not able to know in advance how many time I needed to iterate. Basicaly there was a condition like a normal do while and as long as the condition was not met the loop is going forward.

parallel/series

It will let you run multiple async functions in parallel then you will get your final callback. It's good when you want to optimize treatments while doing multiple operations (exactly how Node.js is working) at once and also wanting to control your flow and get your result back into the final callback. The opposite of this is series where you will perform all of your operations one after the other and then get all results at the end.

retry

Retry is great you can iterate a number of time and you can also set a delay between every iteration. Like a forEach with a setTimeout but easily managed.

forever

I did work with forever also. This one is nothing more than a a while(1) but again async will add the magic to you and let you manage your process in a better way. I worked with forever in order to create a kind of daemon. Basically I was working with a Amazon Simple Queue Service (SQS). As long as there was no items in my queue nothing is happening, and when I get one the whole process were starting.

That's a brief overview of what Async is about. Async is one of my highest usage as module dependency on a daily basis like Lodash for TypeScript/Node.js. Hope you guys enjoy this overview, again I'm inviting you to check the documentation out in order to find the right async function for your needs and I hope it will helps you.