Complete normalization of class jsdocs

Author: huntharo

src/iterable-mapper.ts

@@ -82,18 +82,41 @@ type NewElementOrError<NewElement = unknown> = {
 };
 
 /**
- * Iterates over a source iterable with specified concurrency,
+ * Iterates over a source iterable / generator with specified `concurrency`,
  * calling the `mapper` on each iterated item, and storing the
- * `mapper` result in a queue of specified max size, before
+ * `mapper` result in a queue of `maxUnread` size, before
  * being iterated / read by the caller.
  *
  * @remarks
  *
- * Optimized for I/O-bound operations (e.g., fetching data, reading files) rather than
- * CPU-intensive tasks. The concurrent processing with backpressure ensures efficient
- * resource utilization without overwhelming memory or system resources.
+ * ### Typical Use Case
+ * - Prefetching items from an async I/O source
+ * - In the simple sequential (`concurrency: 1`) case, allows items to be prefetched async, preserving order, while caller processes an item
+ * - Can allow parallel prefetches for sources that allow for out of order reads (`concurrency:  2+`)
+ * - Prevents the producer from racing ahead of the consumer if `maxUnread` is reached
+ *
+ * ### Error Handling
+ *   The mapper should ideally handle all errors internally to enable error handling
+ *   closest to where they occur. However, if errors do escape the mapper:
+ *
+ *   When `stopOnMapperError` is true (default):
+ *   - First error immediately stops processing
+ *   - Error is thrown from the `AsyncIterator`'s next() call
+ *
+ *   When `stopOnMapperError` is false:
+ *   - Processing continues despite errors
+ *   - All errors are collected and thrown together
+ *   - Errors are thrown as `AggregateError` after all items complete
+ *
+ * ### Usage
+ * - Items are exposed to the `mapper` via an iterator or async iterator (this includes generator and async generator functions)
+ * - IMPORTANT: `mapper` method not be invoked when `maxUnread` is reached, until items are consumed
+ * - The iterable will set `done` when the `input` has indicated `done` and all `mapper` promises have resolved
+ *
+ * @example
  *
  * Consider a typical processing loop without IterableMapper:
+ *
  * ```typescript
  * const source = new SomeSource();
  * const sourceIds = [1, 2,... 1000];
@@ -109,43 +132,81 @@ type NewElementOrError<NewElement = unknown> = {
  * We could prefetch the next read (300ms) while processing (20ms) and writing (500ms),
  * without changing the order of reads or writes.
  *
- * Using IterableMapper as a prefetcher:
+ * @example
+ *
+ * Using `IterableMapper` as a prefetcher and blocking writes, without changing the order of reads or writes:
+ *
  * ```typescript
  * const source = new SomeSource();
  * const sourceIds = [1, 2,... 1000];
  * // Pre-reads up to 8 items serially and releases in sequential order
  * const sourcePrefetcher = new IterableMapper(sourceIds,
  *   async (sourceId) => source.read(sourceId),
- *   { concurrency: 1 }
+ *   { concurrency: 1, maxUnread: 10 }
  * );
  * const sink = new SomeSink();
- * for await (const item of sourcePrefetcher) {
+ * for await (const item of sourcePrefetcher) {    // may not block for fast sources
  *   const outputItem = doSomeOperation(item);     // takes 20 ms of CPU
  *   await sink.write(outputItem);                 // takes 500 ms of I/O wait, no CPU
  * }
  * ```
  *
  * This reduces iteration time to 520ms by overlapping reads with processing/writing.
  *
- * For maximum throughput, make the writes concurrent with
- * IterableQueueMapper (to iterate results with backpressure when too many unread items) or
- * IterableQueueMapperSimple (to handle errors at end without custom iteration or backpressure):
+ * @example
+ *
+ * Using `IterableMapper` as a prefetcher with background writes, without changing the order of reads or writes:
  *
  * ```typescript
  * const source = new SomeSource();
  * const sourceIds = [1, 2,... 1000];
  * const sourcePrefetcher = new IterableMapper(sourceIds,
  *   async (sourceId) => source.read(sourceId),
+ *   { concurrency: 1, maxUnread: 10 }
+ * );
+ * const sink = new SomeSink();
+ * const flusher = new IterableQueueMapperSimple(
+ *   async (outputItem) => sink.write(outputItem),
  *   { concurrency: 1 }
  * );
+ * for await (const item of sourcePrefetcher) {    // may not block for fast sources
+ *   const outputItem = doSomeOperation(item);     // takes 20 ms of CPU
+ *   await flusher.enqueue(outputItem);            // will periodically block for portion of write time
+ * }
+ * // Wait for all writes to complete
+ * await flusher.onIdle();
+ * // Check for errors
+ * if (flusher.errors.length > 0) {
+ *  // ...
+ * }
+ * ```
+ *
+ * This reduces iteration time to about to `max((max(readTime, writeTime) - cpuOpTime, cpuOpTime))
+ * by overlapping reads and writes with the CPU processing step.
+ * In this contrived example, the loop time is reduced to 500ms - 20ms = 480ms.
+ * In cases where the CPU usage time is higher, the impact can be greater.
+ *
+ * @example
+ *
+ * For maximum throughput, allow out of order reads and writes with
+ * `IterableQueueMapper` (to iterate results with backpressure when too many unread items) or
+ * `IterableQueueMapperSimple` (to handle errors at end without custom iteration and applying backpressure to block further enqueues when `concurrency` items are in process):
+ *
+ * ```typescript
+ * const source = new SomeSource();
+ * const sourceIds = [1, 2,... 1000];
+ * const sourcePrefetcher = new IterableMapper(sourceIds,
+ *   async (sourceId) => source.read(sourceId),
+ *   { concurrency: 10, maxUnread: 20 }
+ * );
  * const sink = new SomeSink();
  * const flusher = new IterableQueueMapperSimple(
  *   async (outputItem) => sink.write(outputItem),
  *   { concurrency: 10 }
  * );
- * for await (const item of sourcePrefetcher) {
+ * for await (const item of sourcePrefetcher) {    // typically will not block
  *   const outputItem = doSomeOperation(item);     // takes 20 ms of CPU
- *   await flusher.enqueue(outputItem);           // usually takes no time
+ *   await flusher.enqueue(outputItem);            // typically will not block
  * }
  * // Wait for all writes to complete
  * await flusher.onIdle();
@@ -184,19 +245,6 @@ export class IterableMapper<Element, NewElement> implements AsyncIterable<NewEle
    * @param mapper Function called for every item in `input`. Returns a `Promise` or value.
    * @param options IterableMapper options
    *
-   * Error Handling:
-   * The mapper should ideally handle all errors internally to enable error handling
-   * closest to where they occur. However, if errors do escape the mapper:
-   *
-   * When stopOnMapperError is true (default):
-   * - First error immediately stops processing
-   * - Error is thrown from the AsyncIterator's next() call
-   *
-   * When stopOnMapperError is false:
-   * - Processing continues despite errors
-   * - All errors are collected and thrown together
-   * - Errors are thrown as AggregateError after all items complete
-   *
    * @see {@link IterableQueueMapper} for full class documentation
    */
   constructor(

src/iterable-queue-mapper-simple.ts

@@ -1,30 +1,55 @@
-import { Mapper } from './iterable-mapper';
+import { IterableMapperOptions, Mapper } from './iterable-mapper';
 import { IterableQueueMapper } from './iterable-queue-mapper';
 
 // eslint-disable-next-line @typescript-eslint/no-explicit-any
 type Errors<T> = { item: T; error: string | { [key: string]: any } | Error }[];
 
 const NoResult = Symbol('noresult');
 
+/**
+ * Options for IterableQueueMapperSimple
+ */
+export type IterableQueueMapperSimpleOptions = Pick<IterableMapperOptions, 'concurrency'>;
+
 /**
  * Accepts queue items via `enqueue` and calls the `mapper` on them
- * with specified concurrency, storing the
- * `mapper` result in a queue of specified max size, before
- * being iterated / read by the caller.  The `enqueue` method will block if
- * the queue is full, until an item is read.
+ * with specified `concurrency`, discards the results, and accumulates
+ * exceptions in the `errors` property.  When empty, `await enqueue()`
+ * will return immediately, but when `concurrency` items are in progress,
+ * `await enqueue()` will block until a slot is available to accept the item.
  *
  * @remarks
  *
- * Note: the name is somewhat of a misnomer as this wraps `IterableQueueMapper`
- * but is not itself an `Iterable`.
+ * ### Typical Use Case
+ * - Pushing items to an async I/O destination
+ * - In the simple sequential (`concurrency: 1`) case, allows 1 item to be flushed async while caller prepares next item
+ * - Results of the flushed items are not needed in a subsequent step (if they are, use `IterableQueueMapper`)
+ *
+ * ### Error Handling
+ *   The mapper should ideally handle all errors internally to enable error handling
+ *   closest to where they occur. However, if errors do escape the mapper:
+ *   - Processing continues despite errors
+ *   - All errors are collected in the `errors` property
+ *   - Errors can be checked/handled during processing via the `errors` property
  *
- * Accepts items for mapping in the background, discards the results,
- * but accumulates exceptions in the `errors` property.
+ *   Key Differences from `IterableQueueMapper`:
+ *   - `maxUnread` defaults to equal `concurrency` (simplifying queue management)
+ *   - Results are automatically iterated and discarded (all work should happen in mapper)
+ *   - Errors are collected rather than thrown (available via errors property)
  *
- * Allows up to `concurrency` mappers to be in progress before
- * `enqueue` will block until a mapper completes.
+ * ### Usage
+ * - Items are added to the queue via the `await enqueue()` method
+ * - Check `errors` property to see if any errors occurred, stop if desired
+ * - IMPORTANT: `await enqueue()` method will block until a slot is available, if queue is full
+ * - IMPORTANT: Always `await onIdle()` to ensure all items are processed
+ *
+ * Note: the name is somewhat of a misnomer as this wraps `IterableQueueMapper`
+ * but is not itself an `Iterable`.
  *
  * @category Enqueue Input
+ *
+ * @see {@link IterableQueueMapper} for related class with more configuration options
+ * @see {@link IterableMapper} for underlying mapper implementation and examples of combined usage
  */
 export class IterableQueueMapperSimple<Element> {
   private readonly _writer: IterableQueueMapper<Element, typeof NoResult>;
@@ -34,29 +59,17 @@ export class IterableQueueMapperSimple<Element> {
   private _isIdle = false;
 
   /**
-   * Create a new `IterableQueueMapperSimple`
+   * Create a new `IterableQueueMapperSimple`, which uses `IterableQueueMapper` underneath, but
+   * automatically iterates and discards results as they complete.
    *
-   * @param mapper Function which is called for every item in `input`.
-   *    Expected to return a `Promise` or value.
-   *
-   *    The `mapper` *should* handle all errors and not allow an error to be thrown
-   *    out of the `mapper` function as this enables the best handling of errors
-   *    closest to the time that they occur.
-   *
-   *    If the `mapper` function does allow an error to be thrown then the
-   *    errors will be accumulated in the `errors` property.
+   * @param mapper Function called for every enqueued item. Returns a `Promise` or value.
    * @param options IterableQueueMapperSimple options
+   *
+   * @see {@link IterableQueueMapperSimple} for full class documentation
+   * @see {@link IterableQueueMapper} for related class with more configuration options
+   * @see {@link IterableMapper} for underlying mapper implementation and examples of combined usage
    */
-  constructor(
-    mapper: Mapper<Element, void>,
-    options: {
-      /**
-       * Number of items to accept for mapping before requiring the caller to wait for one to complete.
-       * @default 4
-       */
-      concurrency?: number;
-    } = {},
-  ) {
+  constructor(mapper: Mapper<Element, void>, options: IterableQueueMapperSimpleOptions = {}) {
     const { concurrency = 4 } = options;
 
     this._mapper = mapper;
@@ -106,7 +119,7 @@ export class IterableQueueMapperSimple<Element> {
   /**
    * Accept a request for sending in the background if a concurrency slot is available.
    * Else, do not return until a concurrency slot is freed up.
-   * This provides concurrency background writes with back pressure to prevent
+   * This provides concurrency background writes with backpressure to prevent
    * the caller from getting too far ahead.
    *
    * MUST await `onIdle` for background `mappers`s to finish

src/iterable-queue-mapper.ts

@@ -1,87 +1,65 @@
 //
 // 2021-08-25 - Initially based on: https://raw.githubusercontent.com/sindresorhus/p-map/main/index.js
 //
-import { IterableMapper, Mapper } from './iterable-mapper';
+import { IterableMapper, IterableMapperOptions, Mapper } from './iterable-mapper';
 import { IterableQueue } from './iterable-queue';
 
-export interface IterableQueueMapperOptions {
-  /**
-   * Number of concurrently pending promises returned by `mapper`.
-   *
-   * Must be an integer from 1 and up or `Infinity`, must be <= `maxUnread`.
-   *
-   * @default 4
-   */
-  readonly concurrency?: number;
-
-  /**
-   * Number of pending unread iterable items.
-   *
-   * Must be an integer from 1 and up or `Infinity`, must be >= `concurrency`.
-   *
-   * @default 8
-   */
-  readonly maxUnread?: number;
-
-  /**
-   * When set to `false`, instead of stopping when a promise rejects, it will wait for all the promises to settle and then reject with an [aggregated error](https://github.com/sindresorhus/aggregate-error) containing all the errors from the rejected promises.
-   *
-   * @default true
-   */
-  readonly stopOnMapperError?: boolean;
-}
+/**
+ * Options for IterableQueueMapper
+ */
+export type IterableQueueMapperOptions = IterableMapperOptions;
 
 /**
  * Accepts queue items via `enqueue` and calls the `mapper` on them
- * with specified concurrency, storing the
- * `mapper` result in a queue of specified max size, before
- * being iterated / read by the caller.  The `enqueue` method will block if
- * the queue is full, until an item is read.
+ * with specified `concurrency`, storing the `mapper` result in a queue
+ * of `maxUnread` size, before being iterated / read by the caller.
+ * The `enqueue` method will block if the queue is full, until an item is read.
  *
  * @remarks
  *
- * This allows performing a concurrent mapping with
- * back pressure for items added after queue creation
- * via a method call.
+ * ### Typical Use Case
+ * - Pushing items to an async I/O destination
+ * - In the simple sequential (`concurrency: 1`) case, allows 1 item to be flushed async while caller prepares next item
+ * - Results of the flushed items are needed in a subsequent step (if they are not, use `IterableQueueMapperSimple`)
+ * - Prevents the producer from racing ahead of the consumer if `maxUnread` is reached
+ *
+ * ### Error Handling
+ *   The mapper should ideally handle all errors internally to enable error handling
+ *   closest to where they occur. However, if errors do escape the mapper:
  *
- * Because items are added via a method call it is possible to
- * chain an `IterableMapper` that prefetches files and processes them,
- * with an `IterableQueueMapper` that processes the results of the
- * `mapper` function of the `IterableMapper`.
+ *   When `stopOnMapperError` is true (default):
+ *   - First error immediately stops processing
+ *   - Error is thrown from the `AsyncIterator`'s next() call
  *
- * Typical use case is for a `background uploader` that prevents
- * the producer from racing ahead of the upload process, consuming
- * too much memory or disk space. As items are ready for upload
- * they are added to the queue with the `enqueue` method, which is
- * `await`ed by the caller.  If the queue has room then `enqueue`
- * will return immediately, otherwise it will block until there is room.
+ *   When `stopOnMapperError` is false:
+ *   - Processing continues despite errors
+ *   - All errors are collected and thrown together
+ *   - Errors are thrown as `AggregateError` after all items complete
+ *
+ * ### Usage
+ * - Items are added to the queue via the `await enqueue()` method
+ * - IMPORTANT: `await enqueue()` method will block until a slot is available, if queue is full
+ * - Call `done()` when no more items will be enqueued
+ * - IMPORTANT: Always `await onIdle()` to ensure all items are processed
  *
  * @category Enqueue Input
+ *
+ * @see {@link IterableMapper} for underlying mapper implementation and examples of combined usage
  */
 export class IterableQueueMapper<Element, NewElement> implements AsyncIterable<NewElement> {
   private _iterableMapper: IterableMapper<Element, NewElement>;
 
   private _sourceIterable: IterableQueue<Element>;
 
   /**
-   * Create a new `IterableQueueMapper`
-   *
-   * @param mapper Function which is called for every item in `input`.
-   *    Expected to return a `Promise` or value.
+   * Create a new `IterableQueueMapper`, which uses `IterableMapper` underneath, and exposes a
+   * queue interface for adding items that are not exposed via an iterator.
    *
-   *    The `mapper` *should* handle all errors and not allow an error to be thrown
-   *    out of the `mapper` function as this enables the best handling of errors
-   *    closest to the time that they occur.
-   *
-   *    If the `mapper` function does allow an error to be thrown then the
-   *   `stopOnMapperError` option controls the behavior:
-   *      - `stopOnMapperError`: `true` - will throw the error
-   *        out of `next` or the `AsyncIterator` returned from `[Symbol.asyncIterator]`
-   *        and stop processing.
-   *     - `stopOnMapperError`: `false` - will continue processing
-   *        and accumulate the errors to be thrown from `next` or the `AsyncIterator`
-   *        returned from `[Symbol.asyncIterator]` when all items have been processed.
+   * @param mapper Function called for every enqueued item. Returns a `Promise` or value.
    * @param options IterableQueueMapper options
+   *
+   * @see {@link IterableQueueMapper} for full class documentation
+   * @see {@link IterableMapper} for underlying mapper implementation and examples of combined usage
    */
   constructor(mapper: Mapper<Element, NewElement>, options: IterableQueueMapperOptions = {}) {
     this._sourceIterable = new IterableQueue({