Skip to content

docs: for FlowLiveUpdater to accomodate latest async/start change #467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 11, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 47 additions & 18 deletions docs/docs/core/flow_methods.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: Flow Running
toc_max_heading_level: 4
description: Run a CocoIndex Flow, including build / update data in the target storage and evaluate the flow without changing the target storage.
---

Expand Down Expand Up @@ -70,8 +71,7 @@ This is to achieve best efficiency.

### One time update

<Tabs queryString="code_lang">
<TabItem value="shell" label="Shell" default>
#### CLI

The `cocoindex update` subcommand creates/updates data in the target storage.

Expand All @@ -81,7 +81,9 @@ Once it's done, the target data is fresh up to the moment when the function is c
python main.py cocoindex update
```

</TabItem>
#### Library API

<Tabs>
<TabItem value="python" label="Python">

The `update()` async method creates/updates data in the target storage.
Expand All @@ -108,8 +110,7 @@ A data source may enable one or multiple *change capture mechanisms*:

Change capture mechanisms enable CocoIndex to continuously capture changes from the source data and update the target data accordingly, under live update mode.

<Tabs queryString="code_lang">
<TabItem value="shell" label="Shell" default>
#### CLI

To perform live update, run the `cocoindex update` subcommand with `-L` option:

Expand All @@ -120,7 +121,9 @@ python main.py cocoindex update -L
If there's at least one data source with change capture mechanism enabled, it will keep running until the aborted (e.g. by `Ctrl-C`).
Otherwise, it falls back to the same behavior as one time update, and will finish after a one-time update is done.

</TabItem>
#### Library API

<Tabs>
<TabItem value="python" label="Python">

To perform live update, you need to create a `cocoindex.FlowLiveUpdater` object using the `cocoindex.Flow` object.
Expand All @@ -145,15 +148,19 @@ my_updater = cocoindex.FlowLiveUpdater(

A `FlowLiveUpdater` object supports the following methods:

* `start()`: Start the updater.
CocoIndex will continuously capture changes from the source data and update the target data accordingly in background threads managed by the engine.
* `abort()`: Abort the updater.
* `wait()` (async): Wait for the updater to finish. It only unblocks in one of the following cases:
* `wait()`: Wait for the updater to finish. It only unblocks in one of the following cases:
* The updater was aborted.
* A one time update is done, and live update is not enabled:
either `live_mode` is `False`, or all data sources have no change capture mechanisms enabled.
* `update_stats()`: It returns the stats of the updater.

```python
my_updater = cocoindex.FlowLiveUpdater(demo_flow)
# Start the updater.
my_updater.start()

# Perform your own logic (e.g. a query loop).
...
Expand All @@ -163,26 +170,47 @@ print(my_updater.update_stats())
# Abort the updater.
my_updater.abort()
# Wait for the updater to finish.
await my_updater.wait()
my_updater.wait()
```

Python SDK also allows you to use the updater as a context manager.
It will abort and wait for the updater to finish automatically when the context is exited.
The following code is equivalent to the code above:
It will automatically start the updater during the context entry, and abort and wait for the updater to finish automatically when the context is exited.
The following code is equivalent to the code above (if no early return happens):

```python
async with cocoindex.FlowLiveUpdater(demo_flow) as my_updater:
with cocoindex.FlowLiveUpdater(demo_flow) as my_updater:
# Perform your own logic (e.g. a query loop).
...
print(my_updater.update_stats())
```

Within a synchronous function, remove `async` before `with`, like this:
CocoIndex also provides asynchronous versions of APIs for blocking operations, including:

```python
with cocoindex.FlowLiveUpdater(demo_flow) as my_updater:
* `start_async()` and `wait_async()`, e.g.

```python
my_updater = cocoindex.FlowLiveUpdater(demo_flow)
# Start the updater.
await my_updater.start_async()

# Perform your own logic (e.g. a query loop).
...
```

# Print the update stats.
print(my_updater.update_stats())
# Abort the updater.
my_updater.abort()
# Wait for the updater to finish.
await my_updater.wait_async()
```
* Async context manager, e.g.

```python
async with cocoindex.FlowLiveUpdater(demo_flow) as my_updater:
# Perform your own logic (e.g. a query loop).
...
print(my_updater.update_stats())
```

</TabItem>
</Tabs>
Expand All @@ -191,8 +219,7 @@ with cocoindex.FlowLiveUpdater(demo_flow) as my_updater:

CocoIndex allows you to run the transformations defined by the flow without updating the target storage.

<Tabs queryString="code_lang">
<TabItem value="shell" label="Shell" default>
### CLI

The `cocoindex evaluate` subcommand runs the transformation and dumps flow outputs.
It takes the following options:
Expand All @@ -208,7 +235,9 @@ Example:
python main.py cocoindex evaluate --output-dir ./eval_output
```

</TabItem>
### Library API

<Tabs>
<TabItem value="python" label="Python">

The `evaluate_and_dump()` method runs the transformation and dumps flow outputs to files.
Expand Down