Long term for Logstash documentation

[bmorelli25]

@karenzone, @lcawl, and I met to discuss Logstash/V3 today. Here are some notes and recommendations.

Summary

Logstash documentation consists of two main content sets:

1. Logstash Reference (LSR): Contains reference content from the logstash repo and plugin reference content from the logstash-docs repo.
2. Logstash Versioned Plugin Reference (VPR): Entirely sourced from the logstash-docs repo.

The logstash-docs repo acts as a middle-ground repository that houses all built AsciiDoc documentation for Logstash plugins.

The problem

The current automation for Logstash documentation is error-prone and time-consuming to manage. Rewriting this automation is necessary, but neither docs-eng nor logstash-eng has the resources to accomplish this by March 25. A practical plan is needed to determine what can realistically be done by this deadline.

Options

1. Leave all Logstash docs behind

We would not migrate Logstash docs to V3 at this time. Instead, we would connect the Logstash AsciiDoc content with the V3 experience and modify the legacy AsciiDoc tooling to allow this functionality. This option avoids additional V3 engineering work but requires tooling on the unknown Asciidoc build side. It is also likely to result in a poor user experience.

I'm really worried about this option because I don't know what is required on the Asciidoc side. Removing all of the other main and master docs will really bork up the Asciidoc build. There be 🐉's and I'm afraid this will turn into to way more work than we think.

2. Treat LSR and VPR as separate content sets (migrate some)

This approach would migrate the LSR to V3 without the plugin content and leave the VPR in AsciiDoc. While this simplifies the migration for part of the Logstash docs, it significantly degrades the user experience since Logstash is not functional without its plugins.

I see no win with this option. It's a poor user experience and combines all of the cons from both options 1 and 3.

3. Migrate both content sets repeatedly with adoc-to-md

1. Maintain the existing automation and layer the adoc-to-md migration tool on top of it.
2. Split the LSR into two toc.yml files:
   1. MD reference content (which will live in the logstash repo).
   2. MD plugin docs (which will live in the logstash-docs repo).
3. Keep the VPR entirely in the logstash-docs repo with its own toc.yml.
4. Use GitHub Actions to automate the adoc-to-md conversion script after each commit to the logstash-docs repo.
5. Source all three toc.yml files with the assembler

Differs a complete automation rewrite, offers a good user experience, and it might actually not be that difficult to implement.

I think it's worth further exploring option 3

Related

• Further musings, research, and discussion on a possible long term plan are in https://github.com/elastic/platform-docs-team/issues/409.
• Previous attempt at updating versioning in the VPR: Support stack versions in versioned plugin docs. docs-tools#70

[karenzone]

Thanks for writing this up, @bmorelli25. I'd like to work with you to clarify some info in your Summary and Problem statement sections. I'm convinced that Option 3 is the way to go.

[bmorelli25]

Meeting with @karenzone 2/3/25

I met with @karenzone today to discuss the Logstash/V3 migration, and we agree that option three is likely attainable within the next few weeks. Option three means that we will maintain the existing logstash-docs automation and layer the adoc-to-md migration tool on top of it.

We’re planning to sync with @siamakp-elastic and @lcawl to ensure we're aligned and can move forward smoothly. In the meantime, there are some tasks that Karen and I will get started on.

Logstash Versioned Plugin Reference (VPR)

The VPR docs currently live in the versioned_plugin_docs branch of the elastic/logstash-docs repo. This content is updated automatically, and any new files/dirs are wiped when this happens.

Example

I’ve migrated a point-in-time snapshot of the VPR docs and published them here.

These docs can be built using docs-builder. To build them, clone the docs-builder repo and run:

dotnet publish "src/docs-builder/docs-builder.csproj" && ./.artifacts/publish/docs-builder/release/docs-builder serve -p ../logstash.md

Short term to-dos:

• Clean up toc.yml that adoc-to-md outputs. @bmorelli25 to scope and sync with Siamak
• Clean up some syntax (not much at first glance). @karenzone to scope by building these docs and poking around.
  • Example of syntax to clean up.

Logstash Reference (LSR)

The LSR docs consist of two parts:

1. LSR Core: Content lives in the elastic/logstash repo. The LSR Core has already been migrated to markdown. This content will eventually (within the next 2-3 weeks) be moved back to the logstash repo, where it will live forever.
2. LSR Plugins: Content lives in the elastic/logstash-docs repo (main branch). The LSR Plugins is the complex part. This content is currently built from the index.asciidoc file in the logstash repo (view here).

@bmorelli25 is testing building this content from a new index.asciidoc file in the logstash-docs repo. This is causing quite a few errors, but I’ll continue investigating this in the following branch: logstash-docs.

To-dos:

• Build a working index.asciidoc file and fix all errors when trying to build from this repo.

Current plugin docs workflow

Right now, Karen's workflow with the Logstash plugin docs (both VPR and LSR Plugins) is:

1. Automation opens a PR in the logstash-docs repo
   1. main branch for LSR Plugins
   2. versioned_plugin_docs branch for VPR
2. Karen reviews and ensures the content will build
3. Karen merges the PR
4. The asciidoc build picks up the changes and they are shortly after deployed

Proposed plugin docs workflow

The goal of the above tasks is to get to the following workflow:

1. Automation opens a PR in the logstash-docs repo
   1. main branch for LSR Plugins
   2. versioned_plugin_docs branch for VPR
2. Karen reviews and ensures the content will build
3. Karen merges the PR
4. A post-commit action runs on both branches outlined above that builds the content with docs-builder
5. After a successful build, the markdown files are checked into a new repository, called logstash-md (no PR/review required)
6. The V3 assembler pulls this content into the larger build system and is deployed shortly after

Visualized

graph LR
    subgraph "Automation Process"
        A[Automation creates PR] --> B{Which Branch?}
        B -->|main| C[LSR Plugins]
        B -->|versioned_plugin_docs| D[VPR]
    end

    subgraph "Review Process"
        C --> E[Karen Reviews]
        D --> E
        E --> F{Builds Successfully?}
        F -->|No| G[Fix Issues]
        G --> E
        F -->|Yes| H[Merge PR]
    end

    subgraph "Post-Merge Process"
        H --> I[Post-commit Action]
        I --> J[Build with docs-builder]
        J --> K{Build Successful?}
        K -->|No| L[Fix Build]
        L --> J
        K -->|Yes| M[Commit to logstash-md repo]
    end

    subgraph "Deployment"
        M --> N[V3 Assembler]
        N --> O[Deploy to Docs Site]
    end

Loading

[karenzone]

@robbavey and @jsvd: FYI. Early phases don't require Logstash dev resources. Looping you in here and for future discussions.

[karenzone]

@bmorelli25: To clarify, during Doc Freeze and migration, we can still work on plugin docs, and merge the content from plugin docgen to these branches:

• versioned_plugin-docs. (Versioned Plugin Reference)
• 8.x and any other 8.x branches (8.18, 8.17, etc.)
• 7.17
  Please correct me if I'm wrong.

[bmorelli25]

Yes. Unlike other repos, you can also merge to main in the logstash-docs repo.

[bmorelli25]

Talked with @reakaleek. Notes:

• We could consider publishing the adoc-to-md image to a public endpoint. However, because the plan would be to run this as a post-commit action, we don't need to. Building directly in the action will only add ~15 seconds to build times.
• We will need a fine-grained token with access to the adoc-to-md repo as it is private.
• The logstash-docs-md repo should be public

Overall, it sounds very doable.

[lcawl]

Hi! Not sure if these are the existing syntax errors mentioned in earlier comments, but while addressing errors in the migrated reference content, the codeblock in the JDBC static filter plugin generated lots of this error: " Both inline and classic callouts are not supported".

It's due to the # run loaders every 2 hours in the codeblock, which I've fixed for now in the output here: https://github.com/elastic/asciidocalypse/pull/10

Ideally it would be fixed in the source files so it doesn't reappear when the content is re-generated. If you want me to create an issue in the logstash or logstash-docs repo to track that request, just lmk!

[karenzone]

Thanks for this cleanup work, @lcawl.

[karenzone]

Yes. Unlike other repos, you can also merge to main in the logstash-docs repo.

In that case, let's talk about removing the asciidoc warning from the logstash-docs repo so that people can use my plugin docs validation hack.
elastic/logstash-docs#1803 (comment)

[reakaleek]

Help:

I cannot run adoc-to-md. I followed the steps in https://github.com/elastic/adoc-to-md?tab=readme-ov-file#converting-asciidoc-content-to-markdown.

cd src/docker
bash create_image.sh

And I get this error:

 => ERROR [ruby_deps 4/7] COPY docs/Gemfile* .                                                                                    0.0s
 => CACHED [ruby_deps 5/7] RUN bundle install --binstubs --system --frozen --without test                                         0.0s
 => ERROR [ruby_deps 6/7] COPY docs/.docker/asciidoctor_2_0_10.patch .                                                            0.0s
------
 > [ruby_deps 4/7] COPY docs/Gemfile* .:
------
------
 > [ruby_deps 6/7] COPY docs/.docker/asciidoctor_2_0_10.patch .:
------
Dockerfile:29
--------------------
  27 |     # docker which lets us lock the versions in place.
  28 |     RUN bundle install --binstubs --system --frozen --without test
  29 | >>> COPY ${DOCS_DIR}/.docker/asciidoctor_2_0_10.patch .
  30 |     RUN cd /var/lib/gems/2.5.0/gems/asciidoctor-2.0.10 && patch -p1 < /asciidoctor_2_0_10.patch
  31 |     
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 7bb57acc-1179-4633-9de1-df0494e0f267::uvl6str2e5f7uy784eak76y0h: "/docs/.docker/asciidoctor_2_0_10.patch": not found

[bmorelli25]

@colleenmcginnis is working on migrating our Logstash Plugin docs from HTML to MD instead of from asciidoc to MD. She has a working prototype and her and @karenzone are finalizing ownership details.

As her PR fixes our short-term needs, I'm going to move this from blocked to backlog and we'll revisit a longer-term solution when the team has more availability.