Skip to content

Commit 8e4c710

Browse files
ruivieiraruivieira
authored
Merge pull request #42 from christinaexyou/add-gorch-tutorial
Add GORCH tutorial
2 parents c1d6340 + ca37e70 commit 8e4c710

File tree

3 files changed

+218
-1
lines changed

3 files changed

+218
-1
lines changed

‎docs/modules/ROOT/pages/component-gorch.adoc

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
= GuardrailsOrchestrator
2+
3+
image::gorch-architecture.svg[GuardrailsOrchestrator architecture diagram]

‎docs/modules/ROOT/pages/gorch-tutorial.adoc

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
= Getting Started with GuardrailsOrchestrator
2+
3+
xref:component-gorch.adoc[GuardrailsOrchestrator] is a service for large language model guardrailing underpinned by the open-source project link:https://github.com/foundation-model-stack/fms-guardrails-orchestrator[fms-guardrails-orchestrator]. GuardrailsOrchestrator is a component of the xref:trustyai-operator.adoc[TrustyAI Kubernetes Operator]. In this tutorial, you will learn how to create a `GuardrailsOrchestrator` CR to
4+
perform detections on text generation output
5+
6+
[NOTE]
7+
GuardrailsOrchestrator is only available in TrustyAI's 1.30.0 community builds and later via KServe Raw Deployment mode.
8+
9+
In order to use it on Open Data Hub or OpenShift AI, first enable `KServe Raw Deployment`. In the `DataScienceIntialization` resource, set the value of `managementState` for the `serviceMesh` component to `Removed`.
10+
11+
[source,yaml]
12+
---
13+
serviceMesh:
14+
auth:
15+
audiences:
16+
- 'https://kubernetes.default.svc'
17+
controlPlane:
18+
metricsCollection: Istio
19+
name: data-science-smcp
20+
namespace: istio-system
21+
managementState: Removed
22+
---
23+
24+
Next, in the `DataScienceCluster` resource,under the spec.components section, set the value of of kserve.serving.managementState to `Removed` and add the following `devFlag`:
25+
26+
[source,yaml]
27+
---
28+
trustyai:
29+
devFlags:
30+
manifests:
31+
- contextDir: config
32+
sourcePath: ''
33+
uri: https://github.com/trustyai-explainability/trustyai-service-operator/tarball/main
34+
managementState: Managed
35+
---
36+
37+
== The GuardrailsOrchestrator
38+
39+
The GuardrailsOrchestrator service defines a new Custom Resource Definition called: *`GuardrailsOrchestrator`*. `GuardrailsOrchestrator` objects are monitored by the xref:trustyai-operator.adoc[TrustyAI Kubernetes operator]. A GuardrailsOrchestrator object represents an orchestration service that invokes detectors on text generation input/output and standalone detections. Therefore, to run an orchestration service, you need to create a `GuardrailsOrchestrator` object with ...
40+
41+
Here is a minimal example of a `GuardrailsOrchestrator` object:
42+
43+
[source,yaml]
44+
---
45+
apiVersion: trustyai.opendatahub.io/v1alpha1
46+
kind: GuardrailsOrchestrator
47+
metadata:
48+
name: gorch-sample
49+
spec:
50+
orchestratorConfig: "fms-orchestr8-config-nlp" <1>
51+
replicas: 1 <2>
52+
---
53+
54+
<1> The orchestratorConfig field specifies a ConfigMap object that contains generator, detector, and chunker arguments.
55+
<2> The replicas field specifies the number of replicas for the orchestrator.
56+
57+
Here is a minimal example of an ochestratorConfig object:
58+
[source,yaml]
59+
---
60+
kind: ConfigMap
61+
apiVersion: v1
62+
metadata:
63+
name: fms-orchestr8-config-nlp
64+
data:
65+
config.yaml: |
66+
generation: <1>
67+
service:
68+
hostname: llm-predictor.guardrails-test.svc.cluster.local
69+
port: 8032
70+
detectors: <2>
71+
regex:
72+
type: text_contents
73+
service:
74+
hostname: "127.0.0.1"
75+
port: 8080
76+
chunker_id: whole_doc_chunker
77+
default_threshold: 0.5
78+
---
79+
80+
<1> The generation field specifies the hostname and port of the large language model predictor service.
81+
<2> The detectors field specifies the hostname and port of the detector service, the chunker ID, and the default threshold.
82+
83+
After you apply the example `orchestratorConfig` and `GuardrailsOrchestrator` above, you can check its readiness by using the following command:
84+
85+
[source,shell]
86+
---
87+
oc get pods | grep gorch-sample
88+
---
89+
90+
The expected output is:
91+
[source,shell]
92+
---
93+
gorch-sample-6776b64c58-xrxq9 3/3 Running 0 4h19m
94+
---
95+
96+
== Details of GuardrailsOrchestrator
97+
In this section, let's review all the possible parameters for the `GuardrailsOrchestrator` object and their usage.
98+
99+
[cols="1,2a", options="header"]
100+
|===
101+
|Parameter |Description
102+
|`replicas`| The number of orchestrator pods to spin up
103+
|`orchestratorConfig`| The name of the ConfigMap object that contains generator, detector, and chunker arguments
104+
|`vLLMGatewayConfig **(optional)**`| The name of the ConfigMap object that contains VLLM gateway arguments
105+
|`otelExporter **(optional)**`| List of paired name and value arguments for configuring OpenTelemetry traces and/or metrics
106+
107+
* `protocol` - sets the protocol for all the OTLP endpoints. Acceptable values are `grpc` or`http`
108+
* `tracesProtocol` - overrides the protocol for traces. Acceptable values are `grpc` or `http`
109+
* `metricsProtocol` - overrides the protocol for metrics. Acceptable values are either `grpc` or `http`
110+
* `otlpEndpoint` - sets the OTLP endpoint. Defaults are `gRPC localhost:4317` and `HTTP localhost:4318`
111+
* `metricsEndpoint` - overrides the OTLP endpoint for metrics
112+
* `tracesEndpoint` - overrides the OTLP endpoint for traces
113+
* `otlpExport` - specifies a list of data types to export. Acceptable values are `traces`, `metrics`, or `traces,metrics`
114+
|===
115+
116+
== Optional Configurations for GuardrailsOrchestrator
117+
118+
== Configuring the Regex Detector and vLLM Gateway
119+
The regex detector and vLLM gateway are two sidecar images that can be used with the GuardrailsOrchestrator service. To enable them, the user must (1) specify their images in a ConfigMap (2) specify detectors they wish to use as well as the routes (3) reference the ConfigMap in the `GuardrailsOrchestrator` object:
120+
121+
Here is an example of a ConfigMap that references the regex detector and vLLM gateway images:
122+
[source,yaml]
123+
---
124+
apiVersion: v1
125+
kind: ConfigMap
126+
metadata:
127+
name: gorch-sample-config
128+
data:
129+
regexDetectorImage: 'quay.io/csantiago/regex-detector@sha256:2dbfa4680938a97d0e0cac75049c43687ad163666cf2c6ddc37643c4f516d144' <1>
130+
vllmGatewayImage: 'quay.io/csantiago/vllm-orchestrator-gateway@sha256:493ac4679d50db9c2c59967dcaa6737a995cd19f319727f33c40f159db6817db <2>
131+
---
132+
133+
<1> The regex detector is a sidecar image that provides regex-based detections
134+
<2> The vLLM gateway is a sidecar image that emulates the vLLM chat completions API and saves preset detector configurations
135+
136+
Here is an example of a vLLM gateway ConfigMap named `fms-orchestr8-config-gateway`:
137+
[source,yaml]
138+
---
139+
kind: ConfigMap
140+
apiVersion: v1
141+
metadata:
142+
name: fms-orchestr8-config-gateway
143+
labels:
144+
app: fmstack-nlp
145+
data:
146+
config.yaml: |
147+
orchestrator:
148+
host: "localhost"
149+
port: 8032
150+
detectors:
151+
- name: regex
152+
detector_params:
153+
regex:
154+
- email
155+
- ssn
156+
- name: other_detector
157+
routes:
158+
- name: pii
159+
detectors:
160+
- regex
161+
- name: passthrough
162+
detectors:
163+
---
164+
165+
Let's review all the required arguments for the regex detector:
166+
167+
[cols="1,2a", options="header"]
168+
|===
169+
|Parameter |Description
170+
|`orchestrator`| The orchestrator service
171+
|`detectors`| A list of preconfigured regexes for common detection actions
172+
|`routes`| The resulting endpoints for detections
173+
|===
174+
175+
Here is an example of a corresponding `GuardrailsOrchestrator` object that references the vLLM Gateway ConfigMap:
176+
177+
[source,yaml]
178+
---
179+
apiVersion: trustyai.opendatahub.io/v1alpha1
180+
kind: GuardrailsOrchestrator
181+
metadata:
182+
name: gorch-sample
183+
spec:
184+
orchestratorConfig: "fms-orchestr8-config-nlp"
185+
vllmGatewayConfig: "fms-orchestr8-config-gateway"
186+
replicas: 1
187+
---
188+
189+
== Configuring the OpenTelemetry Exporter for Metrics & Tracing
190+
Traces and metrics are provided for the observability of the GuardrailsOrchestrator service via the OpenTelemetry Operator.
191+
192+
Pre-requisites:
193+
194+
* Install the Red Hat OpenShift distributed tracing platform from the OperatorHub. Create a Jaeger instance using the default settings.
195+
196+
* Install the Red Hat build of OpenTelemetry from the OperatorHub. Create an OpenTelemetry instance
197+
198+
Here is a minimal example of a `GuardrailsOrchestrator` object that has OpenTelemetry configured:
199+
200+
[source,yaml]
201+
---
202+
apiVersion: trustyai.opendatahub.io/v1alpha1
203+
kind: GuardrailsOrchestrator
204+
metadata:
205+
name: gorch-test
206+
spec:
207+
orchestratorConfig: "fms-orchestr8-config-nlp"
208+
vllmGatewayConfig: "fms-orchestr8-config-gateway"
209+
replicas: 1
210+
otelExporter:
211+
protocol: "http"
212+
otlpEndpoint: "localhost:4318"
213+
otlpExport: "metrics"
214+
---

‎docs/modules/ROOT/pages/lm-eval-tutorial.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ xref:component-lm-eval.adoc[LM-Eval] is a service for large language model evalu
77
[NOTE]
88
====
99
LM-Eval is only available since TrustyAI's 1.28.0 community builds.
10-
In order to use it on Open Data Hub, you need to use either ODH 2.20 (or newer) or add the following `devFlag` to you `DataScienceCluster` resource:
10+
In order to use it on Open Data Hub, you need to use either ODH 2.20 (or newer) or add the following `devFlag` to your `DataScienceCluster` resource:
1111
1212
[source,yaml]
1313
----

0 commit comments

Comments
 (0)