Skip to content

Commit ad2cdd4

Browse files
committed
Add documentation for InferenceGraph router timeouts
1 parent 154e471 commit ad2cdd4

File tree

1 file changed

+42
-0
lines changed
  • docs/modelserving/inference_graph/image_pipeline

1 file changed

+42
-0
lines changed

docs/modelserving/inference_graph/image_pipeline/README.md

+42
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,48 @@ are executed in sequence: it first sends the image as request to `cat-dog-classi
146146
When the condition is not matched the graph short circuits and returns the response from the previous step. Refer to [gjson syntax](https://github.com/tidwall/gjson/blob/master/SYNTAX.md)
147147
for how to express the condition and currently KServe only supports this with REST protocol.
148148

149+
### InferenceGraph router timeouts
150+
You can set custom timeout values for the `InferenceGraph` router.
151+
This is useful when `InferenceService`s are slow to start (e.g. when downloading / loading the model), or take a long time to complete.
152+
153+
- `serverRead` specifies the number of seconds to wait before timing out a request read by the server (default is `60`).
154+
- `serverWrite` specifies the maximum duration in seconds before timing out writes of the response (default is `60`).
155+
- `serverIdle` specifies the maximum amount of time in seconds to wait for the next request when keep-alives are enabled (default is `180`).
156+
- `serviceClient` specifies a time limit in seconds for requests made to the graph components by HTTP client (uses Go's [DefaultTransport](https://pkg.go.dev/net/http#DefaultTransport) values by default).
157+
158+
```bash
159+
kubectl apply -f - <<EOF
160+
apiVersion: "serving.kserve.io/v1alpha1"
161+
kind: "InferenceGraph"
162+
metadata:
163+
name: "dog-breed-pipeline"
164+
spec:
165+
routerTimeouts:
166+
serverRead: 300
167+
serverWrite: 300
168+
serverIdle: 300
169+
serviceClient: 150
170+
nodes:
171+
root:
172+
routerType: Sequence
173+
steps:
174+
- serviceName: cat-dog-classifier
175+
name: cat_dog_classifier # step name
176+
- serviceName: dog-breed-classifier
177+
name: dog_breed_classifier
178+
data: $request
179+
condition: "[@this].#(predictions.0==\"dog\")"
180+
resources:
181+
requests:
182+
cpu: 100m
183+
memory: 256Mi
184+
limits:
185+
cpu: 1
186+
memory: 1Gi
187+
188+
EOF
189+
```
190+
149191
## Test the InferenceGraph
150192
Before testing the `InferenceGraph`, first check if the graph is in the ready state and then get the router url for sending the request.
151193
```bash

0 commit comments

Comments
 (0)