You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/modelserving/inference_graph/image_pipeline/README.md
+42
Original file line number
Diff line number
Diff line change
@@ -146,6 +146,48 @@ are executed in sequence: it first sends the image as request to `cat-dog-classi
146
146
When the condition is not matched the graph short circuits and returns the response from the previous step. Refer to [gjson syntax](https://github.com/tidwall/gjson/blob/master/SYNTAX.md)
147
147
for how to express the condition and currently KServe only supports this with REST protocol.
148
148
149
+
### InferenceGraph router timeouts
150
+
You can set custom timeout values for the `InferenceGraph` router.
151
+
This is useful when `InferenceService`s are slow to start (e.g. when downloading / loading the model), or take a long time to complete.
152
+
153
+
-`serverRead` specifies the number of seconds to wait before timing out a request read by the server (default is `60`).
154
+
-`serverWrite` specifies the maximum duration in seconds before timing out writes of the response (default is `60`).
155
+
-`serverIdle` specifies the maximum amount of time in seconds to wait for the next request when keep-alives are enabled (default is `180`).
156
+
-`serviceClient` specifies a time limit in seconds for requests made to the graph components by HTTP client (uses Go's [DefaultTransport](https://pkg.go.dev/net/http#DefaultTransport) values by default).
157
+
158
+
```bash
159
+
kubectl apply -f - <<EOF
160
+
apiVersion: "serving.kserve.io/v1alpha1"
161
+
kind: "InferenceGraph"
162
+
metadata:
163
+
name: "dog-breed-pipeline"
164
+
spec:
165
+
routerTimeouts:
166
+
serverRead: 300
167
+
serverWrite: 300
168
+
serverIdle: 300
169
+
serviceClient: 150
170
+
nodes:
171
+
root:
172
+
routerType: Sequence
173
+
steps:
174
+
- serviceName: cat-dog-classifier
175
+
name: cat_dog_classifier # step name
176
+
- serviceName: dog-breed-classifier
177
+
name: dog_breed_classifier
178
+
data: $request
179
+
condition: "[@this].#(predictions.0==\"dog\")"
180
+
resources:
181
+
requests:
182
+
cpu: 100m
183
+
memory: 256Mi
184
+
limits:
185
+
cpu: 1
186
+
memory: 1Gi
187
+
188
+
EOF
189
+
```
190
+
149
191
## Test the InferenceGraph
150
192
Before testing the `InferenceGraph`, first check if the graph is in the ready state and then get the router url for sending the request.
0 commit comments