Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to HttpClient auto-configuration #115

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Improvements to HttpClient auto-configuration #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .sdkmanrc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
# See https://sdkman.io/usage#config
# A summary is to add the following to ~/.sdkman/etc/config
# sdkman_auto_env=true
java=17.0.14-tem
java=21-tem
6 changes: 6 additions & 0 deletions langchain4j-ollama-spring-boot-starter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@
</exclusions>
</dependency>

<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-spring-boot-autoconfigure</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(note for myself): need to do the same for OpenAI SB starter

<version>${project.version}</version>
</dependency>

<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-http-client-spring-restclient</artifactId>
Expand Down
142 changes: 17 additions & 125 deletions ...4j-ollama-spring-boot-starter/src/main/java/dev/langchain4j/ollama/spring/AutoConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,48 +1,31 @@
package dev.langchain4j.ollama.spring;

import dev.langchain4j.autoconfigure.http.HttpClientAutoConfiguration;
import dev.langchain4j.http.client.HttpClientBuilder;
import dev.langchain4j.http.client.spring.restclient.SpringRestClient;
import dev.langchain4j.model.chat.listener.ChatModelListener;
import dev.langchain4j.model.ollama.*;
import dev.langchain4j.model.ollama.OllamaChatModel;
import dev.langchain4j.model.ollama.OllamaEmbeddingModel;
import dev.langchain4j.model.ollama.OllamaLanguageModel;
import dev.langchain4j.model.ollama.OllamaStreamingChatModel;
import dev.langchain4j.model.ollama.OllamaStreamingLanguageModel;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.autoconfigure.web.client.RestClientAutoConfiguration;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.core.task.AsyncTaskExecutor;
import org.springframework.core.task.support.ContextPropagatingTaskDecorator;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.web.client.RestClient;

import static dev.langchain4j.ollama.spring.Properties.PREFIX;

@AutoConfiguration(after = RestClientAutoConfiguration.class)
@AutoConfiguration(after = HttpClientAutoConfiguration.class)
@EnableConfigurationProperties(Properties.class)
public class AutoConfig {

private static final String TASK_EXECUTOR_THREAD_NAME_PREFIX = "LangChain4j-Ollama-";

private static final String CHAT_MODEL_HTTP_CLIENT_BUILDER = "ollamaChatModelHttpClientBuilder";

private static final String STREAMING_CHAT_MODEL_HTTP_CLIENT_BUILDER = "ollamaStreamingChatModelHttpClientBuilder";
private static final String STREAMING_CHAT_MODEL_TASK_EXECUTOR = "ollamaStreamingChatModelTaskExecutor";

private static final String LANGUAGE_MODEL_HTTP_CLIENT_BUILDER = "ollamaLanguageModelHttpClientBuilder";

private static final String STREAMING_LANGUAGE_MODEL_HTTP_CLIENT_BUILDER = "ollamaStreamingLanguageModelHttpClientBuilder";
private static final String STREAMING_LANGUAGE_MODEL_TASK_EXECUTOR = "ollamaStreamingLanguageModelTaskExecutor";

private static final String EMBEDDING_MODEL_HTTP_CLIENT_BUILDER = "ollamaEmbeddingModelHttpClientBuilder";

@Bean
@ConditionalOnMissingBean
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This annotation is important because it allows developers to define their own OllamaChatModel bean if they need additional customisations.

Same for the other model beans in this class.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I guess ollamaStreamingLanguageModel should also be annotated with @ConditionalOnMissingBean?

@ConditionalOnProperty(PREFIX + ".chat-model.base-url")
OllamaChatModel ollamaChatModel(
@Qualifier(CHAT_MODEL_HTTP_CLIENT_BUILDER) HttpClientBuilder httpClientBuilder,
HttpClientBuilder httpClientBuilder,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly that there is currently no way to customize http client and/or executor per model type (e.g., use different configuration for non-streaming and streaming model) or provider (e.g., use different configuration for different LLM providers)?

Properties properties,
ObjectProvider<ChatModelListener> listeners
) {
Expand All @@ -69,20 +52,11 @@ OllamaChatModel ollamaChatModel(
.build();
}

@Bean(CHAT_MODEL_HTTP_CLIENT_BUILDER)
@ConditionalOnProperty(PREFIX + ".chat-model.base-url")
@ConditionalOnMissingBean(name = CHAT_MODEL_HTTP_CLIENT_BUILDER)
HttpClientBuilder ollamaChatModelHttpClientBuilder(ObjectProvider<RestClient.Builder> restClientBuilder) {
return SpringRestClient.builder()
.restClientBuilder(restClientBuilder.getIfAvailable(RestClient::builder))
// executor is not needed for no-streaming OllamaChatModel
.createDefaultStreamingRequestExecutor(false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, with current configuration, task executor will always be created, even if it is not required (e.g., only non-streaming OllamaChatModel is used in the application)?

}

@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(PREFIX + ".streaming-chat-model.base-url")
OllamaStreamingChatModel ollamaStreamingChatModel(
@Qualifier(STREAMING_CHAT_MODEL_HTTP_CLIENT_BUILDER) HttpClientBuilder httpClientBuilder,
HttpClientBuilder httpClientBuilder,
Properties properties,
ObjectProvider<ChatModelListener> listeners
) {
Expand All @@ -108,42 +82,11 @@ OllamaStreamingChatModel ollamaStreamingChatModel(
.build();
}

@Bean(STREAMING_CHAT_MODEL_HTTP_CLIENT_BUILDER)
@ConditionalOnProperty(PREFIX + ".streaming-chat-model.base-url")
@ConditionalOnMissingBean(name = STREAMING_CHAT_MODEL_HTTP_CLIENT_BUILDER)
HttpClientBuilder ollamaStreamingChatModelHttpClientBuilder(
ObjectProvider<RestClient.Builder> restClientBuilder,
@Qualifier(STREAMING_CHAT_MODEL_TASK_EXECUTOR) AsyncTaskExecutor executor) {
return SpringRestClient.builder()
.restClientBuilder(restClientBuilder.getIfAvailable(RestClient::builder))
.streamingRequestExecutor(executor);
}

@Bean(STREAMING_CHAT_MODEL_TASK_EXECUTOR)
@ConditionalOnProperty(PREFIX + ".streaming-chat-model.base-url")
@ConditionalOnMissingBean(name = STREAMING_CHAT_MODEL_TASK_EXECUTOR)
@ConditionalOnClass(name = "io.micrometer.context.ContextSnapshotFactory")
AsyncTaskExecutor ollamaStreamingChatModelTaskExecutorWithContextPropagation() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setThreadNamePrefix(TASK_EXECUTOR_THREAD_NAME_PREFIX);
taskExecutor.setTaskDecorator(new ContextPropagatingTaskDecorator());
return taskExecutor;
}

@Bean(STREAMING_CHAT_MODEL_TASK_EXECUTOR)
@ConditionalOnProperty(PREFIX + ".streaming-chat-model.base-url")
@ConditionalOnMissingBean(name = STREAMING_CHAT_MODEL_TASK_EXECUTOR)
@ConditionalOnMissingClass("io.micrometer.context.ContextSnapshotFactory")
AsyncTaskExecutor ollamaStreamingChatModelTaskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setThreadNamePrefix(TASK_EXECUTOR_THREAD_NAME_PREFIX);
return taskExecutor;
}

@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(PREFIX + ".language-model.base-url")
OllamaLanguageModel ollamaLanguageModel(
@Qualifier(LANGUAGE_MODEL_HTTP_CLIENT_BUILDER) HttpClientBuilder httpClientBuilder,
HttpClientBuilder httpClientBuilder,
Properties properties
) {
LanguageModelProperties languageModelProperties = properties.getLanguageModel();
Expand All @@ -167,20 +110,10 @@ OllamaLanguageModel ollamaLanguageModel(
.build();
}

@Bean(LANGUAGE_MODEL_HTTP_CLIENT_BUILDER)
@ConditionalOnProperty(PREFIX + ".language-model.base-url")
@ConditionalOnMissingBean(name = LANGUAGE_MODEL_HTTP_CLIENT_BUILDER)
HttpClientBuilder ollamaLanguageModelHttpClientBuilder(ObjectProvider<RestClient.Builder> restClientBuilder) {
return SpringRestClient.builder()
.restClientBuilder(restClientBuilder.getIfAvailable(RestClient::builder))
// executor is not needed for no-streaming OllamaLanguageModel
.createDefaultStreamingRequestExecutor(false);
}

@Bean
@ConditionalOnProperty(PREFIX + ".streaming-language-model.base-url")
OllamaStreamingLanguageModel ollamaStreamingLanguageModel(
@Qualifier(STREAMING_LANGUAGE_MODEL_HTTP_CLIENT_BUILDER) HttpClientBuilder httpClientBuilder,
HttpClientBuilder httpClientBuilder,
Properties properties
) {
LanguageModelProperties languageModelProperties = properties.getStreamingLanguageModel();
Expand All @@ -203,43 +136,11 @@ OllamaStreamingLanguageModel ollamaStreamingLanguageModel(
.build();
}

@Bean(STREAMING_LANGUAGE_MODEL_HTTP_CLIENT_BUILDER)
@ConditionalOnProperty(PREFIX + ".streaming-language-model.base-url")
@ConditionalOnMissingBean(name = STREAMING_LANGUAGE_MODEL_HTTP_CLIENT_BUILDER)
HttpClientBuilder ollamaStreamingLanguageModelHttpClientBuilder(
@Qualifier(STREAMING_LANGUAGE_MODEL_TASK_EXECUTOR) AsyncTaskExecutor executor,
ObjectProvider<RestClient.Builder> restClientBuilder
) {
return SpringRestClient.builder()
.restClientBuilder(restClientBuilder.getIfAvailable(RestClient::builder))
.streamingRequestExecutor(executor);
}

@Bean(STREAMING_LANGUAGE_MODEL_TASK_EXECUTOR)
@ConditionalOnProperty(PREFIX + ".streaming-language-model.base-url")
@ConditionalOnMissingBean(name = STREAMING_LANGUAGE_MODEL_TASK_EXECUTOR)
@ConditionalOnClass(name = "io.micrometer.context.ContextSnapshotFactory")
AsyncTaskExecutor ollamaStreamingLanguageModelTaskExecutorWithContextPropagation() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setThreadNamePrefix(TASK_EXECUTOR_THREAD_NAME_PREFIX);
taskExecutor.setTaskDecorator(new ContextPropagatingTaskDecorator());
return taskExecutor;
}

@Bean(STREAMING_LANGUAGE_MODEL_TASK_EXECUTOR)
@ConditionalOnProperty(PREFIX + ".streaming-language-model.base-url")
@ConditionalOnMissingBean(name = STREAMING_LANGUAGE_MODEL_TASK_EXECUTOR)
@ConditionalOnMissingClass("io.micrometer.context.ContextSnapshotFactory")
AsyncTaskExecutor ollamaStreamingLanguageModelTaskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setThreadNamePrefix(TASK_EXECUTOR_THREAD_NAME_PREFIX);
return taskExecutor;
}

@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(PREFIX + ".embedding-model.base-url")
OllamaEmbeddingModel ollamaEmbeddingModel(
@Qualifier(EMBEDDING_MODEL_HTTP_CLIENT_BUILDER) HttpClientBuilder httpClientBuilder,
HttpClientBuilder httpClientBuilder,
Properties properties
) {
EmbeddingModelProperties embeddingModelProperties = properties.getEmbeddingModel();
Expand All @@ -255,13 +156,4 @@ OllamaEmbeddingModel ollamaEmbeddingModel(
.build();
}

@Bean(EMBEDDING_MODEL_HTTP_CLIENT_BUILDER)
@ConditionalOnProperty(PREFIX + ".embedding-model.base-url")
@ConditionalOnMissingBean(name = EMBEDDING_MODEL_HTTP_CLIENT_BUILDER)
HttpClientBuilder ollamaEmbeddingModelHttpClientBuilder(ObjectProvider<RestClient.Builder> restClientBuilder) {
return SpringRestClient.builder()
.restClientBuilder(restClientBuilder.getIfAvailable(RestClient::builder))
// executor is not needed for no-streaming OllamaEmbeddingModel
.createDefaultStreamingRequestExecutor(false);
}
}
}
5 changes: 3 additions & 2 deletions ...-ollama-spring-boot-starter/src/test/java/dev/langchain4j/ollama/spring/AutoConfigIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package dev.langchain4j.ollama.spring;

import dev.langchain4j.autoconfigure.http.HttpClientAutoConfiguration;
import dev.langchain4j.model.StreamingResponseHandler;
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.chat.StreamingChatLanguageModel;
Expand Down Expand Up @@ -65,7 +66,7 @@ private static String baseUrl() {
}

ApplicationContextRunner contextRunner = new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(AutoConfig.class));
.withConfiguration(AutoConfigurations.of(HttpClientAutoConfiguration.class, AutoConfig.class));

@Test
void should_provide_chat_model() {
Expand Down Expand Up @@ -189,7 +190,7 @@ void should_provide_streaming_chat_model_with_custom_task_executor() {
ThreadPoolTaskExecutor customExecutor = spy(new ThreadPoolTaskExecutor());

contextRunner
.withBean("ollamaStreamingChatModelTaskExecutor", ThreadPoolTaskExecutor.class, () -> customExecutor)
.withBean("httpClientTaskExecutor", ThreadPoolTaskExecutor.class, () -> customExecutor)
.withPropertyValues(
"langchain4j.ollama.streaming-chat-model.base-url=" + baseUrl(),
"langchain4j.ollama.streaming-chat-model.model-name=" + MODEL_NAME,
Expand Down
60 changes: 60 additions & 0 deletions langchain4j-spring-boot-autoconfigure/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-spring</artifactId>
<version>1.0.0-beta2-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>langchain4j-spring-boot-autoconfigure</artifactId>
<name>LangChain4j :: Spring Boot AutoConfiguration</name>

<dependencies>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>

<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-http-client</artifactId>
<optional>true</optional>
</dependency>

<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-http-client-spring-restclient</artifactId>
<version>${project.version}</version>
<optional>true</optional>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure-processor</artifactId>
<optional>true</optional>
</dependency>

<!-- needed to generate automatic metadata about available config properties -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>

<!-- test dependencies -->

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>

</dependencies>

</project>
Loading