Standardized approaches to automating hyperparameter-tuning?

[MogwaiMomo]

First: Loving the latest (very modular) version of Bertopic (v0.14) - I'm learning so much playing around with each "lego block" :)

Second: I'm now trying to figure out the best (or at least most repeatable and formalized) method for tuning all the hyperparameters that make up the bertopic 'stack' to get the highest coherence score possible ... but public threads on the subject seem to be pretty scarce.

@MaartenGr I've seen you recommend using OCTIS for this, and I'm currently trying to figure out how to go about doing so. Would you say that Bertopic v0.14 is written in a way that's compatible with OCTIS? For example, the OCTIS documentation has a specific section on "Implementing Your Own Model" that specifically says:

"[OCTIS] Models inherit from the class AbstractModel defined in octis/models/model.py . To build your own model, your class must override the train_model(self, dataset, hyperparameters) method which always requires at least a Dataset object and a Dictionary of hyperparameters as input and should return a dictionary with the output of the model as output."

Would you say the BERTopic() class meets the above requirements "out of the box"? Is there anything specific that needs to be done/changed for bertopic models to be able to be plugged into an OCTIS-based parameter-tuning pipeline?

As always, thank you in advance!

Momoko

P.S. I will keep trying to answer this question on my own, but any clues or specifics on "how to use OCTIS to evaluate/optimize bertopic models" would be hugely appreciated!

[MaartenGr]

First: Loving the latest (very modular) version of Bertopic (v0.14) - I'm learning so much playing around with each "lego block" :)

Thank you for your kind words! Glad to hear that the "lego block" approach is appreciated.

Second: I'm now trying to figure out the best (or at least most repeatable and formalized) method for tuning all the hyperparameters that make up the bertopic 'stack' to get the highest coherence score possible ... but public threads on the subject seem to be pretty scarce.

As highlighted, there are two parts in your question that I think are important here, namely parameter tuning combined with an evaluation metric, namely coherence. First, parameter tuning will remain important regardless of the evaluation metric as out-of-the-box solutions seldom work perfectly with every single use case. In the case of BERTopic, and from my personal view, I believe parameter tuning in BERTopic should be done first with human evaluation. I believe it is more important to answer questions like "Does the number of topics resonate with domain experts?" and "Is my model in line with its intended use case?" than it does maximizing coherence. This brings me to the second component, namely coherence scores and evaluation metrics in general. The difficulty with evaluation metrics in topic modeling is that they often are not fully subjective. If I decide on a topic to be cohesive, many others might disagree with it. I have seen it time and time again where even domain experts disagree on things like topic cohesiveness, diversity, number of topics, topic distributions, etc. As a result, I would typically advise refraining from tuning the hyperparameters to optimize coherence as there is a good chance of overfitting to an imperfect metric.

Having said that, it does not mean that you should forego any evaluation metric. The reason for me often advising OCTIS is that it implements a wide variety of evaluation metrics, which together, give a much better overview of a model's quality than a single metric.

Would you say the BERTopic() class meets the above requirements "out of the box"? Is there anything specific that needs to be done/changed for bertopic models to be able to be plugged into an OCTIS-based parameter-tuning pipeline?

BERTopic does not meet the requirements out of the box. The reason is mostly explained above but it all boils down to each use case being different and therefore needing a different set of evaluation metrics, custom or not.

Having said that, you can find some code for using OCTIS with BERTopic here although it was not meant to be a framework for doing so at is was merely part of the paper. Instead of making sure BERTopic follows the suggested class, it creates an output used by their scoring functions as mentioned here.

[MogwaiMomo]

Hi @MaartenGr - as always, thank you for your prompt reply!

I've seen you mention the importance of human evaluation before selecting models in other threads. I fully understand. The challenge that I'm trying (perhaps vainly or ill-advisedly) to figure out is building a UI whereby non-technical/non-coding users can use BERTopic to cluster and sort documents by theme* and get decent results on an initial run without digging into the code or model's parameters, as they wouldn't know how to do this (or have the time to learn).

That said: I suppose what I'm really after isn't a way to omit or skip human evaluation of results at all, but simply build something whereby the user can, with a single click/command:

1. Have bertopic go through multiple runs (with a range of values for each hyperparameter already set as a starting point)
2. Sort the models by some kind of scoring metric (but, as you say, maybe give the end user the ability to change the sort order to different scoring metrics, rather than just only focus on coherence. I'm just thinking off the top of my head here ...)
3. Allow the end user to inspect the "top" 5 or 10 models visually, and then choose the one whose results feel "right" to them.

So ... like doing a grid search, I suppose, but with the added feature of evaluating by multiple scoring metrics, rather than just one? Hence your recommendation for using OCTIS, clearly!

In any case, thank you for pointing me to some demo code on combining bertopic with OCTIS, as well as your original paper - I'll review both carefully and keep thinking on this.

(Also thank you for advising against blindly sticking to coherence as a single score. This may explain some wonky results I've gotten in the past)

** For documents in the 1000-5000 range, like a table of survey responses. As low as 250-500 docs would be ideal but possibly not feasible.

[MaartenGr]

No problem, glad I can be of help!

I've seen you mention the importance of human evaluation before selecting models in other threads. I fully understand. The challenge that I'm trying (perhaps vainly or ill-advisedly) to figure out is building a UI whereby non-technical/non-coding users can use BERTopic to cluster and sort documents by theme* and get decent results on an initial run without digging into the code or model's parameters, as they wouldn't know how to do this (or have the time to learn).

That sounds like an interesting use case. The users of BERTopic are quite diverse concerning their backgrounds and I think many would be happy with an easy-to-use interface abstracting many of the power options away.

So ... like doing a grid search, I suppose, but with the added feature of evaluating by multiple scoring metrics, rather than just one? Hence your recommendation for using OCTIS, clearly!

The main difficulty with doing something like this would I think be the evaluation metrics. Since the focus is on user experience, it might be worthwhile to include some evaluation metrics that are not directly related to the model's performance but more to user experience. For example, users might prefer to have dozens of topics instead of hundreds or thousands of topics. So using the number of topics but also their sizes as evaluation metrics might be closer to what users would experience as ideal.

This would also allow you to restrict the grid search to things that might be of interest to users, like the number of topics generated, their sizes, coherence, but also the level of abstraction, etc. All in all, it still needs to be performant and grid searching almost always takes quite some time to run. So careful selection of parameters in important here.

** For documents in the 1000-5000 range, like a table of survey responses. As low as 250-500 docs would be ideal but possibly not feasible.

This might be possible if you are using an algorithm that clusters all documents, like k-Means. Combine that with some of the newer representation models, like KeyBERTInspired or OpenAI, and you have got nice labels for small sets of documents.