Guided Topic Modelling seed word list split?

[noahberhe]

What would be the impact using:

seed_topic_list = [["drug", "cancer doctor"],
                   ["windows drive", "dos file"]]

vs.

seed_topic_list = [["drug", "cancer", "doctor"],
                   ["windows", "drive", "dos", "file"]]

I guess there's no difference as the words for each seed topic would be concatenated anyway?

[MaartenGr]

In BERTopic, it creates a single string for each seeded topic and creates an embedding from that single string. In your case, there would be no difference in the resulting embeddings as the strings get concatenated anyway.

[noahberhe]

Hi @MaartenGr

Just to double back on this, then what is the value of creating a list of lists as seed_topic_list?
Are we not ultimately concatenating them to create one long list of words from which to choose the best fit.

So essentially BERTopic could very easily pick "cancer" and "drive" as the top 2 words representing the topic of that document?

Thanks
Noah

[MaartenGr]

I might have been a bit unclear here. From an embedding perspective, whether you concatenate those strings does not matter:

BERTopic/bertopic/_bertopic.py

Lines 2961 to 2964 in 809414b

	# Create embeddings from the seeded topics
	seed_topic_list = [" ".join(seed_topic) for seed_topic in self.seed_topic_list]
	seed_topic_embeddings = self._extract_embeddings(seed_topic_list, verbose=self.verbose)
	seed_topic_embeddings = np.vstack([seed_topic_embeddings, embeddings.mean(axis=0)])

However, the c-TF-ID representations do get a small nudge from each individual word, so there would be a difference whether you concatenate or not:

BERTopic/bertopic/_bertopic.py

Lines 3150 to 3152 in 809414b

	if self.seed_topic_list:
	seed_topic_list = [seed for seeds in self.seed_topic_list for seed in seeds]
	multiplier = np.array([1.2 if word in seed_topic_list else 1 for word in words])

The visualization in the documentation here might be helpful.

[noahberhe]

I think I get it now, it's not the fact that seed words appear in the same sub-list that groups them into one topic, it's the fact that the seed words themselves are semantically similar to each other and distinct to other seed-words in another sublist.

The list-of-lists is more of a visual cue to help organise them, it is still on us to ensure each sublist represents a distinct, clearly delineated topic through careful choice of words

[MaartenGr]

Indeed, it is a two-step process. First, the seed words are concatenated together to create a single semantic representation which is used to steer document embeddings toward those representations. Second, however, although the list-of-lists is a visual cue for the first step, it is a necessity for the second step. Namely, if they were concatenated together, then it is likely they would not exist in the c-TF-IDF's vocabulary. Instead, each individual word gets a higher value in the topic representations. Therefore, it influences both the clustering process as well as the topic representation process.

[noahberhe]

Ok cool, I'm going to have to process that one a little bit! So I take it that words e.g. a verb like 'remove' (or any other word really) can appear in multiple seed topic sub-lists, albeit in a different context ('context' meaning accompanied by different words in each sub-list)?

Also ... do you have any best practises when picking these seed topics? I previously tried to run Unguided BERTopic and use some of the topics I saw to define some seed topics, but it was a pretty random exercise with no thought given to any rules that are best adhered to, e.g how similar they were, any word overlaps, the structure of each seed topic, the length of each seed topic, etc?

[MaartenGr]

Ok cool, I'm going to have to process that one a little bit! So I take it that words e.g. a verb like 'remove' (or any other word really) can appear in multiple seed topic sub-lists, albeit in a different context ('context' meaning accompanied by different words in each sub-list)?

Indeed! For each topic, they will have a different context but the word itself, remove, will get a higher c-TF-IDF score by definition.

Also ... do you have any best practises when picking these seed topics? I previously tried to run Unguided BERTopic and use some of the topics I saw to define some seed topics, but it was a pretty random exercise with no thought given to any rules that are best adhered to, e.g how similar they were, any word overlaps, the structure of each seed topic, the length of each seed topic, etc?

Understanding the data and domain is vital here. These seed topics only work well if they match some of the latent topics in the data. It is a zero-shot approach that only works well if the seed topics are defined well. In many cases, you already expect a few topics to appear and you can use guided topic modeling to make sure they are extracted.

[noahberhe]

Thank you Maarten. In my dataset, some docs are very short, and others are longer and can contain more than one seed topic.
As a result at the moment, the model sometimes picks the wrong seed topic for each doc.

I guess I would need to list all of the topics per doc (not just one) that are above a given probability threshold?
So calculate probabilities = True and use the list of probs per doc to select for the topics?

[MaartenGr]

That is indeed one way of approaching it. The downside of doing that is that calculating the probabilities, without cuML's HDBSCAN, is rather slow. You resolve this with cuML's HDBSCAN but that generally requires a decent GPU. Another option is to use the .approximate_distribution function in BERTopic. It allows for calculating the distribution of topics in a document, even on a token level.

[noahberhe]

Cool! I will give that a go.
So I guess a starting threshold for whether the tokenwise probability is affirmative, is 0.5?
And I guess the tokenwise probabilities won't necessarily align with the topics that were first assigned to that document by the model?

[MaartenGr]

So I guess a starting threshold for whether the tokenwise probability is affirmative, is 0.5?

The starting threshold depends on whether or not c-TF-IDF or embeddings or used. Personally, I would start with 0.1 and increment/decrease based on visual output.

And I guess the tokenwise probabilities won't necessarily align with the topics that were first assigned to that document by the model?

Correct, it is a different procedure and is unlikely to match exactly.