Consuming too much time for Dynamic Topic Modeling on Google Colab

[rcmoraleshernandez]

Hello Marteen,

I really love BERTopic but I have a question about time consuming for DTM.

I have 70,000 scientific articles (**Just ** Title+Abstract) from 2015 to 2021 (10,000 per year). Around 160 words per paper (mean) after stopwords.

I force to just 15 topics (nr_topics=16) and 20 words per topic. The time/date data is just the year on each article.

I use Google Pro+, but for a DTM run consumes around 15-20 hours.
The last one was for almost 20 hours:
7it [19:55:42, 10248.99s/it]

Is that a correct time consuming? Is there a way to reduce time consuming to save compute units =save $$ in Google Colab Pro+? My big issue is because I want to increase articles to 140,000.

Thank you very much in advance,
Roberto Carlos

[MaartenGr]

Generally, it should not run for that long of a time. For me, it typically runs within a couple of minutes. Could you share your full code? Also, did you make sure to set nr_bins?

[rcmoraleshernandez]

Hi Marteen,
For new version 0.14 I had to create representation_model because 'diversity=' doesn't work
representation_model = MaximalMarginalRelevance(diversity=0.8)
I still do not run a model with 0.14 because I want to be sure to use a tip to save time for .topic_over_time

Sorry for the mistake, BUT the model I used for topic over time was used with BERTopic verison 0.12):

topic_model = BERTopic(embedding_model="all-MiniLM-L6-v2",diversity=0.8,language="english", calculate_probabilities=True, n_gram_range=(1,3), verbose=True,top_n_words=20, nr_topics=15)

'===== CODE used for topic over time who spent 19 hours (executed in Nov-2022) ===
'## TODO Add your file path ###
file_path = '/content/drive/MyDrive/Colab Notebooks/DATASETS/'
file_name = 'dataset_2015-2021_70mil_15nov2022_SDG_binario_para_BERT.csv'

'## Load the articles
df = pd.read_csv(file_path + file_name)

print(df.shape)

'## The column 'tit_abs' is the one with title+abstract text
'## Example tit_abs text (normalized and eliminated stopwordds:
'## 'human rabies neglected preventable disease nigeria bac round objectives...'

pip install bertopic
from enum import auto
from bertopic import BERTopic

doc = df['tit_abs'].tolist()

topic_model = BERTopic(embedding_model="all-MiniLM-L6-v2",diversity=0.8,language="english", calculate_probabilities=True, n_gram_range=(1,3), verbose=True,top_n_words=20, nr_topics=15)

topics, probs = topic_model.fit_transform(doc)

freq = topic_model.get_topic_info()
freq

'## Save model
topic_model.save("/content/drive/MyDrive/Colab Notebooks//BERTOPIC MODELS/2015-2021_15nov2022_15topics_ng_3_DIV0_8")

'## FOR TOPIC OVER TIME
years = df.year.to_list()
years

topics_over_time = topic_model.topics_over_time(doc,years)

7it [19:55:42, 10248.99s/it]

topic_model.visualize_topics_over_time(topics_over_time)

'##Save html topicover time
fig3= topic_model.visualize_topics_over_time(topics_over_time)
fig3.write_html("/content/drive/MyDrive/Colab Notebooks/2015-2021 BERTOPIC 15nov22/2015-2021_15topic_DIV08_OVER_TIME_15nov.html")
fig3
'##======================

I hope this help. Thanks, I really, really appreciate your time and support.
Roberto Carlos

[MaartenGr]

It might be a combination of using diversity and a large n-gram range. I would advise starting off without using diversity and using the default n-gram range and then adding that to see if it slows things down. If so, then it might be worthwhile to use a CountVectorizer for the n-gram range and setting min_df a bit higher there. This allows for a much smaller vocabulary and much faster calculations. Also, I would advise double-checking that len(set(years)) is indeed 7. If that does not work perhaps setting nr_bins here would help.

[rcmoraleshernandez]

I appreciate your suggestions and I will apply them to see the results and I will inform you. Greetings

[rcmoraleshernandez]

Hi Maarten,
I took your advices. Building model without using diversity and using the default n-gram .... topics_over_time made the calculation in less than 1 minute!! Excellent.
Then, I used Countvectorizer for changing ngram_range parameter and MMR for diversity parameter. And after that, update modelo with .update_topics
Later applied DTM and works taking seconds or minutes. Great.

Do you have any suggestions to get 'clear' topic frequency development for text like abstracts? I use 70,000-130,000 articles for 7years?. I'm making tests with different configurations (outlier reduction), but I'm going to use global_tuning and evolutionary_tuning . But I appreciate any suggestions.
Again, thanks for your time and feedback.
Roberto Carlos

[MaartenGr]

I believe the evolutionary_tuning might be a good parameter to try out as it focuses a bit more on the development of topics over time whereas global_tuning is often used to make the topic representations a bit more readable for timestamps and topics where there are too few documents.

[rcmoraleshernandez]

Hi Maarten,
Now that I use Countvectorizer to apply n-gram (1,3) or (1,2), when I apply Topic Over Time, the dataset with 130,000 or 70,000 scientific papers, Topics stay very, very stable over time. All topics!!
Even using evolutionary_tuning =True, and global_tuning =False, the topic evolution are too stable I believe.
I really do not believe that all topics should be so stables.
Even reducing outliers, the results is the same.
Thanks for your support and time.
Roberto

##main Code

Create your representation model

representation_model = MaximalMarginalRelevance(diversity=1)
vectorizer_model = CountVectorizer(stop_words="english", ngram_range=(1, 2), min_df=10)
topic_model = BERTopic(top_n_words = 20, representation_model=representation_model, vectorizer_model=vectorizer_model, language="english", calculate_probabilities=True, verbose=False)

Reduce outliers

new_topics = topic_model.reduce_outliers(doc, topics)
topic_model.update_topics(doc, topics=new_topics)
##DTM
topics_over_time = topic_model.topics_over_time(doc, years, global_tuning=False, evolution_tuning=True)

[MaartenGr]

The stability that you refer to is the slope of the line, right? In that case, it is important to note that evolutionary_tuning and global_tuning do not impact the slope at all as they merely change the representation of a topic at timestep t. Moreover, in general, the CountVectorizer also does not change the counts, and therefore the slope, of the topics. It seems that this is a result of your clustering.