Scam Email Matching Algorithm

[naseemshah]

This issue explores an email matching algorithm designed to combat scam emails. The algorithm analyzes sender details, content, and links to identify fraudulent messages while ensuring legitimate emails are not mistakenly flagged.

[kiranmurali93]

Did some research on this issue and these are the summary:

• Since we are dealing with something which we need to run in the browser, we need a small language model
• Model should be under ~50MB.
• Since we need to assign a score rather than categorising we need a model which can understand the context
• Something which came across is ONNX runtime which I have to look more into that
• Some models that I came across which can fit into our requirements are
  • MiniLM, ELECTRA, and ALBERT

[naseemshah]

This is cool! While LLMs work fine for this usecase, I have some concern about the size of the LLM (since we are aiming to run in browser locally) after fine tuning with the scam emails datasets.

My initial hypothesis is that fine tuning with latest scam emails periodically (which increases the model size) will work till we reach a size of almost ~100MB. From this point onwards we might need to explore having a vector DB + basic RAG pipeline will make sense or not. Downloading almost 200+ MB model each time a new user joins or a new update to the model will require all existing users to download the model as a whole again.

But with my initial research, I think a local vector db that syncs incrementally + LLM is a good idea when we have beyond 100k emails.

[naseemshah]

I just asked ChatGPT to what would be the scenario of fine tuning with 10k emails vs 100k emails. Here it is:

Model	Base Model Size	Fine-Tuning Method	Increase for 10k Emails	Increase for 100k Emails	Notes
GPT-NeoX (Small)	~50 MB	Full Fine-Tuning	~2-5 MB	~5-10 MB	Full fine-tuning updates all weights; size increase depends on learned adjustments.
GPT-2 (Small)	~124 MB	LoRA (Low-Rank Updates)	~1-3 MB	~2-4 MB	LoRA stores small, separate update matrices; size increase is proportional to adapter size.
LLaMA-2 (7B)	~800 MB (quantized)	Prefix-Tuning	<1 MB	<1 MB	Prefix embeddings are task-specific and lightweight; size increase is minimal.
Mistral (7B)	~500 MB (4-bit)**	Adapter-Tuning	~2-4 MB	~4-6 MB	Adapter layers add minimal overhead, making them efficient for lightweight fine-tuning.
BLOOMZ (560M)	~350 MB	Full Fine-Tuning	~10-15 MB	~20-30 MB	Full fine-tuning with large models results in more noticeable size increases.

I think we need to test this first.

[kiranmurali93]

I have a rough design on how to go about it. What we can do is split the problem into sub problems, lets say things we can do with code and we cannot do with code. I came up with a simple design doc. In this first lets take up 3 features which we can extract from an email.

For now I have taken 3 features

• Header
• URLs
• Content

For Header analysis I have just marked as pattern matching but I have to look into it more since I'm sure there would be some other way with which we can analyse email header

For urls its just straight forward, we can prepare a set of black listed urls and then compare. These urls can be stored in the local storage of the respective clients.

Now for the content comes the tricky part where we need some model. The other challenge with content is that we might need to define sub features as well, lets say something like caps count, or common occurring words like "URGENT" etc.

Now based on these results we should be able to calculate a score, this part is also code-able we can give some weights to headers, urls and content and based on that we just calculate percentage.

Final part is just showing it in the UI.

[kiranmurali93]

For header analysis these are few things to check:

• From, reply-to domain, return path email domain should be matching
• These above domains shouldn’t be from the list of blacklisted domains, common blacklisted domains and by user as well.
• SPF, dmarc status should be true
• DKIM signature should be there