[Feature] Regex for website scraping

[NXTler]

Greetings,
I recently started experimenting with this extension and found that there are a lot of characters wasted due to website formatting, like multiple newlines for no apparent reason. I propose the integration of the regex system into the scraping process, so that we can have more information to process (due to increased information density) and better formatting (like removing the extra space before a new paragraph, like some websites have).

[NXTler]

In case someone is searching for a quick and dirty solution - you can replace at const text line 708 in index.js with something like

const rawtext = await extractTextFromHTML(data, 'p'); // Only extract text from <p> tags
const text = rawtext
 .replace(/\n\n\n\n+/g, '\n\n\n') //removes excessive new lines
 .replace(/\s\s+/g, ' ') //removes double spaces
 .replace(/([\u2700-\u27BF]|[\uE000-\uF8FF]|\uD83C[\uDC00-\uDFFF]|\uD83D[\uDC00-\uDFFF]|[\u2011-\u26FF]|\uD83E[\uDD10-\uDDFF])/g, ''); //removes emojis

This cleans up the output a bit, but I'm unsure if you actually get more information with this.

[Cohee1207]

There's already a method that post-processes text in core code. So if you need specific processing, add it there.

https://github.com/SillyTavern/SillyTavern/blob/938c8a9a364919433ac1cb0e6d82710116dabf86/public/scripts/utils.js#L1515

[NXTler]

Thanks, I will experiment a bit and try to get more info into the budget with this. When I have something working I might commit it.

[Cohee1207]

.replace(/\n\n\n\n+/g, '\n\n\n') //removes excessive new lines
.replace(/\s\s+/g, ' ') //removes double spaces

These look good generally.

.replace(/([\u2700-\u27BF]|[\uE000-\uF8FF]|\uD83C[\uDC00-\uDFFF]|\uD83D[\uDC00-\uDFFF]|[\u2011-\u26FF]|\uD83E[\uDD10-\uDDFF])/g, ''); //removes emojis

This is questionable. I think if you're scraping articles about emojis, losing emojis would be detrimental.

[NXTler]

For the last one I just search how to remove emojis in js and haven't spend much time testing it, I'm also unsure if that's the correct way.

[NXTler]

Okay, this is how it looks now:

        const rawtext = await extractTextFromHTML(data, 'p'); // Only extract text from <p> tags

        const text = rawtext
            .replace(/^\n+|\n+$/g, '')                                                // Remove leading and trailing new lines
            .replace(/\n{4,}/g, '\n\n\n')                                             // Remove excessive new lines
            .replace(/(?<!.) +(?=(?:|["'([])(?:\w+ ){2})| +(?!.)/g, '')               // Remove leading and trailing spaces
            .replace(/(?<! )  (?! )/g, ' ')                                           // Remove double spaces
            .replace(/​(?<= \w+) +(?=[.,?!:;])/g, '')                                  // Remove leading spaces from Punctiation
            .replace(/(?<= \w+)[.,!?:;](?=\w+(?: |, ))/g, '$& ')                      // Add missing spaces between sentences
            .replace(/(\p{Emoji_Presentation}|\p{Extended_Pictographic})(?=\1)/g, '') // Remove spammed emojis
            ;

This really cleans up the output without changing the actual information that much. Emojis are still a bit weird, as they sometimes have or not have leading/trailing spaces.

EDIT: fixed some issues with the replacements, should work in more cases now while preserving more information.
EDIT: Further improved leading space detection

[NXTler]

Seems to work correctly with the character budget.

[NXTler]

Okay, after some edits, I would call these expressions good. They differ quite a lot from the ones in postProcessText, so how should go about adding these? Or should I rather create a new function for this?

[Cohee1207]

I'm not sure about a new function, especially around removing emojis. The rest of the regex look fine.

[NXTler]

It doesn't remove emojis completely, it simply collapses multiple into one to handle unnecessary spamming from some websites.

[Cohee1207]

You mean if the same emoji is repeated multiple times? You can also target the emoji with a unicode group instead. This example will match all emojis in a string:

/(\p{Emoji_Presentation}|\p{Extended_Pictographic})/gu

[NXTler]

Yes, that's what I meant. If one emoji is repeated multiple times, it will be reduced to only one instance.
I wasn't aware of that functionality. In that case, the regex would look like this:
.replace(/(\p{Emoji_Presentation}|\p{Extended_Pictographic})(?=\1)/gu, '') // Remove spammed emojis

So, how should I add these to postProcessText? I have some different approaches for some Regexs (see removing double spaces and leading and trailing spaces) to retain more information.

[NXTler]

@Cohee1207 I'm sorry for the ping, as I can see that you are very busy and probably have better more important problems. However, I have some time to spend and would like to complete my issue based on you suggestions.

[Cohee1207]

These should go to postProcessText.