Generate PO files only for components that contain some localized code #2024
Comments
|
Your usecase is not something that usually Lingui users do. Usually catalogs created for whole app or for the slice. And path's to this catalogs used somewhere in the runtime code, so creating catalog even if they empty is expected behaviour. Regarding your changes, i'm not sure that the value of this feature would be worth effort of having and maintaing this in the codebase. How you are going to load this catalogs after all? Have a loading code in every component? Maybe it's better/easier for you to use lingui api and write your own extractor for your specific case. |
I assume this relates to "ideally in the same directory as the component" part, right? I am discussing that part separately here #2024 assuming that the build process could collect the catalogs recursively from the whole app and the result linked from root of the app in a way similar as outlined here https://lingui.dev/ref/conf#catalogsmergepath Need to say I don't yet fully understand the life cycle of it. Anyway, this issue is mainly about the empty files. I am interested in not generating the empty files even in the typical scenarios described in the examples https://lingui.dev/ref/conf#examples The thing is that our app consists of:
So overall about 80% of out PO files are "empty" which doesn't feel right... And we don't want to have everything in a single PO file because it would be extremely complicated to translate just parts of it related to specific component with the AI when a part of our codebase changes. |
|
+1 for what @hejtmii is talking about, I tried having one big catalog per language ( @timofei-iatsenko How is this problem solved in any kind of repositories bigger than 1-man-army? Extracting things to one big file just doesn't work if you're working with literally anyone other than yourself. I strongly agree with @hejtmii, as even in a POC repository where I have only one translated file I already generated 14+ I'll be happy to work with @hejtmii on this one just so we could solve the issue; a separate extractor, or some options to it, could be the solution, but I'm not super keen on it yet. On the other hand, the default extractor has lots of issues with globs and it is not entering and reproducing paths with nested directories correctly, so maybe the proper way to solve it is actually a custom extractor and documenting it? 🤔 |
|
@Palid Let's look at each point separately
In my opinion, that was a huge bad decision by original Lingui authors to implement 2 actions in one command. That exactly what Lingui supports this flow with Another option could be disabling line numbers or source references completely using Po formatter settings.
Could you share reproductions so we can work on them. I hear about that for the first time. We are also opened to contributors and happily accept PRs. |
|
@timofei-iatsenko Gladly can work on that, considering I have two projects that could use this. I'll provide a reproduction repository for all those separate problems! I feel like there's a couple of separate issues, related to the tool's legacy (even though it's not that old). Seems that docs suggest a default solution that will result in tons of conflicts, but if you try to go other way, it creates some other issues with the tooling that makes some of the features of LingUI no longer available, like loaders. I'll try to list problems below:
Considering all of above, maybe the easiest way to at least remove some of the troubles would be to define development and production steps, and figure out where can we simplify&improve the tooling? Having so many different abilities to extract translations, while still not being able to configure it the way you want (e.g. generating Before getting to the reproduction repository, I can share the tree and lingui config where this nesting is already a problem:
Running Most of the files in here are entirely empty, other than the headers, which could make it entirely skippable for extraction, though it still generates them (I guess that's the problem @hejtmii mentioned). This amount of files has been generated using React Native Reusables initial template and it already created so many empty files, that aren't even properly nested (which is currently impossible with the extraction mechanism). The even worse thing is that you can't really use |
Have you had the chance to try this setup, or are you just speculating about what might happen? The @lingui/loader is designed to work with templates right away. It automatically merges translations, with a fallback to the messages from the template if message is not presented in the translation catalog. You don't need anything from what you described.
I, honestly, never have a need for that, so never used.
Yes, suggested approach would be huge catalog per language per entry point. You don't need to add catalogs to the gitignore. You need to add to git ignore only template. Furthermore, you also don't need to extract and commit on every commit. Commit your catalogs only when the translation changed, not on every file change.
You don't need.
You also don't need it. Prerequisite:
"build": "lingui extract-template && vite build",
The Flow
Hope that helps. Do you translate in feature branches or only when feature is merged to a |
I actually did try to import Attaching the code example below, slightly modified for clarity reasons. This import "server-only";
import { I18n, MessageDescriptor, setupI18n } from "@lingui/core";
import { msg } from "@lingui/macro";
import { setI18n } from "@lingui/react/server";
import linguiConfig from "../../../lingui.config";
export type Lang = (typeof linguiConfig.locales)[number];
export const languages: Record<Lang, MessageDescriptor> = {
en: msg`English`,
no: msg`Norwegian`,
};
const translations = require("src/i18n/prod-messages");
// Code for `translations` below:
/**
*
* if (process.env.NODE_ENV === "production" || process.env.TEST_ENV === "test") {
* const en = require("../locales/en/messages.js");
* const no = require("../locales/no/messages.js");
* module.exports = {
* en,
* no,
* };
* }
*/
type MessagesFile = Record<string, string>;
export async function loadLinguiMessages(lang: string): Promise<MessagesFile> {
if (
process.env.NODE_ENV === "development" &&
process.env.TEST_ENV !== "test"
) {
const msgFile = await import(`src/locales/${lang}/messages.po`);
return {
[lang]: msgFile.messages,
};
} else {
return {
[lang]: translations[lang].messages,
};
}
}
const { locales } = linguiConfig;
// optionally use a stricter union type
type SupportedLocales = string;
type AllI18nInstances = { [K in SupportedLocales]: I18n };
let catalogs: MessagesFile[] = [];
let allMessages: MessagesFile;
let hasInitializedCatalogs = false;
let allI18nInstances: AllI18nInstances = {};
async function getAllInstances(): Promise<AllI18nInstances> {
if (!hasInitializedCatalogs) {
const messages = await Promise.all(locales.map(loadLinguiMessages));
catalogs = messages;
allMessages = catalogs.reduce((acc, oneCatalog) => {
return { ...acc, ...oneCatalog };
}, {});
allI18nInstances = locales.reduce((acc, locale) => {
const messages = allMessages[locale] ?? {};
const i18n = setupI18n({
locale,
messages: { [locale]: messages } as any,
});
return { ...acc, [locale]: i18n };
}, {});
hasInitializedCatalogs = true;
}
return Promise.resolve(allI18nInstances);
}
export async function getI18nInstance(locale: Lang) {
const allI18nInstances = await getAllInstances();
return allI18nInstances[locale];
}
export async function getI18nInstanceWithLocale(locale: Lang) {
const instance = await getI18nInstance(locale);
setI18n(instance);
return instance;
}And for the loading I'm just using a webpack loader config: /* ... */
webpack: (config) => {
config.module.rules.push({
test: /\.po$/i,
loader: "@lingui/loader",
});
return config;
},
/* ... */
I think we're talking about a few different problems here. The final issue is loading those multiple catalogs in development - lingui doesn't really provide any way to do it well, as you'd have to manually define the imports in every single component, which kind of ruins the idea of good developer experience and ease of use. It'd be perfect if the loader could understand lingui config and deliver the translations based on default or defined priority (e.g Your suggestion will be good enough for this particular problem as long as I use a dedicated service for the translations and never change the To sum it up, which one would you prefer?
I'd very much prefer the first choice, even though it's going to make maintaining it definitely harder. I'm willing to take over the development for this myself, as this feature would be really beneficial for my $DAYJOB stuff. |
|
Pinging @timofei-iatsenko as you might have not noticed the wall of text above, I'd love to help on that in addition to the turbopack PR. 😄 We can have a chat on something like matrix if you'd prefer, [email protected] if you are able to chat there. |
|
@Palid hey, sorry for the delay. I found a time to continue discussing the issue. Firstly, thanks for rising this concern and sharing the feedback. Second, I'm really happy to improve the flow in this regard. What I don't want is to implement something half-way done, or implement something that will lead users into the dead end (like it is now with For the issue described in this thread we can add an option for the CLI "--omit-empty/--skip-empty/--no-emit-empty", which is planned to be used together with catalog per file approach (say we will fix glob issue). But what next? How to load them into application? The loader's behavior is already weird enough. Usually we specify the file we want to load. And we're expecting that this file would somehow preprocessed by the loader. If we want to create a loader which will load many files, crawling the file system, we should specify some virtual path? import('virtual:lingui-catalog:en')Which would load and merge all catalogs per file under the hood. This is on top of my head, quite naive approach, though. I'm afraid that this is very far from a normal use case for the webpack (and it's derivatives such as rspack or turbopack) and potentially could lead to incompatibilities. With Vite it should be fine, because Vite is supporting "virtual" modules out of the box. Maybe you had a better idea?
It doesn't fallback into the whole file. You need to understand a bit of the history to understand that decisions. Historically, the loader was used as a shortcut or alias to just a trigger the same logic "lingui compile" is using. Loader isn't pure. It compiles the full catalog as normal That's why, if you will try to load a catalog file which is not in the lingui configuration, you receive an error even though the file exists in the filesystem. Other things you need to consider, is that lingui is removing original messages from the source code. So there is nothing to fallback if the message key is not presented in the catalog, and you will see ugly message id instead. That's why it's very important to have up-to-date catalogs before application build. Even if these catalogs would not have a translation, there should be an empty POT actually resolving this problem. You create POT file right before the actual production build, so you have it always up to date and then POT is used as a backbone to create a language catalogs. Unfortunately there is nothing I can do here better, because it's a react ecosystem, there are plenty of different bundlers, different workflows and combinations and so on. Let's contact me in the discord, i think it could be faster to iterate there and we can come up on some good solution. Thanks! |
Is your feature request related to a problem? Please describe.
Our codebase is split into a large number of small files. Many of them are just app logic / Redux reducers / thunks etc.
We prefer the options to store localizations for each component separately using the
{name}macro. (ideally in the same directory as the component so we can let AI help us with it and have it coupled with a nearby source code to provide extra context - I will file a separate feature request for that...)But it turns out that
extractgenerates "empty" PO files even for code that doesn't need them, which kind of spams our repo.Describe proposed solution
Do not generate PO files for source files without localizations
Describe alternatives you've considered
Make it configurable, keeping current behavior as default so that a breaking change is not introduced?
Additional context
Please let me know if a PR for this could be accepted or if there are some important internal reason why you decided to generate PO files even for components without localizations.
If the idea is passable, I could create a PR for that.
The text was updated successfully, but these errors were encountered: