Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

convert-from json/lingui doesnt work as expected #1641

Closed
1 of 3 tasks
sirpy opened this issue May 7, 2023 · 15 comments
Closed
1 of 3 tasks

convert-from json/lingui doesnt work as expected #1641

sirpy opened this issue May 7, 2023 · 15 comments

Comments

@sirpy
Copy link

sirpy commented May 7, 2023

Describe the bug
Having translations in json format and a new lingui config set to "po" format
Running extract --convert-from minimal doesnt copy the already translated strings
Converting from "minimal" to "lingui" works partialy, the translated strings is converted to an array of chars and there's a field "obsolete:true", sources are missing

To Reproduce

  1. have translated messages in minimal/lingui format
  2. set lingui config to use "po" format
  3. run lingui extract --convert-from minimal/lingui

Expected behavior
Translated strings are copied to new format

Additional context
Converting from "minimal" to "lingui" does work as expected

  • jsLingui version 4.0.0
  • Macro support:
  • I'm using SWC with @lingui/swc-plugin
  • I'm using Babel with babel-macro-plugin
  • I'm not using macro
  • Your Babel config (e.g. .babelrc) or framework you use (Create React App, NextJs, Vite)
@sirpy sirpy changed the title convert-from json/lingui to po doesnt work as expected convert-from json/lingui doesnt work as expected May 7, 2023
@timofei-iatsenko
Copy link
Collaborator

Unfortunately, this functionality was completely abandoned due to lack of interest from my side (and absence of unit tests from previous contributors).

We are happily accept PR's fixing this. Or you can do the converting as one time job manually for your codebase. Using existing formatters this should be pretty straightforward to do.

@sirpy
Copy link
Author

sirpy commented May 10, 2023

@thekip can you explain how to do it manually?

@timofei-iatsenko
Copy link
Collaborator

timofei-iatsenko commented May 10, 2023

fs.readFile() -> oldformat.parse() -> newFormat.serialize() -> fs.writeFile()

// pseudocode, i didn't test it 
import { formatter as createPoFormatter  } from "@lingui/format-po"
import { formatter as createJsonFormatter  } from "@lingui/format-json"
import fs from "fs/promises"

const poFormatter = createPoFormatter()
const jsonFormatter = createJsonFormatter()

async main() {
  const catalog = jsonFormatter.parse(await fs.readFile('myfile.json', "utf8"));
  await fs.writeFile('myfile.po', poFormatter.serialize(newFileConent))
  console.log('Done!')
}

main();

@sirpy
Copy link
Author

sirpy commented May 17, 2023

same issue, it does not serialize correctly the minimal json format to any other format

@timofei-iatsenko
Copy link
Collaborator

timofei-iatsenko commented May 17, 2023

What are you expecting as "correct"? minimal has a reduced amount of data comparing to other formats. Obviously, while converting, it will not take this data from "nowhere".

Do you use any TMS? If yes, it's easier to just re-export your strings from there in a proper format.

@sirpy
Copy link
Author

sirpy commented May 18, 2023

When converting the minimal format to any other type, it will not keep the translation.
I did manual conversion like so:

const catalog = jsonFormatter.parse(fs.readFileSync(`src/language/locales/${item}/catalog.json`, 'utf8'))
    const newCatalog = {}
    Object.keys(catalog).forEach((k) => {
        newCatalog[k] = {
            translation: catalog[k],
            message: k,
            description: '',
            origin: [],
        }
    })
    const output = poFormatter.serialize(newCatalog, {})

Then i've encountered a new issue with the new msgids, my language files are using the previous msgids.
When running lingui extract, it doesnt seem to be using the new msg id format (hashes)
But when running the website it does show hashes instead of msg strings.
So all in all the migration process to V4 seems broken

@sirpy
Copy link
Author

sirpy commented May 18, 2023

So it seems when extracting in "po" format it doesnt use hash msgid but in the UI I do see the msgid hash.
Is that the expected result? this makes it very hard to locate the missing translation

@timofei-iatsenko
Copy link
Collaborator

try this:

import { generateMessageId } from "@lingui/message-utils/generateMessageId"
const catalog = jsonFormatter.parse(fs.readFileSync(`src/language/locales/${item}/catalog.json`, 'utf8'))
    const newCatalog = {}
    Object.keys(catalog).forEach((k) => {
        newCatalog[generateMessageId(k)] = {
            translation: catalog[k],
            message: k,
            description: '',
            origin: [],
        }
    })
    const output = poFormatter.serialize(newCatalog, {})

@sirpy
Copy link
Author

sirpy commented Jun 4, 2023

@thekip that wouldnt work as message id is also composed of context.
the only solution i found was to use po files. then copying previous translation from the json files.
also it seems that using the lingui format isnt working as the i8ln.load require a key-value json. dynamically loading po files seems to convert them to key value json but that doesnt happen with lingui raw json format. but that could be a vite lingui plugin issue.

@timofei-iatsenko
Copy link
Collaborator

for json files you have to use a different notation, otherwise plugin/loader will not recognize them from other not related json's.

// for other extension you have to use `?lingui` suffix
const { messages } = await import(`./locales/${language}.json?lingui`);

And yes, loading, minimal/lingui formats directly to i18n.load was never supported, as you need to compile your catalogs first. (it is not only convert po to json but also compiles ICU expressions)

@pniebrzydowski
Copy link

And yes, loading, minimal/lingui formats directly to i18n.load was never supported, as you need to compile your catalogs first. (it is not only convert po to json but also compiles ICU expressions)

Do I understand correctly that there is then no way to load a translation catalog from an external source (e.g., CDN, AWS S3 bucket, separate project repository, etc) and every change in translation catalogs will require a new build?

@timofei-iatsenko
Copy link
Collaborator

@pniebrzydowski not really. Every translation catalog should be compiled before use in runtime. But this doesn't mean you need to re-build your whole application, only catalogs

You can compile catalogs and put them to an external source as compiled versions, OR you can put them as sources and compile on the fly (for example using some lambda or cloud function). Also take a look at https://github.com/lingui/js-lingui/tree/main/packages/remote-loader

@timofei-iatsenko
Copy link
Collaborator

Continuing the topic. Compiling on the fly has no official solution or tutorial. Actually lingui compile or bundler loaders doing many things under the hood (for example fallbackLanguages or template fallback) but if you don't need all these you simply need to generate hash for messages from its source using @lingui/message-utils. If you have any further questions you can ask them in our discord, we are happy to help.

@pniebrzydowski
Copy link

Thanks - with the remote-loader package, it appears that will only work with the JSON "minimal" format, correct?

@timofei-iatsenko
Copy link
Collaborator

The remote loader is quite abandoned, i mentioned it just as an example how it could be achieved. Here is a dedicated issue #1551

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants