Message extraction

[martinsnajdr]

Hello,

what's the recommended approach to extract messages from the source code and generate the JSON automatically?

Thank you.

[godfrednanaowusu]

why is this not a priority, large projects need this

[amannn]

Right, I agree there's some limitation—i18nAlly is only supported in VSCode and JetBrains. If you're using other editors, extraction currently needs to be done manually.

I'm happy to use this discussion to see if there's more interest from the community in this. I guess this could be implemented via a Babel / SWC plugin, but there might be some edge cases which are somewhat tough to support. For now, I don't have priorities to pursue this. I guess this could also be implemented in userland if someone has a strong need.

[ReesMorris]

We ended up implementing a (rather hacky) solution to mimic the react-intl extraction method after one of our en.json files hit 3,000+ lines. It was becoming practically impossible to maintain.

Essentially we wrapped useTranslations in a custom useIntl hook, exposing it as formatMessage which accepts a defaultMessage and description, e.g:

formatMessage({
  defaultMessage: '{first} et al.',
  description: 'Multiple authors'
}, { first })

These two values get hashed to generate an id which is what next-intl is told the key is for the JSON entry. To populate the JSON file we run i18n which is a custom script to extract all strings out of the formatMessage declaration. We pretty much did the same for getTranslations as well.

Here's a Gist of the above, but use at your own peril as it's definitely not optimised (e.g. hashing every single string at runtime, and having to run i18n to update the JSON files), but we've had no complaints about it. Would be great to see something similar baked in at some point down the line, though.

[vasanthid]

@amannn I'm also interested in a CLI command feature that can scan all files, extract strings, and place them in the correct language files, similar to what's offered by i18next-parser.

Our project, a Next.js 14 app router without i18n routing, is quite extensive and might expand further as we work with distributed teams. We're currently assessing i18n libraries and were leaning towards next-intl, but the absence of an extraction tool might lead us to consider other options.

I have a couple of questions:
a) Could you suggest any tools or methods for automating the extraction process in large projects, aside from VS Code plugins?
b) Is there a possibility that this feature could be given priority in the future?

[trangcongthanh]

@amannn At my work, we have dedicated translation teams for each project. Some of them are 3rd party services, so we can't give them access to our codebase to translate our app. It's not just a limitation for developers because of the editor plugin; it also blocks our ability to outsource the translation to other teams 😿.

[amannn]

@trangcongthanh The JSON files that next-intl typically operates with can be uploaded to any translation management system, making it possible for other project members like translators to collaborate (see for example: Localization management with Crowdin).

[michalpulpan]

👋
Since this was a required feature for our project and we really like usability of the next-intl, I decided to write a simple script for this functionality this morning.
Feel free to check it out, I tested it on our project and it seems to work fine. However, for the first run, make sure to backup your messages folder (even though script creates timestamped backups).

The structure of messages in our project looks like this: messages/{LOCALE}/{NAMESPACE}.json:

.
└── messages/
    ├── cs/
    │   ├── blog.json
    │   └── common.json
    ├── en/
    │   ├── blog.json
    │   └── common.json
    └── de/
        ├── blog.json
        └── common.json

So feel free to adjust it to your use-case.

[vasanthid]

@michalpulpan That's great. Thank you so much. I'll test it and let you know.

[amannn]

@felix-quotez just shared his message extraction strategy in #1700 (comment):

import { defineMessages } from '@tw/intl/define-messages'
import { useTranslations } from '@tw/intl/use-translations'

defineMessages({
  propertyNameIsIgnoredInFavorOfIdBelow: {
    id: 'a.b',
    description: 'foo',
    defaultMessage: 'bar',
  },
})

export function Component() {
  const t = useTranslations('a')
  return t('b')
}

defineMessages uses @formatjs/cli to extract messages and puts it into a format for next-intl.

[amannn]

An argument that @remcohaszing has just brought up for message extraction: It helps to purge unused messages automatically.

Another thought I had recently is that maybe defining messages inline could be helpful in regard to AI generation of code, similar to how TailwindCSS works really well with these tools. But AI tools are also improving and multi file editing is already a possibility, so not sure if this will be a benefit in the long run.

One aspect surely is though that defining messages inline is a bit more friendly for quick prototyping.

I haven't made my mind up yet on this subject, for the time being I'm happy to discuss experiences that people have made and to explore the pro's and con's of this approach—also potentially community-based solutions.

[remcohaszing]

When I was using react-intl, I really liked that I could define messages inline. Something I liked less is the use of a hash as an id. It means the message source is not clear from the translation file. Also changing a message, changes the hash. That means that when sorting, a message change causes it to be moved to somewhere else entirely.

I also really liked extract-react-intl-messages / babel-plugin-react-intl-auto. But when I switched from Babel to TypeScript, it was really inconvenient to depend on a Babel plugin.