Re: Question about Weblate scripts

[Ricardo Lafuente]

Hello Libor,

Thank you for your detailed analysis and support!

After resetting (well, checking out) the weblate branch and re-running the script with the debug branch, I can confirm it's working as expected. I'm not sure what was hanging everything up, but things look normal and seem to work according to your comments.

If you'd like, I can go back to the script and test those conditions, but things are working now (and I'm sure we'll run into something again once we expand the translation pipeline)

I will now make another attempt to add Weblate components for the website pages. After reading and testing Weblate's Markdown support, it might be the case that it's not sufficient for our use case -- for instance, I'm not sure it can handle the markdown frontmatter properly. We'll experiment further, but maybe the meeting today is a good moment to discuss a possible plan B in case this is not an option:

1. Manually set up a pipeline using a2po, which is what Weblate uses behind the scenes for its Markdown support, but maybe it can be fine-tuned for our specific purposes. This will be quite a bit of work though
   
2. Have a different translation pipeline without Weblate for the website files (e.g. just edit them in the repository?). This is of course a last-resource compromise which we'd definitely like to avoid

Thank you again for your careful help!

Ricardo

On 2023-10-30 13:37, Libor Polčák wrote:

> > At this point we could really use some
> >         guidance -- are the failures expected? Do they occur because of
> >         our changes to the weblate script?
> >         
> >       
>       Can you report what you get by:
>       
>       
>       git remove -v
>       
>       
>       (You should see a line like:
>       
>       pagure    ssh://git@pagure.io/JShelter/webextension.git (fetch)
>       
>       )
>       
>       
>       I will use $PAGURE as the name in the first column bellow. So in
>       my case $PAGURE==pagure, you might see origin or something
>       different.
>       
>       
>       git fetch $PAGURE
>       
>       
>       git checkout dd7144fe7803c438f77e55c56dd248b0fea0252b
>       
>       git switch -c debugbranch
>       
>       git stash
>       
>       git status
>       
>       
>       Please have a look at the list of Untracked files. Remove/backup
>       anything in tools/i18n/ or common/_locales/ or any other
>       suspicious file.
>       
>       
>       $ LC_ALL="C" tools/i18n/weblate2main.py
>       
>       I suppose that you have a branch weblate mirroring
>       https://hosted.weblate.org/git/jshelter/webextension/. Make sure
>       that your working tree is clear. Use git stash or similar commands
>       if necessary.
>       
>       
>       Locales in current branch debugbranch
>       
>       cs en ru
>       
>       Press Enter to continue or exit with SIGINT (Ctrl+C)
>       
>       
>       M    tools/i18n/weblate2main.py
>       
>       Switched to branch 'weblate'
>       
>       Write YES if you want to import locale fr ... NO
>       
>       M    tools/i18n/weblate2main.py
>       
>       Switched to branch 'debugbranch'
>       
>       
>       There should be no new commit and the script should not produce
>       any error.
>       
>       
>       Is that what you get or something different? See also my inline
>       comments below.
>       
>       
> > Hi all,
> >         
> >         
> >         We've been experimenting with Weblate for handling the website
> >         Markdown files. Even though their Markdown support is
> >         experimental, it's apparently doable. We created a component,
> >         website-faq (markdown requires a component per file) to try
> >         things out; apparently strings are extracted properly, but we
> >         still have to confirm whether the translations are properly
> >         applied back.
> >         
> >         
> >         But we couldn't run the i18n helper scripts. First, there was a
> >         python error:
> >         
> >         
> >         Traceback (most recent call last):
> >         
> >            File
> >         "/home/rlafuente/repos/jshelter/./tools/i18n/weblate2main.py",
> >         line 111, in <module>
> >         
> >              bl[key]["description"] == "" and "description" not in
> >         wl[key] and \
> >         
> >              ~~~~~~~^^^^^^^^^^^^^^^
> >         
> >         KeyError: 'description'
> >         
> >       
>       
>       That's weird. I can run weblate2main.py at my computer fine from
>       dd7144fe7803c438f77e55c56dd248b0fea0252b.
>       
>       
>       I tried running that script and not importing the partial French
>       translation of one of our users and also with the import. Both
>       times it created a good commit.
>       
>       
>       One question: Do you have a partial pt translation? What is the
>       output of "ls common/_locales/"? At my computer, it is:
>       
>       
>       cs  en  ru
>       
>       
>       I tried to create a pt directory and replicate your error, but if
>       I do not create the message.json, I get:
>       
>       
>           f = open("%s/%s/messages.json" % (LOCALES_PATH, locale))
>       
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>       
>       FileNotFoundError: [Errno 2] No such file or directory:
>       'common/_locales/pt/messages.json'
>       
>       
>       When I create that file by copying the English file, everything
>       works.
>       
>       
>       Have you added anything to common/_locales/en/messages.json?
>       
>       
> > The KeyErrors can be dealt with by using
> >         .get() instead, so we made these changes which seemed to address
> >         the error:
> >         
> >         
> >                       # Check if the only difference is a missing empty
> >         description
> >         
> >                       if bl[key]["message"] == wl[key]["message"] and \
> >         
> >         -                bl[key]["description"] == "" and "description"
> >         not in wl[key] and \
> >         
> >         +                not bl[key].get("description") and
> >         "description" not in wl[key] and \
> >         
> >       
>       
>       Here, I expect that there is always the description key in the
>       base EN JSON.
>       
>       
> >                   (("placeholders" not in
> >         bl[key] and "placeholders" not in wl[key]) or \
> >         
> >         -                (["placeholders"] in bl[key] and "placeholders"
> >         in wl[key] and \
> >         
> >         +                ("placeholders" in bl[key] and "placeholders"
> >         in wl[key] and \
> >         
> >       
>       
>       I am not sure what bl[key] holds but this looks like a good catch.
>       Can you confirm that there is not an array?
>       
>       
> > -                bl[key]["placeholders"]
> >         == wl[key]["placeholders"])):
> >         
> >         + bl[key].get("placeholders") == wl[key]["placeholders"])):
> >         
> >                               continue # Skip such strings
> >         
> >         
> >         After the fix, running ./tools/i18n/weblate2main.py from the
> >         root repo directory gives out a string of failures:
> >         
> >         
> >         I suppose that you have a branch weblate mirroring
> >         https://hosted.weblate.org/git/jshelter/webextension/. Make sure
> >         that your working tree is clear. Use git stash or similar
> >         commands if necessary.
> >         
> >         
> >         Locales in current branch weblate
> >         
> >         fr cs ru en
> >         
> >         Press Enter to continue or exit with SIGINT (Ctrl+C)
> >         
> >       
>       
>       The script asks me whether to import French:
>       
>       Write YES if you want to import locale fr ... NO
>       
>       
>       I guess that you imported French before.
>       
> >         
> >         Your branch is up to date with 'origin/weblate'.
> >         
> >         patching file common/_locales/cs/messages.json
> >         
> >         Hunk #1 FAILED at 85.
> >         
> >         Hunk #2 FAILED at 164.
> >         
> >         Hunk #3 FAILED at 192.
> >         
> >         Hunk #4 FAILED at 200.
> >         
> >         Hunk #5 FAILED at 223.
> >         
> >         Hunk #6 FAILED at 245.
> >         
> >         Hunk #7 FAILED at 294.
> >         
> >         Hunk #8 FAILED at 319.
> >         
> >         Hunk #9 FAILED at 355.
> >         
> >         Hunk #10 FAILED at 446.
> >         
> >         Hunk #11 FAILED at 624.
> >         
> >         Hunk #12 FAILED at 661.
> >         
> >         Hunk #13 FAILED at 781.
> >         
> >         Hunk #14 FAILED at 833.
> >         
> >         Hunk #15 FAILED at 845.
> >         
> >         Hunk #16 FAILED at 877.
> >         
> >         Hunk #17 FAILED at 939.
> >         
> >         Hunk #18 FAILED at 955.
> >         
> >         Hunk #19 FAILED at 971.
> >         
> >         Hunk #20 FAILED at 995.
> >         
> >         Hunk #21 FAILED at 1007.
> >         
> >         Hunk #22 FAILED at 1059.
> >         
> >         Hunk #23 FAILED at 1119.
> >         
> >         Hunk #24 FAILED at 1131.
> >         
> >         Hunk #25 FAILED at 1147.
> >         
> >         Hunk #26 FAILED at 1163.
> >         
> >         Hunk #27 FAILED at 1179.
> >         
> >         Hunk #28 FAILED at 1187.
> >         
> >         Hunk #29 FAILED at 1203.
> >         
> >         Hunk #30 FAILED at 1355.
> >         
> >         Hunk #31 FAILED at 1379.
> >         
> >         Hunk #32 FAILED at 1391.
> >         
> >         Hunk #33 FAILED at 1411.
> >         
> >         33 out of 33 hunks FAILED -- saving rejects to file
> >         common/_locales/cs/messages.json.rej
> >         
> >       
>       
>                   sp = subprocess.run(["diff", "-Naur",
>       "--ignore-all-space", "--ignore-blank-lines",
>       
>                                        tmp.name,
>       git_translation_filename],
>       
>                                        capture_output=True)
>       
>                   patch_tmp = tempfile.NamedTemporaryFile(mode="w",
>       suffix = ".patch", prefix = "jshelter_%s_" % locale, delete=False)
>       
>       
>       I guess that you might have a different version of diff that does
>       not support  "-Naur --ignore-all-space --ignore-blank-lines"?
>       
>       
>       Hmm, I guess that you are using different locales. Can you try:
>       
>       
>       LC_ALL="C" tools/i18n/weblate2main.py
>       
>       ?
>       
>       
>                   patch_tmp.write(sp.stdout.decode("utf-8"))
>       
>                   patch_tmp.close()
>       
>                   os.remove(tmp.name)
>       
>       
>       Maybe try to remove/comment that line and have a look what was
>       created.
>       
>       
>       At my computer with dd7144fe7803c438f77e55c56dd248b0fea0252b, I do
>       not get to that code branch as there is nothing to apply.
>       
>       
>                   subprocess.run(["patch", "-R",
>       git_translation_filename, patch_tmp.name])
>       
>        
>       
>       Best wishes
>       
>       
>       Libor
>       
>