Re: Lilipond/nikud bug: Pango-bug ? [paps OK]

[Yotam Medini יותם מדיני]

Hello Behdad,

Thanks for you note.

Lilipond - as far as I understand - does not open Hebrew shaper directly,
but rather gets indirectly to Hebrew specific code, by Pango loading 
   lib/pango/1.6.0/modules/pango-hebrew-fc.so

Please have a look at the report of 
   http://code.google.com/p/lilypond/issues/detail?id=541
and specifically the screenshot of PostScript images.

From what I have seen so far there is no problem with bidi-ordering 
of characters, but there is a problem in positioning characters when
diacritics (nikud) marks are used.

I still wish I could get a short 'paps'-like program that uses pango_shape(...).

Following is a gdb 'where' report, of some relevant state of Lilipond.

regards -- yotam

 (gdb) where
 #0  Lookup_MarkBasePos (gpi=0xbfc4fa64, st=0x85e9e98, buffer=0x86407a8, 
flags=1, 
     context_length=65535, nesting_level=1) at harfbuzz-gpos.c:2356
 #1  0xb7e61a9c in GPOS_Do_Glyph_Lookup (gpi=0xbfc4fa64, lookup_index=2, 
     buffer=0x86407a8, context_length=65535, nesting_level=1)
     at harfbuzz-gpos.c:5985
 #2  0xb7e61c1a in GPOS_Do_String_Lookup (gpi=0xbfc4fa64, lookup_index=2, 
     buffer=0x86407a8) at harfbuzz-gpos.c:6109
 #3  0xb7e62073 in HB_GPOS_Apply_String (face=0x8547c30, gpos=0x8587a28, 
     load_flags=0, buffer=0x86407a8, dvi=0 '\0', r2l=1 '\001')
     at harfbuzz-gpos.c:6286
 #4  0xb7e4de89 in pango_ot_ruleset_position (ruleset=0x8547778, 
buffer=0x857eb90)
     at pango-ot-ruleset.c:235
 #5  0xb5a07304 in hebrew_engine_shape (engine=0x856ad20, font=0x8569190, 
     text=0x852e9b4 "×\233Ö·", length=4, analysis=0x8568834, glyphs=0x856adb0)
     at hebrew-fc.c:380
 #6  0xb7e11c48 in _pango_engine_shape_shape (engine=0x856ad20, font=0x8569190, 
     text=0x852e9b4 "×\233Ö·", length=4, analysis=0x8568834, glyphs=0x856adb0)
     at pango-engine.c:71
 #7  0xb7e268b0 in pango_shape (text=0x852e9b4 "×\233Ö·", length=4, 
     analysis=0x8568834, glyphs=0x856adb0) at shape.c:51
 #8  0x0817d9dc in Pango_font::pango_item_string_stencil (this=0x852ecc8, 
     item=0x8568828, address@hidden, tight_bbox=true) at pango-font.cc:107
 #9  0x0817efdb in Pango_font::text_stencil (this=0x852ecc8, address@hidden, 
     tight=true) at pango-font.cc:320
 #10 0x0817f4ce in Pango_font::word_stencil (this=0x852ecc8, address@hidden)
     at pango-font.cc:286
 #11 0x082104b9 in Text_interface::interpret_string (layout_smob=0xb73d2648, 
     props=0xb72b9790, markup=0xb6d8b1d0) at text-interface.cc:38
 #12 0x0821055f in Text_interface::interpret_markup (layout_smob=0xb73d2648, 
     props=0xb72b9790, markup=0xb6d8b1d0) at text-interface.cc:46
 #13 0xb7f33f3c in ceval (x=<value optimized out>, env=0xb72b9580) at 
eval.c:4536
 #14 0xb7f2f592 in scm_apply_2 (proc=0xb71b7bd0, arg1=0xb73d2648, 
     arg2=0xb72b9790, args=0xb72b9748) at eval.c:4701
 #15 0x08210647 in Text_interface::interpret_markup (layout_smob=0xb73d2648, 
     props=0xb72b9790, markup=0xb72b95f0) at text-interface.cc:54
 #16 0xb7f33f3c in ceval (x=<value optimized out>, env=0xb72b9770) at 
eval.c:4536
 #17 0xb7f33521 in scm_call_1 (proc=0xb7360850, arg1=0xb7386920) at eval.c:4661
 #18 0x080f3743 in Grob::try_callback (this=0x84cc7f8, sym=0xb5baf350, 
     proc=0xb7360850) at grob-property.cc:171
 #19 0x080f388e in Grob::internal_get_property (this=0x84cc7f8, sym=0xb5baf350)
     at grob-property.cc:140
 #20 0x080f7e23 in Grob::get_stencil (this=0x84cc7f8) at grob.cc:102
 #21 0x080f7e51 in grob_stencil_extent (me=0x84cc7f8, a=X_AXIS) at grob.cc:643
 #22 0x080f7ef5 in Grob::stencil_width (smob=0xb7386920) at grob.cc:690
 #23 0xb7f2e95f in scm_apply (proc=0xb78129e8, arg1=0xb7386920, 
     args=<value optimized out>) at eval.c:4884
 #24 0xb7f33521 in scm_call_1 (proc=0xb78129e8, arg1=0xb7386920) at eval.c:4661
 #25 0x080f3743 in Grob::try_callback (this=0x84cc7f8, sym=0xb5baf5d0, 
     proc=0xb78129e8) at grob-property.cc:171
 #26 0x080f388e in Grob::internal_get_property (this=0x84cc7f8, sym=0xb5baf5d0)
     at grob-property.cc:140
 #27 0x080f7497 in Grob::extent (this=0x84cc7f8, refp=0x84cc7f8, a=X_AXIS)
     at grob.cc:398
 #28 0x08129b63 in Hyphen_spanner::set_spacing_rods (smob=0xb737bdb8)
     at lyric-hyphen.cc:120
 #29 0xb7f2e95f in scm_apply (proc=0xb78040f8, arg1=0xb737bdb8, 
     args=<value optimized out>) at eval.c:4884
 #30 0xb7f33521 in scm_call_1 (proc=0xb78040f8, arg1=0xb737bdb8) at eval.c:4661
 #31 0x080f3743 in Grob::try_callback (this=0x84cce08, sym=0xb5baf390, 
     proc=0xb78040f8) at grob-property.cc:171
 #32 0x080f388e in Grob::internal_get_property (this=0x84cce08, sym=0xb5baf390)
     at grob-property.cc:140
 #33 0x08209b58 in System::pre_processing (this=0x84c0f38) at system.cc:294
 #34 0x08186c22 in Paper_score::process (this=0x84c0ee8) at paper-score.cc:132
 #35 0x080e73e3 in ly_format_output (context=0xb73bc3c0)
     at global-context-scheme.cc:32
 #36 0x081b3fa4 in Score::book_rendering (this=0x84becb0, layoutbook=0x843d7d8, 
     default_def=0x84bed18, book_key=0x843dc38) at score.cc:206
 #37 0x08094ff0 in Book::process (this=0x84becf0, default_paper=0x847f770, 
     default_layout=0x84bed18) at book.cc:149
 #38 0x0809489d in ly_parser_print_book (book_smob=0xb73e1098, 
     default_paper=0xb75c7088, default_layout=0xb740c0a8, output=0xb6d8ac70)
     at book-scheme.cc:57
 #39 0xb7f47d84 in scm_gsubr_apply (args=0x1) at gsubr.c:220
 #40 0xb7f2eb27 in scm_apply (proc=<value optimized out>, arg1=0xb7845138, 
     args=<value optimized out>) at eval.c:4921
 #41 0xb7f33cdb in ceval (x=<value optimized out>, env=0xb73e0f60) at 
eval.c:4372
 #42 0xb7f363fb in scm_primitive_eval (exp=0xb73e11b0) at eval.c:5924
 #43 0x08188156 in internal_ly_parse_scm (ps=0xbfc50cac) at parse-scm.cc:62
 #44 0x081881c0 in ly_parse_scm (
     s=0x843324e "(if (pair? toplevel-scores)\n  (toplevel-book-handler\n   
parser\n   (apply ly:make-book $defaultpaper $defaultheader 
toplevel-scores)))\n", 
     n=0xbfc50e10, address@hidden, safe=false) at parse-scm.cc:124
 #45 0x0824f4ad in Lily_lexer::yylex (this=0x8431f68) at lexer.ll:337
 #46 0x08251d2e in yylex (s=0xbfc52438, loc=0xbfc520f8, v=0x8431ed0)
     at parser.yy:2533
 #47 0x0825328e in yyparse (my_lily_parser=0x8431ed0) at out/parser.cc:2493
 #48 0x0825dae5 in Lily_parser::do_yyparse (this=0x8431ed0) at parser.yy:2313
 #49 0x0811861b in Lily_parser::parse_file (this=0x8431ed0, address@hidden, 
     address@hidden, address@hidden) at lily-parser.cc:114
 #50 0x081170ac in ly_parse_file (name=0xb6dab990) at lily-parser-scheme.cc:131
 #51 0xb7f34e03 in ceval (x=0x404, env=0xb77a9c58) at eval.c:4223
 #52 0xb7f3355d in scm_call_0 (proc=0xb77a9cc0) at eval.c:4655
 #53 0xb7f91480 in scm_body_thunk (body_data=0xbfc52e28) at throw.c:356
 #54 0xb7f91ac6 in scm_c_catch (tag=0xb6dd2e60, body=0xb7f91460 
<scm_body_thunk>, 
     body_data=0xbfc52e28, handler=0xb7f91420 <scm_handle_by_proc>, 
     handler_data=0xbfc52e48, pre_unwind_handler=0, 
     pre_unwind_handler_data=0xbfc52e4c) at throw.c:204
 #55 0xb7f91c16 in scm_catch_with_pre_unwind_handler (key=0x8641130, 
     thunk=0xb77a9cc0, handler=0xb77a9ca0, pre_unwind_handler=0x204)
     at throw.c:588
 #56 0xb7f47d84 in scm_gsubr_apply (args=0x1) at gsubr.c:220
 #57 0xb7f2eb27 in scm_apply (proc=<value optimized out>, arg1=0xb78a3b10, 
     args=<value optimized out>) at eval.c:4921
 #58 0xb7f33cdb in ceval (x=<value optimized out>, env=0xb77a9d00) at 
eval.c:4372
 #59 0xb7f366d3 in scm_eval_body (code=0xb7812f28, env=0xb77a9d48) at 
eval.c:2987
 #60 0xb7f36876 in call_closure_1 (proc=0xb77a9d60, arg1=0xb6dab990)
     at eval.c:5250
 #61 0xb5b999d3 in scm_srfi1_for_each (proc=0xb77a9d60, arg1=0xb77aa028, 
     args=<value optimized out>) at srfi-1.c:1512
 #62 0xb7f34c04 in ceval (x=0x404, env=0xb77a9d98) at eval.c:4357
 #63 0xb7f34e46 in ceval (x=0xb77a9df0, env=0xb77a9d98) at eval.c:3388
 #64 0xb7f352fc in ceval (x=<value optimized out>, env=0xb77a9f38) at 
eval.c:3639
 #65 0xb7f33521 in scm_call_1 (proc=0xb78141a8, arg1=0xb77aa028) at eval.c:4661
 #66 0x0812d0a6 in main_with_guile () at main.cc:439
 #67 0xb7f4c266 in invoke_main_func (body_data=0xbfc535c4) at init.c:367
 #68 0xb7f1e2a2 in c_body (d=0xbfc53548) at continuations.c:350
 #69 0xb7f91ac6 in scm_c_catch (tag=0x104, body=0xb7f1e290 <c_body>, 
     body_data=0xbfc53548, handler=0xb7f1e2b0 <c_handler>, 
     handler_data=0xbfc53548, 
     pre_unwind_handler=0xb7f91310 <scm_handle_by_message_noexit>, 
     pre_unwind_handler_data=0x0) at throw.c:204
 #70 0xb7f1e802 in scm_i_with_continuation_barrier (body=0xb7f1e290 <c_body>, 
     body_data=0xbfc53548, handler=0xb7f1e2b0 <c_handler>, 
     handler_data=0xbfc53548, 
     pre_unwind_handler=0xb7f91310 <scm_handle_by_message_noexit>, 
     pre_unwind_handler_data=0x0) at continuations.c:326
 #71 0xb7f1e8e3 in scm_c_with_continuation_barrier (
     func=0xb7f4c220 <invoke_main_func>, data=0xbfc535c4) at continuations.c:368
 #72 0xb7f90bb9 in scm_i_with_guile_and_parent (
     func=0xb7f4c220 <invoke_main_func>, data=0xbfc535c4, parent=0x0)
     at threads.c:695
 #73 0xb7f90cae in scm_with_guile (func=0xb7f4c220 <invoke_main_func>, 
     data=0xbfc535c4) at threads.c:683
 #74 0xb7f4c1ff in scm_boot_guile (argc=5, argv=0xbfc536c4, 
     main_func=0x812cb3a <main_with_guile>, closure=0x0) at init.c:350
 #75 0x0812ed2d in main (argc=5, argv=0xbfc536c4) at main.cc:637
 (gdb) 



On Tue, 12 Feb 2008 13:16:43 -0500
Behdad Esfahbod wrote:

> From: Behdad Esfahbod <address@hidden>
> To: Yotam Medini 
> Cc: Dov Grobgeld, Behdad Esfahbod,  address@hidden
> Subject: Re: Lilipond/nikud bug: Pango-bug ? [paps OK]
> Date: Tue, 12 Feb 2008 13:16:43 -0500
> Sender: Behdad Esfahbod <address@hidden>
> 
> Hi Yotam,
> 
> I'm puzzled.  Do you really mean LilyPond opens the hebrew shaper object
> file itself and uses it directly?  If yes, that's an unheard hack!
> 
> There's nothing fancy to using pango_itemize() and pango_shape()
> correctly.  I have a hard time imagining what may be going wrong for
> you.  Using them correctly to fully implement line breaking, bidi
> reordering, and all different kinds of PangoAttributes though takes more
> than a few lines and is exactly why pango-layout.c is six thousand
> lines.  I can't immediately think why LiliPond shouldn't be able to use
> PangoLayout easily.
> 
> 
> Regards,
> 
> behdad