Re: [Texmacs-dev] [Announce] nogencc-0.6

[Joris van der Hoeven]

On Sun, 5 May 2002, David Allouche wrote:

> On Saturday 04 May 2002 12:20, Joris van der Hoeven wrote:
> > Did you try to compile it with gcc 2.96 (before I retry)?
> 
> I did not. I will take care of that issue when the code is stable. That 
> is probably anytime soon now.

Yes, please try it with different versions of gcc and
on different platforms if possible.

> > >   Minor cleanup in array.h and array.cc
> > >
> > >   Refcounting fix in hashmap and rel_hashmap
> >
> > Please make the necessary changes in the code only.
> 
> Those were cleanups in the rewritten code. I let a few ugliness and a 
> couple of refcounting leak slip through previous versions and I fixed 
> them when fiddling with const and references.
> 
> > >   operator-> defined in basic.h are now const-correct.
> >
> > Please let us deal with the const problem after the change.
> 
> Again, this only affect code I have rewritten anyway. I really do not see 
> the point in postponing that particular change.

Please do not put any const's in the code yet.
I want to postpone *all* changes which do not *directly*
have to do with removing gencc. When making profound changes,
one has to be patient and rigourous. I do not want to repeat
this hundred times. In any case, you can make a backup of
what you have done and make the change later.

> > >   More regular naming of .cc files.
> >
> > Please keep the same names as before.
> > We will do additional cleanup afterwards.
> 
> I did that to allow me to implement aggregate compilation easily and in a 
> way consistent with gencc. Now wigdet and boxes .cc files are named after 
> the code module they used to define.
> 
> I previously had to introduce an exception on the naming scheme 
> (Boxes/basic.cc was renamed Boxes/basic_boxes.cc) to make separate 
> compilation work; otherwise the object generated from Boxes/basic.cc 
> overwrote the object generated from Types/basic.cc. The new naming scheme 
> avoid that kind of name clash, is much more regular, and cause no change 
> in code.

I don't like this, but I guess this is necessary to avoid name clashes.

> In future releases, I will include a script which renames all 
> files back the original name, so diff would not be confused.

Yes, please do so. And maybe a script for the opposite direction too.

> > > Aggregate compilation did not improve the compilation time. Disabling
> > > automatic template instanciation does not seem to help either. In the
> > > next version I will try a simpler reference conting system in the
> > > hope of improving the compilation time.
> >
> > Bad news. Maybe the C++ template system is just too monstruous
> > (as I think it is; too much wisdom kills the wisdom) and far from
> > competitive with the inherently better gencc system :^)
> 
> Actually, benchmarking shows that aggregate compilation is faster without 
> optimization, but can be slower with optimization.

OK, so we will opt for non-aggregate compilation on the long run.

> Obviously, aggregate compilation helps save time on parsing (headers 
> files are included only once) and creation of generic code 
> (uninstanciated templates), thus the better performance w/o optimization. 
> But big compilation units seem to slow down the optimizer. It might 
> simply be an algorithmic problem (optimization algorithms which are 
> slower than O(N) cause slowdown on big units), or it might be that big 
> compilation units allow more non-local optimizations. In that case, it 
> may even be useful to compile everything in one big monster compilation 
> unit to get the best out of the optimizer.

We would have to check this last point.

> > Did you try to compile with
> >
> > export CXXFLAGS="-O3 -fexpensive-optimizations -malign-loops=2
> > -malign-jumps=2 -malign-functions=2 -fno-exceptions -fno-rtti
> > -fno-implicit-templates"
> 
> I did not. I will use it for optim_level=high in the next release.
> 
> I will not disable automatic template instanciation. Once the 
> .rpo files have been generated, that cause no increase in compilation 
> time. Actually, it even decrease compilation time since only needed 
> template methods are instanciated. And the .rpo files can be generated 
> rather quickly by using a first pass of non-optimizing compilation, and 
> they could even be included in the distribution, since they are platform 
> independant and compatible across versions of g++2.

Yes, but it may also affect the runtime performance and
the size of the executable. Please test these issues.

> > I do not think that reference counting will affect the compilation time
> > a lot.
> 
> My reimplementation of reference counting, resources and events used a 
> lot of useless indirections. That had probably no impact on generated 
> code since that used only one-line inline methods, but it did have a 
> mesurable impact on compilation time.

I do *not* want you to reimplement reference counting yet.
Again, such changes have to be postponed after removing gencc.

> By the way, I understand that inline is a hint, but I think it is a safe 
> bet to consider that one-statement pass-thru inline functions (which 
> essentially produce no code) ARE inlined when compiling with 
> optimizations.

This is a belief. Anyway, I don't want cascades of inline functions now
(and maybe never). Your goal was to remove gencc. Please stick to that and
we'll discuss other issues afterwards.

> On Sunday 05 May 2002 00:37, Joris van der Hoeven wrote:
> > I retried a compilation. This time I have hundreds of messages of the
> > form
> >
> >     In file included from Basic/Data/upgradetm.hh:14,
> >                      from ./Basic/Data/upgradetm.cc:13:
> >     Basic/Types/path.hh:66:24: warning: no newline at end of file
> >     Basic/Types/path.hh:66:24: warning: no newline at end of file
> >
> > You probably did not remove the Emacs comments in the right way
> > at the end of the files.
> 
> I will have a look. Weirdly, I have no such error here.

Yes, but that is precisely one illustration why changes like removing
gencc are dangerous: it affects the stability of the system
in a different way on each platform.

> > Compilation remains extremely slow; I have the impression
> > that compiling one single file is about thrice as fast as
> > compiling one aggregate in the official distribution
> > (whence an overall slowdown by a factor between two and three :^( ).
> > Certain particular files, like Typeset/Env/env_exec.cc are terrible...
> > On the other hand, separate compilation also has advantages, of course.
> 
> I have benchmarked compilation here. I guess the slowdown you 
> observed is due to the use of optimizations by default in my distribution.
> 
> The following results were obtained here with gcc-2.95.4, and for 
> compilation an linking only, that is after generation of support files 
> and a first pass of non-optimized compilation (for generating the .rpo 
> files). The reported time is the "user" time as mesured with the "time" 
> program.
> 
> optim=none -> no optimization flag.
> optim=medium -> -O2
> optim=high -> -O3 -fexpensive-optimizations -malign-loops=2
>               -malign-jumps=2 -malign-functions=2
> 
>        version \ optim  | none        medium      high      
> ------------------------+-----------------------------------------------
> TeXmacs-1.0.0.1         | 1m55.820s   13m7.290s   12m47.790s
> nogencc-0.6 (SEPARATE)  | 3m22.500s   13m54.550s  13m59.570s
> nogencc-0.6 (AGGREGATE) | 1m41.090s   13m52.910s  14m10.760s
> 
> And then delta relative to the official release
> 
>        version \ optim  | none      medium   high      
> ------------------------+-----------------------------------------------
> TeXmacs-1.0.0.1         |   0          0       0
> nogencc-0.6 (SEPARATE)  | + 75 %    +  6 %   +  9 %
> nogencc-0.6 (AGGREGATE) | - 13 %    +  6 %   + 11 %

I obtained quite different timings: 35m for nogencc-0.6 and
14m for the official distribution (using gcc 2.96 on a 1GHz laptop).

> I am currently working on a version which actually compiles faster than 
> the official one in all cases but unoptimized separate compilation.

That is cool, but please don't cheat by modifying the source code:
as you understand by now, this should be done after removing gencc.

> > Also, it is important to allow static compilation: for the official
> > TeXmacs binary distributions, I prefer static binaries, which are only
> > slightly larger in size and much more stable from the user point of
> > view. Standard distributions which support TeXmacs (like Debian and
> > Mandrake Cooker) build their own dynamic versions anyway.
> 
> I know static compilation is important, but linker flags are arcane. I 
> will fix it when other issues are resolved.

OK, please fix these other issues.

Keep up the good work,

Joris