Re: PSPP Windows/SET TEMP/File save times

[Ben Pfaff]

Now I am confused.  You mentioned that portable files were
involved:

>     >     I should mention that the file was saved in the
>     >     'portable' format (9Mb/90mins) - I've since saved the
>     >     'portable' file in the 'system' format instead (3Mb/2
>     >     mins), and the 'system' version also seems to load
>     >     much more quickly.

Can you clarify which actions, on which file formats, took what
amount of time?  If it is clear what is slow, and how important
that is, then it is clear what parts of PSPP need work.

Thanks,

Ben.

"David Purdy" <address@hidden> writes:

> Ben,
>  
> Many thanks - just to note that this involved a *system* rather than portable
> format save:
>  
>> 2) Delete all but 8 fields, reduces .sav file to 4 Mb
>> 3) Save process took approx. 1hr 40 mins (8:40-10:20 pm)
> Regards,
>  
> Dave.
>
>     ----- Original Message -----
>     From: Ben Pfaff
>     To: David Purdy
>     Cc: address@hidden
>     Sent: Tuesday, August 05, 2008 6:12 PM
>     Subject: Re: PSPP Windows/SET TEMP/File save times
>    
>     Loading and, especially, saving to the portable file format is
>     not well optimized.  It is difficult to do in a fast and portable
>     manner, because numbers in portable files are expressed in an
>     awkward manner that is difficult to work with quickly.  It's
>     quite possible that this is the problem, especially given that
>     the system file format worked much faster.
>    
>     "David Purdy" <address@hidden> writes:
>    
>     > Correction: a file save with SET TEMP defined in WinXP has produced no
>     > material improvement in lapsed time over a comparable file save without
>     SET
>     > TEMP defined (>90 minutes to save 9 Mb; see earliest message):
>     > 
>     > 1) Source SPSS .sav file 417 fields x 510,121 records = 265 Mb
>     > 
>     > 2) Delete all but 8 fields, reduces .sav file to 4 Mb
>     > 
>     > 3) Save process took approx. 1hr 40 mins (8:40-10:20 pm)
>     > 
>     > PC = WinXP Pro SP3, 78 Gb hard disk space available, 2048 Mb RAM (Dell
>     > Dimension 8300 3.2GHz Pentium 4)
>     > 
>     > Regards,
>     > 
>     > Dave.
>     > 
>     > ----- Original Message -----
>     > From: unknown-1
>     > To: David Purdy
>     > Cc: address@hidden
>     > Sent: Sunday, August 03, 2008 9:37 PM
>     > Subject: Re: 0.6.0-10 PSPP windows - Queries (WinXP)
>     > 
>     > [cut]
>     >
>     >     2) Quicker 'Save' - I've had some mixed results, maybe influenced by
>     >     working with a large file (>500,000 records).   After trying
>     something
>     >     similar to the very slow save reported earlier (after deleting all
>     but
>     >     about 7 fields), I got the impression that the file was being
>     created much
>     >     more quickly.
>     >
>     > Nice that is what I expected from using the temp variable/directory.
>     > 
>     > [cut]
>     >
>     >     ----- Original Message -----
>     >     From: David Purdy
>     >     To: address@hidden
>     >     Cc: Ben Pfaff [PSPP]
>     >     Sent: Friday, August 01, 2008 11:08 AM
>     >     Subject: Re: 0.6.0-10 PSPP windows - Queries (WinXP)
>     >   
>     >     Ben,
>     >     
>     >     Many thanks for the feedback.
>     >     
>     >     > 1) Very long file save time ?
>     >     
>     >     I was using the GUI version - the disk didn't seem to be especially
>     >     active, nor PSPP using a lot of CPU time (I think).   I should
>     mention
>     >     that the file was saved in the 'portable' format (9Mb/90mins) - I've
>     since
>     >     saved the 'portable' file in the 'system' format instead (3Mb/2
>     mins), and
>     >     the 'system' version also seems to load much more quickly.
>     >     
>     >     [cut]
>     >   
>     >         ----- Original Message -----
>     >         From: Ben Pfaff
>     >         To: David Purdy
>     >         Cc: address@hidden
>     >         Sent: Friday, August 01, 2008 4:52 AM
>     >         Subject: Re: 0.6.0-10 PSPP windows - Queries (WinXP)
>     >       
>     >         "David Purdy" writes:
>     >       
>     >         > 1) Very long file save time ? - I'm using a large third party
>     SPSS
>     >         .sav file
>     >         > (417 fields x 510,121 records = 265 Mb), but am only
>     interested in 7
>     >         fields.
>     >         > To ease the processing load, I deleted the unwanted fields in
>     PSPP,
>     >         but the
>     >         > resultant .sav file took over 90 minutes to save (9 Mb).   Are
>     there
>     >         any
>     >         > settings that could be adjusted to reduce the file save time ?
>     >       
>     >         Wow!  That's a long time.  I don't know, off-hand, why it would
>     >         take that long.  Is the disk active during that time?  Is PSPP
>     >         using a lot of CPU time?  Is this in the GUI or the command-line
>     >         syntax version?
>     >       
>     >         Increasing the workspace to a value larger than the size of your
>     >         file might help, e.g. SET WORKSPACE=300000.
>     >       
>     >         [cut]
>     >
>     > _______________________________________________
>     > Pspp-users mailing list
>     > address@hidden
>     > http://lists.gnu.org/mailman/listinfo/pspp-users
>    
>     --
>     "I knew it was a bad sign when I drove through Hell (MI) on Wednesday
>     and there was snow on the ground and all the standing water had a
>     layer of ice on the surface...  Yes folks, Hell has frozen over
>     and my cable modem *still* wasn't working..." --Ed Glowacki
>

-- 
"A computer is a state machine.
 Threads are for people who cant [sic] program state machines."
--Alan Cox