[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Import large field-delimited file with strings and numbers

From: CdeMills
Subject: Re: Import large field-delimited file with strings and numbers
Date: Mon, 8 Sep 2014 10:53:58 -0700 (PDT)

Joao Rodrigues wrote
> I need to import a large CSV file with multiple columns with mixed 
> string and number entries, such as:
> field1, field2, field3, field4
> A,        a,        1,       1.0,
> B,        b,        2,        2.0,
> C,        c,        3,        3.0,
> and I want to pass this on to something like
> cell1 ={[1,1] = A; [2,1] = B; [3,1] = C};
> cell2 ={[1,1] = a; [2,1] = b; [3,1] = c};
> arr3 =[1 2 3]';
> arr4 =[1.0 2.0 3.0]';
> furthermore, some columns can be ignored, the total number of entries is 
> known and there is a header.
> How can I perform the import within reasonable time and little memory 
> overhead? Below are a few of my attempts.
> Octave offers a wide range of functions to import files (csvread, 
> dlmread, textscan, textread, fscanf, fgetline) but as far as I can tell 
> none seems to get the job done.
> csvread and dlmread don't work because they only handle numerical data.
> textscan works eats up all the memory (the file is 200 MB, textscan's 
> memory usage was into the GB's). It doesn't allow to provide a priori 
> the size of the object.

Hello Joaõ,
this is the perfect job for the "dataframe" package. It reads a CSV file and
returns an object which try to mimics a matrix. You can select ranges,
perform arithmetic operations, ...

You'll find in on the octave-forge source.



View this message in context:
Sent from the Octave - General mailing list archive at

reply via email to

[Prev in Thread] Current Thread [Next in Thread]