Re: Loading a large and unusually formatted dataset into an Octave matrix

[Helios de Rosario]

> Message: 1
> Date: Tue, 18 Jun 2013 14:01:29 -0400
> From: Przemek Klosowski <address@hidden>
> Subject: Re: Loading a large and unusually formatted dataset into an
>       Octave  matrix
> On 06/14/2013 02:04 PM, Elliot Gorokhovsky wrote:
>> > Hello! I am a new octave user and I am trying to predict the price
of
>> > bitcoins 15 minutes in advance via neural networks for use on the
>> > website btcoracle.com <http://btcoracle.com>
<http://btcoracle.com>.
>> I have about a gigabyte of
>> > data that looks like this:
> 
>
{"data":1279408157,"price":"0.04951","amount":"20","price_int_":"4951","amou
>
nt_int":"2000000000","tid":"1","price_currency":"USD","item":"BTC","trade_typ
> e":""},
>
{"data":1279424586,"price":"0.05941","amount":"50.01","price_int_":"5941","a
>
mount_int":"5001000000","tid":"2","price_currency":"USD","item":"BTC","trade_
> type":""},
>
{"data":1279475336,"price":"0.08080","amount":"5","price_int_":"8080","amoun
>
t_int":"500000000","tid":"3","price_currency":"USD","item":"BTC","trade_type"
> :""}


This looks like a JSON file. You can parse with JSONlab:
http://iso2mesh.sourceforge.net/cgi-bin/index.cgi?jsonlab

Just enclose your data between square brackets "[ ]" so that the
function "loadjson" understands them as a vector of JSON objects, and
you'll get a struct array you can work with. For instance, if your file
named "db.json" is:

[
{"data":1279408157,"price":"0.04951","amount":"20","price_int_":"4951","amount_int":"2000000000","tid":"1","price_currency":"USD","item":"BTC","trade_type":""},
{"data":1279424586,"price":"0.05941","amount":"50.01","price_int_":"5941","amount_int":"5001000000","tid":"2","price_currency":"USD","item":"BTC","trade_type":""},
{"data":1279475336,"price":"0.08080","amount":"5","price_int_":"8080","amount_int":"500000000","tid":"3","price_currency":"USD","item":"BTC","trade_type":""}
]

Then do:

> db = loadjson("db.json");
db =

  1x3 struct array containing the fields:

    data
    price
    amount
    price_int_
    amount_int
    tid
    price_currency
    item
    trade_type

You can get each column as a vector or a cell array, e.g.
> [db.data]
ans =

  1.2794e+009  1.2794e+009  1.2795e+009

> {db.price}
ans =
{
  [1,1] = 0.04951
  [1,2] = 0.05941
  [1,3] = 0.08080
}

Some of your data are strings, so you may want transform them into
numeric values. cellfun may come in handy:

> cellfun(@str2num, {db.price})
ans =

   0.049510   0.059410   0.080800

Regards,
Helios

PS. Some other JSON parsers may not be happy with that structure,
lacking a root object. You may want add such a root object to your
file:

{
"root" :
[
{"data":1279408157,"price":"0.04951","amount":"20","price_int_":"4951","amount_int":"2000000000","tid":"1","price_currency":"USD","item":"BTC","trade_type":""},
{"data":1279424586,"price":"0.05941","amount":"50.01","price_int_":"5941","amount_int":"5001000000","tid":"2","price_currency":"USD","item":"BTC","trade_type":""},
{"data":1279475336,"price":"0.08080","amount":"5","price_int_":"8080","amount_int":"500000000","tid":"3","price_currency":"USD","item":"BTC","trade_type":""}
]
}

In that case, loadjson will give you a struct with a field "root",
which will contain the struct array previously mentioned.

INSTITUTO DE BIOMECÁNICA DE VALENCIA
Universidad Politécnica de Valencia • Edificio 9C
Camino de Vera s/n • 46022 VALENCIA (ESPAÑA)
Tel. +34 96 387 91 60 • Fax +34 96 387 91 69
www.ibv.org

  Antes de imprimir este e-mail piense bien si es necesario hacerlo.
En cumplimiento de la Ley Orgánica 15/1999 reguladora de la Protección
de Datos de Carácter Personal, le informamos de que el presente mensaje
contiene información confidencial, siendo para uso exclusivo del
destinatario arriba indicado. En caso de no ser usted el destinatario
del mismo le informamos que su recepción no le autoriza a su divulgación
o reproducción por cualquier medio, debiendo destruirlo de inmediato,
rogándole lo notifique al remitente.