Re: [hfdb] Scope (Was Re: Grand Unified Hardware Database)

[Zenaan Harkness]

On Sat, 2004-07-24 at 03:12, David Zeuthen wrote:
> Hi,
> 
> On Thu, 2004-07-22 at 10:19 +1000, Zenaan Harkness wrote:
> > On Wed, 2004-07-21 at 16:54, James K. Lowden wrote:
> > > On Wed, 21 Jul 2004 Zenaan Harkness <address@hidden> wrote:
> > > > I have a device - a "Microsoft Natural Keyboard". I'm using it right
> > > > now. How do I add it to the database? 
> > > 
> > > That's really two questions: What data go in what tables, and how does one
> > > move the data into said tables.  
> > > 
> > > What do you know about this keyboard?  Then let's walk through where each
> > > piece belongs.  
> > 
> 
> My hunch feeling is that we need to consider each device type (printer,
> scanner, storage, camera, input, usb hub) in detail to look at what
> information is needed to store. 

Agreed.

> Second, I think it's good to partition this data into three categories:
> 
>  1. Data we extract from the hardware itself in order to uniquely
>     identify what we're dealing with; and
> 
>  2. Data that is useful for software using the hardware but cannot be
>     extracted from the hardware proper; and
> 
>  3. Metadata about the hardware, e.g. textual content consumed only by
>     human beings for the purpose of giving more information about the
>     keyboard

Useful for discussion purposes yes. The data will have a natural
physical representation as we enter it (obviously we will want to
clearly link related data).

> Note that some device types like USB hubs are pretty uninteresting for
> the HFDB to deal with since a) they always work with any OS with a
> working USB stack;

So I guess you'd define the X-Box as not having a working USB stack? :)

Of course, GNU/Linux on the xbox does support USB hubs.

>  and b) all interesting data can be extracted for the
> hardware itself. 
>
> That is not to say they shouldn't be included, we could keep them in the
> HFDB and track data e.g. what colour the hub is, but that might not be
> terribly useful :-)

I will personally only be entering useful data :)

> So, looking at my keyboard, I'd say:
> 
> Data that can be extracted:
> ===========================
> The USB product id is 0x058f, the USB product id is 0x9410 and the USB
> device revision is 0x0101. This uniquely identify my device. 
> 
> (In some cases, like printers, we need a bit more, like the IEEE1284 ID
> string, since vendors erroneously reuse the supposed to be unique keys I
> gave above.)
> 
> Data that is useful for software:
> =================================
> 
> My laptop got a nonstandard key on the keyboard is a button with a
> 'power off' symbol and the keycode is <some_kay_code_value> (I haven't
> looked it up yet). So, we might be interested in storing this and one
> day export it to HAL in this fashion
> 
>  input.keyboard.poweroff_button_keycode =  <some_key_code_value>
> 
> (Ideally we need to go through all available software and see where the
> effort of proving additional data that cannot be extracted from the
> hardware helps.)

If it pikes someone's interest. On the other hand, immediate utility
will tend to be the main driver for people entering data here. Although
once manufacturers start to come on board, we will see more
comprehensive data I'd think.

> (One example is the ACME applet for GNOME that enables the user to
> associate actions with multimedia keys on the keyboard - perhaps if the
> HFDI exposes data that says keycode 0x123 corresponds to the 'volume-'
> we can enhance that piece of software, ACME, so it 'just works' when
> plugging in the keyboard.)

I see what you're saying. This would be useful.

> Metadata
> ========
> 
> "The keyboard is a from Microconnectors and looks like this
> 
>  http://www.microconnectors.com/mackey.html
>  (Google cache: 
> http://66.102.9.104/search?q=cache:xtSLPOPDKrQJ:www.microconnectors.com/mackey.html++site:www.microconnectors.com+microconnectors&hl=en)
> 
> It also features two-port USB HUB."
> 
> > OK, I can say the following from my experience:
> > 
> > * It's a Microsoft Natural Keyboard (may be "... Pro").
> 
> This we can usually probe from the hardware.
> 
> > * It's a US-English style layout, I think 104 keys (Windows + Menu keys)
> 
> Specifically for keyboards, this is difficult to probe from the
> hardware; users can also switch language layouts so I think this is not
> important to represent (of course, if we can represent it, great, let's
> do it).
> 
> > * It's got 16 special-function keys across the top, above the Fn keys:
> >  - back
> >  - forward
> >  - stop
> >  - refresh
> >  - search
> >  - favourites
> >  - web/home
> >  - mail
> >  - mute
> >  - volume -
> >  - volume +
> >  - play/pause
> >  - stop
> >  - prev track
> >  - next track
> >  - media
> > * It's got 3 special function keys above the keypad:
> >  - my computer
> >  - calculator
> >  - sleep
> 
> We should store these keycodes as category 2 data.

I assume you mean by "category 2", "less important"?

Would we actually categorize like this in the actual physical
representation, and if so, how is that useful?

> > * It has three led lights for num-lock, caps-lock and scroll-lock.
> 
> Pretty standard for a keyboard, but might be useful to export as
> category 2 data.
> 
> > * It has a cable that splits near the computer end, with PS/2 and USB
> > connectors.
> > * It has a 2 port USB hub built in.
> 
> This is category 3 data.

Sorry, you'll have to explain what you mean by categories here. Perhaps
"even less useful"?

> > * It has a serial number: 5157703025042, and other stickers underneath.
> 
> Probably not generic enough to be useful :-)
> 
> > * I've been able to make Linux consoles and X work with it.
> > * The PS connector works under Linux kernels 2.4.[246].
> > * I've been able to make it work using the USB connector with 2.4.4.
> 
> Specifically USB HID and PS2 keyboard/mouse works with virtually every
> operating with USB respectively PS2 support. However, for the sake of
> argument (since it sadly applies to other hardware) we should store this
> somehow.

I think James put it clearly - we store which kernels have USB stacks,
and what features those stacks (and input drivers) support, and the
keyboard has physical connectors - if at least one of the physical
connectors is USB, then we can cross match this against kernels that
have USB/input drivers that support keyboards and say that "yes it's
supported, but no the multimedia keys" etc.

> > * I haven't tried to use the special "multimedia keys".
> 
> ACME or xev would be helpful here :-)

thanks

> I think what we should next is to come up with a schema design for the
> database. Perhaps it need not to be more complex that this back-of-the-
> envelope design?

I guess we will need that web page soon. Anyway, "take one" from a few
weeks back is here:

http://www.schemamania.org/projects/chd/chd.sql

http://www.schemamania.org/projects/chd/chd.dia

Of course, with all the discussions of XML with RMS, we might have to
look into XML import filter or something. I hope to do enough reading on
it this week.

Which actually leads me to think, perhaps Richard didn't realise that we
already have a pretty good start on an SQL database... hmmm.

> We have a table for each physical bus type that contains the properties
> we can extract from the hardware. 
> 
>  USBDevice
> 
>  IEEE1394Device
> 
>  ...

This is a good example to start - some devices (iPod?, printers,
external HDDs) have both USB and firewire connections.

James anticipated this in the schema he put together, and created a
DeviceConnectors table (again, we can do this in SQL or XML files, and
in fact the work of defining the schema is useful regardless - we should
be using a proper design, whatever the data entry method).

> The set of properties that make the device unique constitute the PRIMARY
> key. This is category 1 data, there shouldn't be any NULL elements.

Yes. We had a bit of discussion about primary keys. In James' SQL
version of the schema (the only version we have at present), we are
using an ID field for each device, constituting its primary key.

If we enter data into XML files, we would either leave such id
generation to a db on import, or else allocate "ranges" to each person
creating such files. I think leaving it to the point it enters the DB
would be the better approach, no discontiguous ID ranges. However once
we start to settle down on the various "lookup" tables (ie. their
contents is entered/ enumerated, such as "device types", "connector
types", etc), then whoever is creating data files can make sure to
either use the correct id for the type, or use the right name for that
type, so as to minimize/ eliminate duplicates.

(And now I see where you're getting at with "categories" :)

> For each device type we have tables specific to each device type
> 
>  Keyboard
> 
>  Mouse
> 
>  Storage
> 
>  Printer
> 
>  PortableAudioPlayer
> 
>  ...

I note in the current schema, there is a table "MechanismTypes" with a
comment saying "laser, inkjet, dot matrix". I'm thinking it would make
sense just to call this table "Printer types" or similar. James?

> with category 2 data. Each table got FOREIGN keys into the tables from
> a. The PRIMARY key is a combination of said FOREIGN keys and other
> useful identifying data (such as IEEE1284 ID for printers).

Perhaps this is one of the reasons that IDs instead of (primary fields)
are used as primary key - it minimizes the duplication of data.

On the other hand, having such "verbose" PKs makes manually looking
through the tables an easier job. I like to have such manual "backup"
mechanisms possible, especially if the DB ensures that the PK and other
constraints are maintained. James?

> Many values here can be NULL; for Keyboard we probably want an elements
> for each keycode for multimedia keys but for my example above only one
> of them will contain useful data.

I think that proper normalization gets rid of most NULL values. You
simply have these "types" tables for various devices. But perhaps I
don't understand exactly what you're saying here - can you please
elaborate?

> Finally we tie everything together in a Device table where that
> references rows from the tables above (Plural since a device may have
> several capabilities (multi-function printers) e.g. three USB
> interfaces, one for printing, scanning and storage (card reader)). In
> the Device table we also store category 3 data.

It might be a good idea to look at the current schema, and go from there
actually, that way we can hone it with your input as well.

> Now, this is by design extensible - we can go ahead and add an element
> to the Keyboard table if some vendor comes out with a keyboard with a
> new multimedia key.

exactly

> The design also lends itself to data submission; I mean once we have the
> design in place we can easily specify what format we like data in. How
> this data is submitted doesn't really matter right now - initially we
> can write this in emails to Zenaan or define a really simple XML format
> or something.

This is a good point. I think it should always be simple for people to
submit data, including those (semi-)permanently "on the team".

> I'm not a big database design myself, it looks like James is, so any
> comments? How should we go about specifying the database schema?

If we are to enter data in XML files, we need to just do so, submitting
the (XML tagged) data to this list, and then we discuss what fields go
where, and start to create some types lists, that we can use for fields,
as well as XML tag names. It should be pretty straightforward I imagine.

> Eventually I'd like to automate this as part of the Project Utopia bits
> we're working on for GNOME. The thing is, the HAL project was
> specifically designed this way - we basically got all the necessary
> category 1 bits in place and (very few) HAL device information files for
> the category 2 bits 

Sounds good. If you didn't do so above, can you please explain
categories in a bit more detail for the benefit of our list.

> (such as this particular card reader is for compact flash so we export a
> property called storage.media=compact_flash and actually GNOME-VFS takes
> advantage of this and renders a nice icon, see 
> http://freedesktop.org/~david/gnome-vfs-take3-2.png).

This raises an interesting point. In XML we have "tags" and
"properties". Both can be used, almost interchangably, at least in many
cases. So then it becomes a question of what makes most sense. I guess
this will pan out as we start to chew over actual data...

> We can also start really slow and look at one device at a time, I
> suggest we start looking at necessary category 1 and 2 data for storage
> devices and portable audio players.

By all means, submit some actual examples, and we'll get underway.

> Once we have successfully shown that we can do this for the ''soft
> properties'' e.g. data used only in the desktop to e.g. render icons
> (Compact Flash) or functionaly (e.g. Portable Audio Player) we can move
> into the more hardcore things such as 'Does this device work on the
> Linux kernel X.Y.Z' or use the 'FooBar' driver for FreeBSD 4.x and
> 'BazBat' driver on GNU/Hurd.

Sounds fine.

> With regards to the concerns raised by Richard of aiming too high and
> starting slow, I don't know. Thing is, we really need all the salient
> features on an RDBMS system as James and Zenaan have pointed out. And if
> my back-of-the-envelope database design above is just somewhat correct
> it shouldn't be that difficult to implement IMHO.

I agree. I feel that my main limitation is my lack of XML experience
right now, but I can see the benefits of simple file-based data entry.
I'll go do some reading this week.

> Another thing, is that I think there is also a demand for the HFDB since
> GNOME have stated in their roadmap that they are interested in utilizing
> HAL (as an optional dependency) and thus the information that the HFDB
> can provide. And I think that is great, so let's go ahead and build it!

Thanks for sharing that. Gives a little extra impetus. I imagine that
many projects, obviously including the various desktops, will be able to
gain significant advantages from the HAL/HFDB combination.

Thanks heaps for jumping on board and contributing,
Zenaan