Re: [Userops] Userops Acid Test v0.1

[Asheesh Laroia]

On Sun, Sep 27, 2015 at 9:20 AM, Christopher Allan Webber <address@hidden> wrote:

Hello all,

I thought it would be useful for me to write up, in an
implementation-agnostic form, a clear example of what I think
(personally) are a set of useful requirements for a Userops type system.
So here's a braindump of what I see are essential properties of such a
system...

  http://dustycloud.org/blog/userops-acid-test/

I wrote this up independently, and I'm sure I'm missing some things.
Consider this Userops Acid Test v0.1?

FWIW the original Acid Test gives users a very quick binary: The thing looks visually correct, or it doesn't.

You can look at http://acid2.acidtests.org/#top and http://www.w3.org/Style/CSS/Test/CSS1/current/test5526c.htm to get a sense of that.

So I propose that a userops Acid Test should have a boolean answer, even if the answer ends up being "No for a stupid reason" for one system or another. We can all live with that. I think this was an issue with Acid Test 1, but I think that everyone lived with it.

 

Here's the bulleted list version, minus the descriptions, if anyone so
cares:

 - Free as in Freedom
 - Reproducible
   + Reproducible packages
   + Reproducible systems
 - Recoverable
   + Recoverable data
   + Recoverable system
 - Friendly GUI
 - Scriptable
 - Collaboration friendly
 - Fleet manageable
 - Secure

Given a list like this, it's not clear what a consumer/chooser of a userops-y system should do. I think that the best option is to ask people to write _reviews_ of userops-y systems, similar to https://tosdr.org/ and use the above as evaluation criteria.

So maybe s/acid test/review guidelines/ and I like your proposal.

 

Yeah I know it's not really a clear pass/fail type test!  But I think
it may be useful as a set of properties to measure a system against.
Some of these are vague and can be interpreted in many
ways... especially that last one! :)

As for the content of the post:

> Free as in Freedom:

+1 to this content

> Reproducible

+0.5

Seems reasonable. Debian and Fedora don't meet this criterion yet, and I don't know any Linux distro that does, so it's not exactly clear what to do about it if you're looking for a boolean.

But if you're looking to make guidelines for people, as they write subjective reviews of something, similar to tos;dr-type thing, I think this is OK.

>  Recoverable

+1

> Friendly GUI

You write: "This probably should be optional; there may be lower level interfaces to the deployment system that some users would prefer to use"

FWIW I am +1 on saying, "If you have to do it by CLI, it doesn't exist." I'm OK with the wording here, though, but I just want to make that remark while reviewing this.

> Scriptable

I _think_ "Create a new Sandstorm package to achieve your desired task" meets these criteria, as well as the Cap'n Proto RPC interfaces that the platform provides, so OK.

Let me know if you don't think so; if so, I might not understand this.

> Collaboration friendly

+1, so long as you would agree "the community of Sandstorm packagers" is a "community" that is collaborating to create things for people.

In particular, you write: "Users should be able to help each other out and share "recipes" of deployment steps"

If there are "deployment steps" that aren't GUI-based then (IMHO) the system is not ready for a userops seal of approval.

> Fleet manageable

It seems like "Install Sandstorm on two different machines, then use backup & restore, addresses this well enough. Let me know if you don't think so; if you don't think so, I might misunderstand.

For me, "fleet' is where things stop being "userops"-y and start being "devops"-y, so I want to caution against emphasizing "fleet management" in a "userops" type thing. In the 13 years I've run a personal server, I've never needed to scale it to more than one machine.

> Secure

Sadly very vague.

If I wanted to be somewhat cruel but realistic on this, I'd say: "If the system makes plaintext on the Internet the default, then it's not secure" and "If the system does not support two-factor authentication, then it's not secure" and "If a compromised app can break into other apps and read their data, then it's not secure".

Naturally, Sandstorm meets the above security criteria (admittedly by outsourcing 2FA to Google or GitHub, though it's customizable). I think "making a good effort" on security is very very different from "is secure", so you should pick which one you want to aim for.

I hope you aim for "secure", but I can live with whatever.

Also I think the only criteria Sandstorm fails at, right now, is "recoverable", which I do intend to work on.

-- Asheesh.