Be careful what you ask for Daniel Silverstone
Date: Wed, 01 Jul 2015 06:13:16 -0000
From: 123-reg <noreply@123-reg.co.uk>
To: dsilvers@digital-scurf.org
Subject: Tell us what you think for your chance to win
X-Mailer: MIME::Lite 3.027 (F2.74; T1.28; A2.04; B3.13; Q3.13)

Tell us what you think of 123-reg!

<!--

.style1 {color: #1996d8}

-->

Well 123-reg mostly I think you don't know how to do email.

In defence of curl | sudo bash - Daniel Silverstone

Long ago, in days of yore, we assumed that any software worth having would be packaged by the operating system we used. Debian with its enormous pile of software (over 20,000 sources last time I looked) looked to basically contain every piece of free software ever. However as more and more people have come to Linux-based and BSD-based systems, and the proliferation of *NIX-based systems has become even more diverse, it has become harder and harder to ensure that everyone has access to all of the software they might choose to use.

Couple that with the rapid development of new projects, who clearly want to get users involved well before the next release cycle of a Linux-based distribution such as Debian, and you end up with this recommendation to bypass the operating system's packaging system and simply curl | sudo bash -.

We, the OS-development literati, have come out in droves to say "eww, nasty, don't do that please" and yet we have brought this upon ourselves. Our tendency to invent, and reinvent, at the very basic levels of distributions has resulted in so many operating systems and so many ways to package software (if not in underlying package format then in policy and process) that third party application authors simply cannot keep up. Couple that with the desire of the consumers to not have their chosen platform discounted, and if you provide Debian packages, you end up needing to provide for Fedora, RHEL, SuSE, SLES, CentOS, Mint, Gentoo, Arch, etc.etc; let alone supporting all the various BSDs. This leads to the simple expedience of curl | sudo bash -.

Nobody, not even those who are most vehemently against this mechanism of installing software, can claim that it is not quick, simple for users, easy to copy/paste out of a web-page, and leaves all the icky complexity of sorting things out up to a script which the computer can run, rather than the nascent user of the software in question. As a result, many varieties of software have ended up using this as a simple installation mechanism, from games to orchestration frameworks - everyone can acknowledge how easy it is to use.

Now, some providers are wising up a little and ensuring that the url you are curling is at least an https:// one. Some even omit the sudo from the copy/paste space and have it in the script, allowing them to display some basic information and prompting the user that this will occur as root before going ahead and elevating. All of these myriad little tweaks to the fundamental idea improve matters but are ultimately just putting lipstick on a fairly sad looking pig.

So, what can be done? Well we (again the OS-development literati) got ourselves into this horrendous mess, so it's up to us to get ourselves back out. We're all too entrenched in our chosen packaging methodologies, processes, and policies, to back out of those; yet we're clearly not properly servicing a non-trivial segment of our userbase. We need to do better. Not everyone who currently honours a curl | sudo bash - is capable of understanding why it's such a bad idea to do so. Some education may reduce that number but it will never eliminate it.

For a long time I advocated a switch to wget && review && sudo ./script approach instead, but the above comment, about people who don't understand why it might be a bad idea, really applies to show how few of those users would even be capable of starting to review a script they downloaded, let alone able to usefully judge for themselves if it is really safe to run. Instead we need something better, something collaborative, something capable of solving the accessibility issues which led to the curl | sudo bash - revolt in the first place.


I don't pretend to know what that solution might be, and I don't pretend to think I might be the one to come up with it, but I can hilight a few things I think we'll need to solve to get there:

  1. Any solution to this problem must be as easy as curl | sudo bash - or easier. This might mean a particular URI format which can have os-specific ways to handle standardised inputs, or it might mean a pervasive tool which does something like that.
  2. Any solution must do its best to securely acquire the content the user actually wanted. This means things like validating SSL certificates, presenting information to the user which a layman stands a chance of evaluating to decide if the content is likely to be what they wanted, and then acting smoothly and cleanly to get that content onto the user's system.
  3. Any solution should not introduce complex file formats or reliance on any particular implementation of a tool. Ideally it would be as easy to implement the solution on FreeBSD in shell, or on Ubuntu as whizzy 3D GUIs written in Haskell. (modulo the pain of working in shell of course)
  4. The solution must be arrived at in a multi-partisan way. For such a mechanism to be as usefully pervasive as curl | sudo bash - as many platforms as possible need to get involved. This means not only Debian, Ubuntu, Fedora and SuSE; but also Arch, FreeBSD, NetBSD, CentOS etc. Maybe even the OpenSolaris/Illumos people need to get involved.

Given the above, no solution can be "just get all the apps developers to learn how to package software for all the OS distributions they want their app to run on" since that way madness lies.

I'm sure there are other minor, and major, requirements on any useful solution but the simple fact of the matter is that until and unless we have something which at least meets the above, we will never be rid of curl | sudo bash - :- just like we can never seem to be rid of that one odd person at the party, noone knows who invited them, and noone wants to tell them to leave because they do fill a needed role, but noone really seems to like.

Until then, let's suck it up and while we might not like it, let's just let people keep on curl | sudo bash -ing until someone gets hurt.


P.S. I hate curl | sudo bash - for the record.

Sometimes recruiters really miss the point... Daniel Silverstone

I get quite a bit of recruitment spam, especially via my LinkedIn profile, but today's Twitter-madness (recruiter scraped my twitter and then contacted me) really took the biscuit. I include my response (stripped of identifying marks) for your amusement:

On Tue, Jun 09, 2015 at 10:30:35 +0000, Silly Recruiter wrote:
> I have come across your profile on various social media platforms today and
> after looking through them I feel you are a good fit for a permanent Java
> Developer Role I have available.

Given that you followed me on Twitter I'm assuming you found a tweet or two in
which I mention how much I hate Java?

> I can see you are currently working at Codethink and was wondering if you
> were considering a change of role?

I am not.

> The role on offer is working as a Java Developer for a company based in
> Manchester. You will be maintaining and enhancing the company's core websites
> whilst using the technologies Java, JavaScript, JSP, Struts, Hibernate XML
> and Grails.

This sounds like one of my worst nightmares.

> Are you interested in hearing more about the role? Please feel free to call
> or email me to discuss it further.

Thanks, but no.

> If not, do you know someone that is interested? We offer a £500 referral fee
> for any candidate that is successful.

I wouldn't inflict that kind of Lovecraftian nightmare of a software stack on
anyone I cared about, sorry.

D.

I then decided to take a look back over my Twitter and see if I could find what might have tripped this. There's some discussion of Minecraft modding but nothing which would suggest JavaScript, JSP, Struts, Hibernate XML or Grails.

Indeed my most recent tweet regarding Java could hardly be construed as positive towards it.

Sigh.

Kimchi Trial 1 - Part 3 - Kimchiguk Daniel Silverstone

Earlier this week I allowed some of my colleagues at work to try the Kimchi and one of them (James) liked it enough to ask for a tub of it just for himself. Being a lovely person I obliged.

My darling Rob decided that what he really wanted was Kimchiguk or Kimchijijae. Being both wonderful and lacking in ingredients, I set to do achieving something.

I read a few recipes, worked out that I lacked critical ingredients and so set about simulating Kimchiguk. I grabbed two and a bit cupfuls of kimchi from the box in the fridge, chopped it up more finely, popped it into a pot along with about a pound of cubed pork belly, two teaspoons of hot pepper flakes, two teaspoons of sugar, and two teaspoons of cornflour. I then topped that off with five cups of water, mixed it up, brought it to the boil and simmered it for 30 minutes. Then I cubed some tofu, popped that in and simmered it for another 10 minutes while some rice cooked. Served in a bowl over rice, my approximation of Kimchiguk turned out pretty well. (Note it fed both of us and there was a serving left over.)

Kimchiguk - Kimchi soup

With a do-over, I'd try harder to get pork shoulder (belly is quite fatty) and given the option I'd wait until I had hot pepper paste because the malty fermented sweetness of hot pepper paste is pretty much impossible to emulate otherwise.

Kimchi Trial 1 - Part 2 Daniel Silverstone

Last night we tried the small amount of 'fresh kimchi' from my epic kimchi making trial.

I marinaded some beef strips in a blended paste of onion, garlic, ginger, and sesame oil for a few hours before frying that up, and serving on rice with stir-fried carrots, leek and spring onion. A little fresh kimchi on the side for flavour.

Delicious dinner

I think the Kimchi was a success. Dearest Rob said that it tasted of Kimchi. The flavours were not very well combined and quite clearly underdeveloped, and the pepper flakes were still a little hard. I expect this to resolve over the next few days as it begins to ferment.

All in all, experiment going well, expect part 3 when I know how it tastes after fermentation begins.

Kimchi Trial 1 - Part 1 Daniel Silverstone

I have spent today making my first ever batch of Kimchi. I have been documenting it in photos as I go, but thought I'd write up what I did so that if anyone else fancies having a go too, we can compare results.

For a start, this recipe is nowhere near "traditional" because I don't have access to certain ingredients such as glutinous rice flour. I'm sure if I searched in many of the asian supermarkets around the city centre I could find it, but I'm lazy so I didn't even try.

I am not writing this up as a traditional recipe because I'm kinda making it up as I go along, with hints from various sources including the great and wonderful Maangchi whose YouTube channel I follow. Observant readers or followers of Maangchi will recognise the recipe as being close to her Easy Kimchi recipe, however since I'm useless, it won't be exact. If this batch turns out okay then I'll write it up as a proper recipe for you to follow.

I started off with three Chinese Leaf cabbages which seemed to be about 1.5kg or so once I'd stripped the less nice outer leaves, cored and chopped them.

Chopped up cabbage

I then soaked and drained the cabbage in cold water...

Soaking cabbage

...before sprinkling a total of one third of a cup of salt over the cabbage and mixing it to distribute the salt.

Salted Cabbage

Then I returned to the cabbage every 30 minutes to re-mix it a total of three times. After the cabbage had been salted for perhaps 1h45m or so, I rinsed it out. Maangchi recommends washing the cabbage three times so that's what I did before setting it out to drain in a colander.

Drained salted cabbage 1h45m later

Maangchi then calls for the creation of a porridge made from sweet rice flour which it turns out is very glutinous. Since I lack the ability to get that flour easily I substituted cornflour which I hope will be okay and then continued as before. One cup of water, one third of a cup of cornflour was heated until it started to bubble and then one sixth of a cup of sugar was added. Stirred throughout, once it went translucent I turned the heat off and proceeded.

Porridge thingy

One half of a red onion, a good thumb (once peeled) of ginger, half a bulb of garlic and one third of a cup of fish sauce went into a mini-zizzer. I then diagonal-chopped about five spring onions, and one leek, before cutting a fair sized carrot into inch long pieces before halving and then thinly slicing it. Maangchi calls for julienned carrots but I am not that patient.

Veggybits

Into the cooled porridge I put two thirds of a cup of korean hot pepper flakes (I have the coarse, but a mix of coarse and fine would possibly be better), the zizzed onion/garlic/ginger/fish mix and the vegetables...

All in the pan

...before mixing that thoroughly with a spatula.

Mixed vegetables

Next came the messy bit (I put latex gloves on, and thoroughly washed my gloved hands for this). Into my largest mixing bowl I put a handful of the drained cabbage into the bowl and a handful of the pepper mix. Thoroughly mixing this before adding another handful of cabbage and pepper mix I repeated until all the cabbage and hot pepper mixed vegetables are well combined. I really got your arms into it, squishing it around, separating the leek somewhat, etc.

Bowl of kimchi

As a final task, I scooped the kimchi into a clicklok type box, pressing it down firmly to try and remove all the air bubbles, before sealing it in for a jolly good fermenting effort. I will have to remove a little tonight for our dinner (beef strips marinated in onion, ginger and garlic, with kimchi on rice) but the rest will then sit to ferment for a bit. Expect a part-2 with the report from tonight's dinner and a part-3 with the results after fermentation.

Box of kimchi


As an aside, I got my hot pepper flakes from Sous Chef who, it turns out, also stock glutinous rice flour -- I may have to get some more from them in the future. (#notsponsored)

Caius -- A heirarchical delegable password safe Daniel Silverstone

A long while ago I, Rob Kendrick, Clive Jones (and possibly others) sat down and tried to come up with a way to store passwords a-la Password Safe. However, being us, we wanted to ensure a number of properties which password safes commonly don't have. We wanted to allow the delegation of access to some subset of the passwords. We also wanted for it to be reasonable to deny that there is content which has not been decrypted.

I was reminded of this work when I was discussing the concept of deniable storage of secrets with a colleague (An idea I'll expand upon in another blog post at another time). I am therefore presenting, with little change other than formatting the design from years ago. I would be very interested if anyone knows of software which meets the properties of the Caius system since I would like to have one but simply don't trust myself (see another future posting) to write it right now.


Caius

The following concepts are assumed to be understood:

The 'Caius' system is a password-safe type system sporting hierarchical delegable access to the data it stores. The 'Caius Fob' is the data-store for the system.

The 'Caius Fob' is a file which consists of a header and then three sections. The header identifies it as such a file, the first section lists a number of 'External IDs' which can be used to access portions of the file. The second section lists ACL entries as defined below. The third section of the file is the encrypted data looked after by this file. It is not intended that the holder of a CaiusFob be able to deny it is a CaiusFob, but it is expected that it be possible to deny an ability to decrypt (perhaps by lacking a password) any ACL entries. Given that the structure of the file is known, it is necessary that there be external IDs for which the password or GPG key is not valid or cannot decrypt an ACL entry, and ACL entries which even if decrypted may not be valid, and ACL entries which even if decrypted and valid may not be used to encode any data blocks.

An External ID

External ID ::=
   LENGTH
   TYPE
   DATA

Where TYPE is one of: * 0: Unused (ID slot placeholder) * 1: GPG key, where DATA is the keyid of the gpg key * 2: Password, where DATA is some hash of a password which can be used to derive a key for decrypting an ACL entry.

The list of external ids forms a numbered sequence where the index (0-based) into the sequence is the External ID number. (EIDnr)

An ACL Entry

ACL Entry ::=
   LENGTH
   EIDnr
   DATA
   HMAC

The EIDnr is the number of the External ID as explained above. The LENGTH is the length of the DATA section which is a key-pair as explained below, encrypted to the external id. The HMAC uses the authentication key in the key-pair in the DATA section, and authenticates the EIDnr, LENGTH, DATA tuple.

One possibility for increasing deniability is to remove the EIDnr from this part of the file, and assume that for every external ID you try to decrypt all ACLs you've not succeeded in decrypting thus-far. This has the benefit of being able to deny that an ACL entry ought to be decryptable with the credentials you hold, but also an increased inability to know if you have successfully unlocked everything appropriate to being able to fully manipulate a CaiusFob. This tradeoff is currently set in favour of better understanding of the content, but a future design feature might suggest EIDnr should always be -1 to indicate "unknown, try every EID".

A key pair

Key Pair ::=
   ENCRYPTIONKEY
   AUTHENTICATIONKEY

The ENCRYPTIONKEY is used to initialise the stream cipher for the data section. The AUTHENTICATIONKEY is used to compute HMACs for the appropriate ACL entries or data blocks (as defined below)

The data section

First consider a set of stream ciphers. There exists always one cipher which we will call the NULL cipher. It is defined such that Cipher(Byte, Offset) == Byte and is always available. Then there is a cipher initialised for each key pair we can successfully extract from the ACL entry section of the file.

Each of these ciphers is initialised and made ready such that the data section can be xored with the bytes as they come out of the stream ciphers in an attempt to decrypt the blocks. Essentially this is equivalent to decrypting the entire data section with each cipher in turn to produce N proposed cleartexts which can then be examined to find decrypted blocks.

Whenever a cipher, combined with the data stream in the file, manages to produce a sequence of eight bytes of value zero, we have reached a synchronisation point and what follows is a data block enciphered with which ever cipher managed to reveal the synchronisation sequence.

Since it is vanishingly unlikely that you will find eight zeros in a row when playing about with arbitrary cipher initialisation, we consider this to be an acceptable synchronisation sequence.

Once we have found a sync sequence, we can know the length of this block and thus we do not look for sync markers again until after the block we have just found.

A data block

Data block ::=
   DATALENGTH
   PADLENGTH
   TYPE
   DATA
   PAD
   HMAC

Such that each field is the obvious, DATA is DATALENGTH bytes of data, the texture of which is defined by TYPE and PAD is PADLENGTH arbitrary bytes which pad this block. HMAC is keyed using the authentication key associated with the stream cipher managing to decrypt this and is over the DATALENGTH, PADLENGTH, TYPE, DATA, PAD tuple.

If TYPE is zero then this is a ''free-space'' block and will typically contain zero bytes of DATA and some number of padding. This is however arbitrary and not enforced, the free space can be DATA if the implementer prefers and implementations are encouraged if possible to randomise the distribution of the consumed space between the DATA and PAD sections.

A node block

TYPE == 1 (Node)
DATA ::=
   MY_ID
   PARENT_ID
   NAME
   NOTES

MY_ID is a unique ID for this node. (generally a random number, probably 64 bits long, perhaps a UUID). PARENT_ID if not all NULLs is the ID for the parent node. If all NULLs then this is the root of a heirarchy. NAME is a NULL terminated byte string of one or more characters which is the name of this node. It may consist only of any characters other than NULL and the forward-slash character. NOTES is a byte string of zero or more characters, NULL terminated. Note that the DATALENGTH of the data block clearly delimits this field but the NULL is present to aid parsing.

A system block

TYPE == 2 (System)
DATA ::=
   PARENT_ID
   USERNAME
   PASSWORD
   EXPIRYDATE
   NOTES

PARENT_ID is the node to which this block belongs. It is required that any system blocks you succeed in decrypting can be placed within a node you succeed in decrypting. If the library encounters a system block which belongs to a node it cannot find then this is considered to be a corrupt system block and will be treated as though it could not be decrypted.

USERNAME is a byte string of one or more characters terminated by a NULL, ditto for PASSWORD and as for a node block, the NOTES are NULL terminated also.

EXPIRYDATE is a byte string of characters in RFC-822 date format.

Implementation notes

It is expected that any implementation of Caius will adhere to the following guidelines in order to enhance the security of content over time.

  1. Any time a block is invalidated (such as by the changing of a password, the obsoleting of an entry, the changing of notes, names, or reparenting a node) anywhere from one to all of the bits in the block can be changed. Trivially, this includes the synchronisation sequence preceeding the block as if the synchronisation sequence isn't present then the block does not exist.

  2. Every time a CaiusFob is altered in any way, some amount of known intra-block padding must be altered in a random way. Ideally this will be done so that it looks like number 1 has happened somewhere else in the file as well. Anywhere from zero to all of the padding can be thusly altered in any given change.

  3. No attempt will be made to write to any part of the file which cannot be positively identified as padding unless the user has explicitly stated that they will accept damage to data they cannot currently decrypt.

  4. No indication will be given to the user if any part of the file was unable to be decrypted or failed an HMAC check. Such data is simply incorrectly decrypted and thus ignored.

  5. Intrablock padding can be positively identified if you have two consecutive blocks in a CaiusFob such that the number of bytes between them could not possibly hold the simplest of free space blocks.

  6. When appending a block to a CaiusFob it is encouraged to place up to 50% of the size of the intrablock spacing before it as random padding, and up to 50% afterwards also. Naturally anywhere between zero and the full amount is acceptable, ideally the implementation would choose at random each time.

I promise that I... Daniel Silverstone

A friend and ex-colleague Francis Irving (@frabcus on Twitter) has recently been on a bit of an anti C/C++ kick, including tweeting about the problems which happen in software written in so called "insecure" languages, and culminating in his promise website which boldly calls for people to promise to not use C/C++ for new projects.

Recently I've not been programming enough. I'm still a member of the NetSurf browser project, and I'm still (slowly) working on Gitano from time to time. I am still (in theory) an upstream on the Cherokee Webserver project (and I really do need to sit down and fix some bugs in logging) and any number of smaller projects as well. I am still part of Debian and would like to start making positive contributions beyond voting and egging others on, but I have been somewhat burned out by everything going on in my life, including both home and work. While I am hardly in any kind of mental trouble, I've simply not had any tuits of late.

I find it very hard to make public promises which I know I am going to break. Francis suggested that the promise can be broken which while it might not devalue it for him (or you) it does for me. I do however think that public promises are a good thing, especially when they foster useful discussion in the communities I am part of, so from that point of view I very much support Francis in his efforts.

Even given all of the above, I'd like to make a promise statement of my own. I'd like to make it in public and hopefully that'll help me to keep it. I know I could easily fail to live up to this promise, but I'm hoping I'll do well and to some extent I'm relying on all you out there to help me keep it. Given we're almost at the end of the month, I am making the promise now and I want it to take effect starting on the 1st of February 2015.

I hereby promise that I will do better at contributing to all the projects I am nominally part of, making at least one useful material contribution to at least one project every week. I also promise to be more mindful of the choices I make when choosing how to implement solutions to problems, selecting appropriate languages and giving full consideration to how my solution might be attacked if appropriate.

I can't and won't promise not to use C/C++ but if you honestly feel you can make that promise, then I'm certain Francis would love for you to head over to his promise website and pledge. Also regardless of your opinions, please do join in the conversation, particularly regarding being mindful of attack vectors whenever you write something.

New blog engine up and running Daniel Silverstone

Finally I have my blog resurrected and migrated to IkiWiki which pleases me no-end. The theme still isn't quite right, but for the first time since I left Livejournal all those years ago, I have web-based comment support once more.

For those interested, you can see the blog source content and also you can see the setup content. For those interested in Gitano you can also see the post-receive hook which pings the wiki should I push content via Git instead of using the website.

For now, at least, you must log into the blog in order to comment. Most OpenID forms are supported, though the Google one seems a little temperamental and Google will be dropping OpenID support soon anyway. If you don't have an OpenID yet, I recommend using Stack Overflow's service, it's simple and effective.

Sorry for the ugly form the blog is taking, if one of you wants to offer a replacement theme / templates / CSS I'd be quite grateful for suggestions. Currently I'm using a modified bootstrap based theme but I'd entertain anything which overall looks clean and better. Even little tweaks to make the edit/comment links in the inline pages nicer would be good.

Yakking… Daniel Silverstone

Myself, Lars Wirzenius and Richard Maw have been working on an idea for a while now to have a website where we post articles to help new free-software hackers get started.

We called this idea “Yakking” and it has finally arrived.

If you visit http://yakking.branchable.com/ then you can read about the regular contributors, see a short article written by Lars about software freedom and get the RSS/ATOM feed URLs for your feed readers.

If you think you might like to contribute, then there’s an obvious way to contact us (email) and also a suggestions box on the website.

This blog is powered by ikiwiki.