Thursday, October 26, 2017

The Upspin manifesto: On the ownership and sharing of data

Here follows the original "manifesto" from late 2014 proposing the idea for what became Upspin. The text has been lightly edited to remove a couple of references to Google-internal systems, with no loss of meaning.

I'd like to thank Eduardo Pinheiro, Eric Grosse, Dave Presotto and Andrew Gerrand for helping me turn this into a working system, in retrospect remarkably close to the original vision.

Augie image Copyright © 2017 Renee French


Outside our laptops, most of us today have no shared file system at work. (There was a time when we did, but it's long gone.) The world took away our /home folders and moved us to databases, which are not file systems. Thus I can no longer (at least not without clumsy hackery) make my true home directory be where my files are any more. Instead, I am expected to work on some local machine using some web app talking to some database or other external repository to do my actual work. This is mobile phone user interfaces brought to engineering workstations, which has its advantages but also some deep flaws. Most important is the philosophy it represents.

You don't own your data any more. One argument is that companies own it, but from a strict user perspective, your "apps" own it. Each item you use in the modern mobile world is coupled to the program that manipulates it. Your Twitter, Facebook, or Google+ program is the only way to access the corresponding feed. Sometimes the boundary is softened within a company—photos in Google+ are available in other Google products—but that is the exception that proves the rule. You don't control your data, the programs do.

Yet there are many reasons to want to access data from multiple programs. That is, almost by definition, the Unix model. Unix's model is largely irrelevant today, but there are still legitimate ways to think about data that are made much too hard by today's way of working. It's not necessarily impossible to share data among programs (although it's often very difficult), but it's never natural. There are workarounds like plugin architectures and documented workflows but they just demonstrate that the fundamental idea—sharing data among programs—is not the way we work any more.

This is backwards. It's a reversal of the old way we used to work, with a file system that all programs could access equally. The very notion of "download" and "upload" is plain stupid. Data should simply be available to any program with authenticated access rights. And of course for any person with those rights. Sharing between programs and people can be, technologically at least, the same problem, and a solved one.

This document proposes a modern way to achieve the good properties of the old world: consistent access, understandable data flow, and easy sharing without workarounds. To do this, we go back to the old notion of a file system and make it uniform and global. The result should be a data space in which all people can use all their programs, sharing and collaborating at will in a consistent, intuitive way.

Not downloading, uploading, mailing, tarring, gzipping, plugging in and copying around. Just using. Conceptually: If I want to post a picture on Twitter, I just name the file that holds it. If I want to edit a picture on Twitter using Photoshop, I use the File>Open menu in Photoshop and name the file stored on Twitter, which is easy to discover or even know a priori. (There are security and access questions here, and we'll come back to those.)

Working in a file system.

I want my home directory to be where all my data is. Not just my local files, not just my source code, not just my photos, not just my mail. All my data. My "phone" should be able to access the same data as my laptop, which should be able to access the same data as the servers. (Ignore access control for the moment.) $HOME should be my home for everything: work, play, life; toy, phone, work, cluster.

This was how things worked in the old single-machine days but we lost sight of that when networking became universally available. There were network file systems and some research systems used them to provide basically this model, but the arrival of consumer devices, portable computing, and then smartphones eroded the model until every device is its own fiefdom and every program manages its own data through networking. We have a network connecting devices instead of a network composed of devices.

The knowledge of how to achieve the old way still exists, and networks are fast and ubiquitous enough to restore the model. From a human point of view, the data is all we care about: my pictures, my mail, my documents. Put those into a globally addressable file system and I can see all my data with equal facility from any device I control. And then, when I want to share with another person, I can name the file (or files or directory) that holds the information I want to share, grant access, and the other person can access it.

The essence here is that the data (if it's in a single file) has one name that is globally usable to anyone who knows the name and has the permission to evaluate it. Those names might be long and clumsy, but simple name space techniques can make the data work smoothly using local aliasing so that I live in "my" world, you live in your world (also called "my" world from your machines), and the longer, globally unique names only arise when we share, which can be done with a trivial, transparent, easy to use file-system interface.

Note that the goal here is not a new file system to use alongside the existing world. Its purpose is to be the only file system to use. Obviously there will be underlying storage systems, but from the user's point of view all access is through this system. I edit a source file, compile it, find a problem, point a friend at it; she accesses the same file, not a copy, to help me understand it. (If she wants a copy, there's always cp!).

This is not a simple thing to do, but I believe it is possible. Here is how I see it being assembled. This discussion will be idealized and skate over a lot of hard stuff. That's OK; this is a vision, not a design document.

Everyone has a name.

Each person is identified by a name. To make things simple here, let's just use an e-mail address. There may be a better idea, but this is sufficient for discussion. It is not a problem to have multiple names (addresses) in this model, since the sharing and access rights will treat the two names as distinct users with whatever sharing rights they choose to use.

Everyone has stable storage in the network.

Each person needs a way to make data accessible to the network, so the storage must live in the network. The easiest way to think of this is like the old network file systems, with per-user storage in the server farm. At a high level, it doesn't matter what that storage is or how it is structured, as long as it can be used to provide the storage layer for a file-system-like API.

Everyone's storage server has a name identified by the user's name.

The storage in the server farm is identified by the user's name.

Everyone has local storage, but it's just a cache.

It's too expensive to send all file access to the servers, so the local device, whatever it is—laptop, desktop, phone, watch—caches what it needs and goes to the server as needed. Cache protocols are an important part of the implementation; for the point of this discussion, let's just say they can be designed to work well. That is a critical piece and I have ideas, but put that point aside for now.

The server always knows what devices have cached copies of the files on local storage. 

The cache always knows what the associated server is for each directory file in its cache and maintains consistency within reasonable time boundaries.

The cache implements the API of a full file system. The user lives in this file system for all the user's own files. As the user moves between devices, caching protocols keep things working.

Everyone's cache can talk to multiple servers.

A user may have multiple servers, perhaps from multiple providers. The same cache and therefore same file system API refers to them all equivalently. Similarly, if a user accesses a different user's files, the exact same protocol is used, and the result is cached in the same cache the same way. This is federation as architecture.

Every file has a globally unique name.

Every file is named by this triple: (host address, user name, file path). Access rights aside, any user can address any other user's file by evaluating the triple. The real access method will be nicer in practice, of course, but this is the gist.

Every file has a potentially unique ACL.

Although the user interface for access control needs to be very easy, the effect is that each file or directory has an access control list (ACL) that mediates all access to its contents. It will need to be very fine-grained with respect to each of users, files, and rights.

Every user has a local name space.

The cache/file-system layer contains functionality to bind things, typically directories, identified by such triples into locally nicer-to-use names. An obvious way to think about this is like an NFS mount point for /home, where the remote binding attaches to /home/XXX the component or components in the network that the local user wishes to identify by XXX. For example, Joe might establish /home/jane as a place to see all the (accessible to Joe) pieces of Jane's world. But it can be much finer-grained than that, to the level of pulling in a single file.

The NFS analogy only goes so far. First, the binding is a lazily-evaluated, multi-valued recipe, not a Unix-like mount. Also, the binding may itself be programmatic, so that there is an element of auto-discovery. Perhaps most important, one can ask any file in the cached local system what its triple is and get its unique name, so when a user wishes to share an item, the triple can be exposed and the remote user can use her locally-defined recipe to construct the renaming to make the item locally accessible. This is not as mysterious or as confusing in practice as it sounds; Plan 9 pretty much worked like this, although not as dynamically.

Everyone's data becomes addressable.

Twitter gives you (or anyone you permit) access to your Twitter data by implementing the API, just as the regular, more file-like servers do. The same story applies to any entity that has data it wants to make usable. At some scaling point, it becomes wrong not to play.

Everyone's data is secure.

It remains to be figured out how to do that, I admit, but with a simple, coherent data model that should be achievable.

Is this a product?

The protocols and some of the pieces, particularly what runs on local devices, should certainly be open source, as should a reference server implementation. Companies should be free to provide proprietary implementations to access their data, and should also be free to charge for hosting. A cloud provider could charge hosting fees for the service, perhaps with some free or very cheap tier that would satisfy the common user. There's money in this space.

What is this again?

What Google Drive should be. What Dropbox should be. What file systems can be. The way we unify our data access across companies, services, programs, and people. The way I want to live and work.

Never again should someone need to manually copy/upload/download/plugin/workflow/transfer data from one machine to another. 

Thursday, September 21, 2017

Go: Ten years and climbing

Drawing Copyright ©2017 Renee French

This week marks the 10th anniversary of the creation of Go.

The initial discussion was on the afternoon of Thursday, the 20th of September, 2007. That led to an organized meeting between Robert Griesemer, Rob Pike, and Ken Thompson at 2PM the next day in the conference room called Yaounde in Building 43 on Google's Mountain View campus. The name for the language arose on the 25th, several messages into the first mail thread about the design:

Subject: Re: prog lang discussion From: Rob 'Commander' Pike Date: Tue, Sep 25, 2007 at 3:12 PM To: Robert Griesemer, Ken Thompson i had a couple of thoughts on the drive home. 1. name 'go'. you can invent reasons for this name but it has nice properties. it's short, easy to type. tools: goc, gol, goa. if there's an interactive debugger/interpreter it could just be called 'go'. the suffix is .go ...

(It's worth stating that the language is called Go; "golang" comes from the web site address ( was already a Disney web site) but is not the proper name of the language.)

The Go project counts its birthday as November 10, 2009, the day it launched as open source, originally on before migrating to GitHub a few years later. But for now let's date the language from its conception, two years earlier, which allows us to reach further back, take a longer view, and witness some of the earlier events in its history.

The first big surprise in Go's development was the receipt of this mail message:

Subject: A gcc frontend for Go
From: Ian Lance Taylor Date: Sat, Jun 7, 2008 at 7:06 PM To: Robert Griesemer, Rob Pike, Ken Thompson One of my office-mates pointed me at http://.../go_lang.html . It seems like an interesting language, and I threw together a gcc frontend for it. It's missing a lot of features, of course, but it does compile the prime sieve code on the web page.

The shocking yet delightful arrival of an ally (Ian) and a second compiler (gccgo) was not only encouraging, it was enabling. Having a second implementation of the language was vital to the process of locking down the specification and libraries, helping guarantee the high portability that is part of Go's promise.

Even though his office was not far away, none of us had even met Ian before that mail, but he has been a central player in the design and implementation of the language and its tools ever since.

Russ Cox joined the nascent Go team in 2008 as well, bringing his own bag of tricks. Russ discovered—that's the right word—that the generality of Go's methods meant that a function could have methods, leading to the http.HandlerFunc idea, which was an unexpected result for all of us. Russ promoted more general ideas too, like the the io.Reader and io.Writer interfaces, which informed the structure of all the I/O libraries.

Jini Kim, who was our product manager for the launch, recruited the security expert Adam Langley to help us get Go out the door. Adam did a lot of things for us that are not widely known, including creating the original web page and the build dashboard, but of course his biggest contribution was in the cryptographic libraries. At first, they seemed disproportionate in both size and complexity, at least to some of us, but they enabled so much important networking and security software later that they become a crucial part of the Go story. Network infrastructure companies like Cloudflare lean heavily on Adam's work in Go, and the internet is better for it. So is Go, and we thank him.

In fact a number of companies started to play with Go early on, particularly startups. Some of those became powerhouses of cloud computing. One such startup, now called Docker, used Go and catalyzed the container industry for computing, which then led to other efforts such as Kubernetes. Today it's fair to say that Go is the language of containers, another completely unexpected result.

Go's role in cloud computing is even bigger, though. In March of 2014 Donnie Berkholz, writing for RedMonk, claimed that Go was "the emerging language of cloud infrastructure". Around the same time, Derek Collison of Apcera stated that Go was already the language of the cloud. That might not have been quite true then, but as the word "emerging" used by Berkholz implied, it was becoming true.

Today, Go is the language of the cloud, and to think that a language only ten years old has come to dominate such a large and growing industry is the kind of success one can only dream of. And if you think "dominate" is too strong a word, take a look at the internet inside China. For a while, the huge usage of Go in China signaled to us by the Google trends graph seemed some sort of mistake, but as anyone who has been to the Go conferences in China can attest, the measurements are real. Go is huge in China.

In short, ten years of travel with the language have brought us past many milestones. The most astonishing is at our current position: a conservative estimate suggests there are at least half a million Go programmers. When the mail message naming Go was sent, the idea of there being half a million gophers would have sounded preposterous. Yet here we are, and the number continues to grow.

Speaking of gophers, it's been fun to watch how Renee French's idea for a mascot, the Go gopher, became not only a much loved creation but also a symbol for Go programmers everywhere. Many of the biggest Go conferences are called GopherCons as they gather together gophers from all over the world.

Gopher conferences are taking off. The first one was only three years ago, yet today there are many, all around the world, plus countless smaller local "meetups". On any given day, there is more likely than not a group of gophers meeting somewhere in the world to share ideas.

Looking back over ten years of Go design and development, it is astounding to reflect on the growth of the Go community. The number of conferences and meetups, the long and ever-increasing list of contributors to the Go project, the profusion of open source repositories hosting Go code, the number of companies using Go, some exclusively: these are all astonishing to contemplate.

For the three of us, Robert, Rob, and Ken, who just wanted  to make our programming lives easier, it's incredibly gratifying to witness what our work has started.

What will the next ten years bring?

- Rob Pike, with Robert Griesemer and Ken Thompson

Saturday, February 25, 2017

The power of role models

I spent a few days a while back in a board meeting for a national astronomy organization and noticed a property of the population in that room: Out of about 40 people, about a third were women. And these were powerful women, too: professors, observatory directors and the like. Nor were they wallflowers. Their contributions to the meeting exceeded their proportion.

In my long career, I had never before been in a room like that, and the difference in tone, conversation, respect, and professionalism was unlike any I have experienced. I can't prove it was the presence of women that made the difference - it could just be that astronomers are better people all around, a possibility I cannot really refute - but it seemed to me that the difference stemmed from the demographics.

The meeting was one-third women, but of course in private conversation, when pressed, the women I spoke to complained that things weren't equal yet. We all have our reference points.

But let's back up for a moment and think about the main point: In a room responsible for overseeing the budget and operation of major astronomical observatories, including things like the Hubble telescope, women played a major role. The contrast with computing is stark.

It really got me thinking. At dinner I asked some of the women to speak to me about this, how astronomy became so (relatively) egalitarian. And one topic became clear: role models. Astronomy has a long history of women active in the field, going all the way back to Caroline Herschel in the early 19th century. Women have made huge contributions to the field. Dava Sobel just wrote a book about the women who laid the foundations for the discovery of the expansion of the universe. Just a couple of weeks ago, papers ran obituaries of Vera Rubin, the remarkable observational astronomer who discovered the evidence for dark matter. I could mention Jocelyn Bell, whose discovery of pulsars got her advisor a Nobel (sic).

The most famous astronomer I met growing up was Helen Hogg, the (adopted) Canadian astronomer at David Dunlap Observatory outside Toronto, who also did a fair bit of what we now call outreach.

The women at the meeting spoke of this, a history of women contributing, of role models to look up to, of proof that women can make major contributions to the field.

What can computing learn from this?

It seems we're doing it wrong. The best way to improve the representation of women in the field is not to recruit them, important though that is, but to promote them. To create role models. To push them into positions of influence. Women leave computing in large numbers because they don't see a path up, or because the culture makes them unwelcome. More women excelling in the field, famous women, brilliant women, would be inspiring.

Men have the power to help fix those things, but they also should have the courage to cede the stage to women more often, to fight the stupid bias that keeps women from excelling in the field. It may take proactive behavior, like choosing a women over a man when growing your team, just because, or promoting women more freely.

But as I see it, without something being done to promote female role models, the way things are going computing will still be backwards a hundred years from now.

Sunday, August 03, 2014


Two long-buried caches of photographs came to light last year. One was a stack of cellulose nitrate negatives made on the Scott Antarctic expedition almost a hundred years ago. Over time, they became stuck together into a moldy brick, but it was possible to tease the negatives apart and see what they revealed. You can view the images at the web site of the New Zealand Antarctic Heritage Trust. The results show ragged edges and mold spots but, even beyond their historical importance, the photographs are evocative and in some cases very beautiful.

The other cache contained images not quite so old and of less general interest but of personal importance. My mother moved from the house she had occupied for decades into a smaller apartment and while preparing to move she found the proverbial shoe box of old pictures in a closet. Some of the images are from my youth, some from hers, and some even from her parents'. One of the photographs, from 1931, shows my paternal great-grandparents. I never met my paternal grandparents, let alone great-grandparents, so this photograph touches something almost primordial for me. And some of the photographs in the box were even older.

Due to the miracle of photography, we are able to see over a hundred years into the past. Of course this is not news; all of us have seen 19th century photographs by the pioneers of the medium. By the turn of the 20th century photography was so common that huge numbers of images, from the historical to the mundane, had been created. And sometimes we are lucky enough to chance upon forgotten images that open a window into a past that would otherwise fade from view.

But such windows are becoming rare. A hundred years from now, there will be far fewer photo caches to find. Although the transition to digital photography has made photos almost unimaginably commonplace—one estimate puts the number of shutter activations at a trillion images worldwide per year—very few of those images become artifacts that can be left in a shoe box.

We live in what has been named a Digital Dark Age. Because digital technology evolves so fast, we are rapidly losing the ability to understand yesterday's media. As file formats change, software becomes obsolete, and hardware becomes outmoded, old digital files become unreadable and unrecoverable.

There are many examples of lost information, but here is an illustrative story of disaster narrowly averted. Early development of the Unix operating system, which became the software foundation for the Internet, was done in the late 1960s and early 1970s on Digital Equipment Corporation computers. Backups were made on a magnetic medium called a DECtape. By the mid 1970s, DECtape was obsolete and by the 1980s there were no remaining DECtape drives that could read the old backups. The scientists in the original Unix lab had kept a box of old backups under the raised floor of the computer room, but the tapes had spontaneously become unreadable because the device to read them no longer existed in the lab or anywhere else as far as anyone knew. And even if it did, no computer that could run the device was still powered on. Fortunately, around 1990 Paul Vixie and Keith Bostic, working for a different company, stumbled across an old junked DECtape drive and managed to get it up and running again by resurrecting an old computer to connect it to. They contacted the Unix research group and offered one last chance to recover the data on the backup tapes before the computer and DECtape drive were finally decommissioned. Time and resources were limited, but some of the key archival pieces of early Unix development were recovered through this combination of charity and a great deal of luck. This story has a happy ending, but not all digital archives survive. Far from it.

The problem is that as technology advances, data needs to be curated. Files need to have their formats converted, and then transferred to new media. A backup disk in a box somewhere might be unreadable a few years from now. Its format may be obsolete, the software to read it might not run on current hardware, or the media might have physically decayed. NASA lost a large part of the data collected by the Viking Mars missions because the iron oxide fell off the tapes storing the data.

Backups are important but they too are temporary, subject to the same problems as the data they attempt to protect. Backup software can become obsolete and media can fail. The same affliction that damaged the Viking tapes also wiped out my personal backup archive; I lost the only copy of my computer work from the 1970s. (It's worth noting my negatives and prints from the period survived.)

It's not just tapes that go bad. Consider CDs and DVDs, media often used for backup. The disks, especially the writable kind use for backups, are very fragile, much more so than the mass-produced read-only kind used to store music and movies. Within a few years, especially in humid environments, the metal film can separate from the backing medium. Even if the backup medium survives, the formats used to store the backups might become obsolete. The software that reads the backups might not run on the next computer one buys. Today, CDs are already becoming relics; many computers today do not even come with a CD or DVD drive. What were once the gold standard for backup are already looking old-fashioned just a few years on. They will be antiquated and obscure a century from now.

To summarize, digital information requires maintenance. It's not sufficient to make backups; the backups also need to be maintained, upgraded, transferred, and curated. Without conscientious care, the data of today will be lost forever in a few years. Even with care, it's possible through software or hardware changes to lose access forever. That shoebox of old backup CDs will be unreadable soon.

Which brings us back to those old photo caches. They held negatives and prints, physical objects that stored images. They needed no attention, no curating, no updating. They sat untended and forgotten for decades, but through all that time faithfully held their information, waiting for a future discoverer. As a result, we can all see what the Scott Antarctic expedition saw, and I can see what my great-grandparents looked like.

It is a sad irony that modern technology makes it unlikely that future generations will see the images made today.

Ask yourself whether your great-grandchildren will be able to see your photographs. If the images exist only as a digital image file, the answer is almost certainly, "No". If, however, there are physical prints, the odds improve. Those digital images need to be made real to endure. Without a print, a digital photograph has no future.

We live in a Digital Dark Age, but as individuals we can shine a little light. If you are one of the uncounted photographers who enjoy digital photography, keep in mind the fragility of data. When you have a digital image you care about, for whatever reason, artistic or sentimental, please make a print and put that print away. It will sit quietly in the dark, holding fast, never forgetting, ready to reveal itself to a grateful future generation.

Friday, January 24, 2014

Self-referential functions and the design of options

I've been trying on and off to find a nice way to deal with setting options in a Go package I am writing. Options on a type, that is. The package is intricate and there will probably end up being dozens of options. There are many ways to do this kind of thing, but I wanted one that felt nice to use, didn't require too much API (or at least not too much for the user to absorb), and could grow as needed without bloat.

I've tried most of the obvious ways: option structs, lots of methods, variant constructors, and more, and found them all unsatisfactory. After a bunch of trial versions over the past year or so, and a lot of conversations with other Gophers making suggestions, I've finally found one I like. You might like it too. Or you might not, but either way it does show an interesting use of self-referential functions.

I hope I have your attention now.

Let's start with a simple version. We'll refine it to get to the final version.

First, we define an option type. It is a function that takes one argument, the Foo we are operating on.

type option func(*Foo)

The idea is that an option is implemented as a function we call to set the state of that option. That may seem odd, but there's a method in the madness.

Given the option type, we next define an Option method on *Foo that applies the options it's passed by calling them as functions. That method is defined in the same package, say pkg, in which Foo is defined.

This is Go, so we can make the method variadic and set lots of options in a given call:

// Option sets the options specified.
func (f *Foo) Option(opts ...option) {
  for _, opt := range opts {

Now to provide an option, we define in pkg a function with the appropriate name and signature. Let's say we want to control verbosity by setting an integer value stored in a field of a Foo. We provide the verbosity option by writing a function with the obvious name and have it return an option, which means a closure; inside that closure we set the field:

// Verbosity sets Foo's verbosity level to v.
func Verbosity(v int) option {
  return func(f *Foo) {
     f.verbosity = v

Why return a closure instead of just doing the setting? Because we don't want the user to have to write the closure and we want the Option method to be nice to use. (Plus there's more to come....)

In the client of the package, we can set this option on a Foo object by writing:


That's easy and probably good enough for most purposes, but for the package I'm writing, I want to be able to use the option mechanism to set temporary values, which means it would be nice if the Option method could return the previous state. That's easy: just save it in an empty interface value that is returned by the Option method and the underlying function type. That value flows through the code:

type option func(*Foo) interface{}

// Verbosity sets Foo's verbosity level to v.
func Verbosity(v int) option {
  return func(f *Foo) interface{} {
     previous := f.verbosity
      f.verbosity = v
     return previous

// Option sets the options specified.
// It returns the previous value of the last argument.
func (f *Foo) Option(opts ...option) (previous interface{}) {
  for _, opt := range opts {
     previous = opt(f)
  return previous

The client can use this the same as before, but if the client also wants to restore a previous value, all that's needed is to save the return value from the first call, and then restore it.

prevVerbosity := foo.Option(pkg.Verbosity(3))

The type assertion in the restoring call to Option is clumsy. We can do better if we push a little harder on our design.

First, redefine an option to be a function that sets a value and returns another option to restore the previous value.

type option func(f *Foo) option

This self-referential function definition is reminiscent of a state machine. Here we're using it a little differently: it's a function that returns its inverse.

Then change the return type (and meaning) of the Option method of *Foo to option from interface{}:

// Option sets the options specified.
// It returns an option to restore the last arg's previous value.
func (f *Foo) Option(opts ...option) (previous option) {
  for _, opt := range opts {
      previous = opt(f)
  return previous

The final piece is the implementation of the actual option functions. Their inner closure must now return an option, not an interface value, and that means it must return a closure to undo itself. But that's easy: it can just recur to prepare the closure to undo the original! It looks like this:

// Verbosity sets Foo's verbosity level to v.
func Verbosity(v int) option {
  return func(f *Foo) option {
     previous := f.verbosity
     f.verbosity = v
     return Verbosity(previous)

Note the last line of the inner closure changed from
     return previous
     return Verbosity(previous)
Instead of just returning the old value, it now calls the surrounding function (Verbosity) to create the undo closure, and returns that closure.

Now from the client's view this is all very nice:

prevVerbosity := foo.Option(pkg.Verbosity(3))

And finally we take it up one more level, using Go's defer mechanism to tidy it all up in the client:

func DoSomethingVerbosely(foo *Foo, verbosity int) {
  // Could combine the next two lines,
  // with some loss of readability.
  prev := foo.Option(pkg.Verbosity(verbosity))
  defer foo.Option(prev)
  // ... do some stuff with foo under high verbosity.

It's worth noting that since the "verbosity" returned is now a closure, not a verbosity value, the actual previous value is hidden. If you want that value you need a little more magic, but there's enough magic for now.

The implementation of all this may seem like overkill but it's actually just a few lines for each option, and has great generality. Most important, it's really nice to use from the point of view of the package's client. I'm finally happy with the design. I'm also happy at the way this uses Go's closures to achieve its goals with grace.

Wednesday, May 01, 2013

Eisenbahnnet: Bohr's trip about spin

The other day I was talking with a friend (yes, I have friends) about the way communication of ideas has changed. The Internet is the obvious advance, but what used to happen when an important new idea needed to be disseminated? As an example of how things used to be, I told him the story of Bohr's famous train trip.

In 1925, two students at the University of Leiden, Sem Goudsmit and George Uhlenbeck, realized that the fourth electron quantum number could be explained if the electron had spin. This was a radical idea (a point particle spinning?), and coming from students was doubly suspect. Physicists throughout Europe were excited yet skeptical. When Bohr was planning a trip from Cophenhagen to Leiden for a conference, it seemed an excellent opportunity to talk to the students to help understand if they were right.

In his book, Inward Bound, Abraham Païs narrates the story as told to him by Bohr twenty years later:

Bohr's train to Leiden made a stop in Hamburg, where he was met by Pauli and Stern who had come to the station to ask him what he thought about spin. Bohr must have said that it was very very interesting (his favorite way of expressing that something was wrong), but he could not see how an electron moving in the electric field of the nucleus could experience the magnetic field necessary for producing fine structure. (As Uhlenbeck said later: 'I must say in retrospect that Sem and I in our euphoria had not really appreciated [this] basic difficulty.') On his arrival in Leiden, Bohr was met at the train by Ehrenfest and Einstein who asked him what he thought about spin. Bohr must have said that it was very very interesting but what about the magnetic field? Ehrenfest replied that Einstein had resolved that. The electron in its rest frame sees a rotating electric field; hence by elementary relativity it also sees a magnetic field. The net result is an effective spin-orbit coupling. Bohr was at once convinced. When told of the factor of two he expressed confidence that this problem would find a natural resolution. He urged Sem and George to write a more detailed note on their work. They did; Bohr added an approving comment.
After Leiden Bohr traveled to Goettingen. There he was met at the station by Heisenberg and Jordan who asked what he thought about spin. Bohr replied that it was a great advance and explained about the spin-orbit coupling. Heisenberg remarked that he had heard this remark before but that he could not remember who made it and when. ... On his way home the train stopped at Berlin where Bohr was met at the station by Pauli, who had made the trip from Hamburg for the sole purpose of asking Bohr what he now thought about spin. Bohr said it was a great advance, to which Pauli replied: 'eine Neue Kopenhagener Irrlehre' (a new Copenhagen heresy). After his return home Bohr wrote to Ehrenfest that he had become 'a prophet of the electron magnet gospel.'

Sneakernet indeed, or perhaps Eisenbahnnet. The idea of the great physicist carrying precious nuggets of wisdom across Europe is romantic and poignant. It also shows how Bohr's insight, and the insight of his brilliant colleagues, did the peer review in real time in two train trips. Bohr, Pauli, Stern, Ehrenfest, Einstein, Heisenberg, Jordan, Pauli: What a peer review it was!

It is one of the greatest oversights of the Nobel committee that Goudsmit and Uhlenbeck were never rewarded. Their colleagues certainly understood the earth-shaking merit of their insight.

Saturday, September 22, 2012

Thank you Apple

Some days, things just don't work out. Or don't work.


I wanted to upgrade (their term, not mine) my iMac from Snow Leopard (10.6) to Lion (10.7). I even had the little USB stick version of the installer, to make it easy. But after spending some time attempting the installation, the Lion installer "app" failed, complaining about SMART errors on the disk.

Disk Utility indeed reported there were SMART errors, and that the disk hardware needed to be replaced. An ugly start.

The good news is that in some places, including where I live, Apple will do a house call for service, so I didn't have to haul the computer to an Apple store on public transit.

Thank you Apple.

I called them, scheduled the service for a few days later, and as instructed by Apple (I hardly needed prompting) prepped a backup using Time Machine.

The day before the repairman was to come to give me a new disk, I made sure the system was fully backed up, for security reasons started a complete erasure of the bad disk (using Disk Utility in target mode from another machine, about which more later), and went to bed.

The day

When I got up, I checked that the disk had been erased and headed off to work. As I left the apartment, the ceiling lights in the entryway flickered and then went out: a new bulb was needed. On the way out of the building, I asked the doorman for a replacement bulb. He offered just to replace it for us. We have a good doorman.

Once at work, things were normal until my cell phone rang about 2pm. It was the Apple repairman, Twinkletoes (some names and details have been changed), calling to tell me he'd be at my place within the hour. Actually, he wasn't an Apple employee, but a contractor working for Unisys, a name I hadn't heard in a long time. (Twinkletoes was a name I hadn't heard for a while either, but that's another story.) At least here, Apple uses Unisys contractors to do their house calls.

So I headed home, arriving before Twinkletoes. At the front door, the doorman stopped me. He reported that the problem with the lights was not the bulb, but the wiring. He'd called in an electrician, who had found a problem in the breaker box and fixed it. Everything was good now.

When I got up to the apartment, I found chaos: the cleaners were mid-job, with carpets rolled up, vacuum cleaners running, and general craziness. Not conducive to work. So I went back down to the lobby with my laptop and sat on the couch, surfing on the free WiFi from the café next door, and waited for Twinkletoes.

Half an hour later, he arrived and we returned to the apartment. The cleaners were still there but the chaos level had dropped and it wasn't too hard to work around them. I saw what the inside of an iMac looks like as Twinkletoes swapped out the drive. By the time he was done, the cleaners had left and things had settled down.

The innards of my 27" iMac

I had assumed that the replacement drive would come with an installed operating system, but I assumed wrong. (When you assume, you put plum paste on your ass.) I had a Snow Leopard installation DVD, but I was worried: it had failed to work for me a few days earlier when I wanted to boot from it to run fsck on the broken drive. Twinkletoes noticed it had a scratch. I needed another way to boot the machine.

It had surprised me when Lion came out that the installation was done by an "app", not as a bootable image. This is an unnecessary complication for those of us that need to maintain machines. Earlier, when updating a different machine, I had learned how painful this could be when the installation app destroyed the boot sector and I needed to reinstall Snow Leopard from DVD, and then upgrade that to a version of the system recent enough to run the Lion installer app. As will become apparent, had Lion come as a bootable image things might have gone more smoothly.

Thank you Apple.

[Note added in post: Several people have told me there's a bootable image inside the installer. I forgot to mention that I knew that, and there wasn't. For some reason, the version on the USB stick I have looks different from the downloaded one I checked out a day or two later, and even Twinkletoes couldn't figure out how to unpack it. Weird.]

Twinkletoes had an OS image he was willing to let me copy, but I needed to make a bootable drive from it. I had no sufficiently large USB stick—you need a 4GB one you can wipe. However I did have a free, big enough CompactFlash card and a USB reader, so that should do, right? Twinkletoes was unsure but believed it would.

Using my laptop, I used Disk Utility to create a bootable image on the CF card from Twinkletoes's disk image. We were ready.

Plug in the machine, push down the Option key, power on.


Turn on the light.


No power.

The cleaners must have tripped a breaker.

I went to the breaker box and found that all the breakers looked OK. We now had a mystery, because the cleaners had had lights on and were using electric appliances—I saw a vacuum cleaner running—but now there was no power. Was the power off to the building? No: the lights still worked in the kitchen and the oven clock was lit. I called the doorman and asked him to get the electrician back as soon as possible and then, with a little portable lamp, went looking around the apartment for a working socket. I found one, again in the kitchen. The iMac was going to travel after all, if not as far as downtown.

The machine was moved, plugged in, option-key-downed, and powered on. I selected the CF card to boot from, waited 15 minutes for the installation to come up, only to have the boot fail. CF cards don't work after all, although the diagnosis of failure is a bit tardy and uninformative.

Thank you Apple.

Next idea. My old laptop has FireWire so we could bring the disk up using target mode and then run the installer on the laptop to install Lion on the iMac.

We did the target mode dance and connected to the newly installed drive, then ran Disk Utility on the laptop to format the drive. Things were starting to look better.

Next, we put the Lion installer stick into the laptop, which was running a recent version of Snow Leopard.

Failure again. This time the problem is that the laptop, all of about four years old, is too old to run Lion. It's got a Core Duo, not a Core 2 Duo, and Lion won't run on that hardware. Even though Lion doesn't need to run, only the Lion installer needs to run, the system refuses to help. My other laptop is new enough to run the installer, but it doesn't have FireWire so it can't do target mode.

Thank you Apple. Your aggressive push to retire old technology hurts sometimes, you know? Actually, more than sometimes, but let's stay on topic.

Twinkletoes has to leave—he's been on the job for several hours now—but graciously lends me a USB boot drive he has, asking me to return it by post when I'm done. I thank him profusely and send him away before he is drawn in any deeper.

Using his boot drive, I was able to bring up the iMac and use the Lion installer stick to get the system to a clean install state. Finally, a computer, although of course all my personal data is over on the backup.

When a new OS X installation comes up, it presents the option of "migrating" data from an existing system, including from a Time Machine backup. So I went for that option and connected the external drive with the Time Machine backup on it.

The Migration Assistant presented a list of disks to migrate from. A list of one: the main drive in the machine. It didn't give me the option of using the Time Machine backup.

Thank you Apple. You told me to save my machine this way but then I can't use this backup to recover.

I called Apple on my cell phone (there's still no power in the room with the land line's wireless base station) and explained the situation. The sympathetic but ultimately unhelpful person on the phone said it should work (of course!) and that I should run Software Update and get everything up to the latest version. He reported that there were problems with the Migration Assistant in early versions of the Lion OS, and my copy of the installer was pretty early.

I started the upgrade process, which would take a couple of hours, and took my laptop back down to the lobby for some free WiFi to kill time. But it's now evening, the café is closed, and there is no WiFi. Naturally.

Back to the apartment, grab a book, return to the lobby to wait for the electrician.

An hour or so later, the electrician arrived and we returned to the apartment to see what was wrong. It was easy to diagnose. He had made a mistake in the fix, in fact a mistake related to what was causing the original problem. The breaker box has a silly design that makes it too easy to break a connection when working in the box, and that's what had happened. So it was easy to fix and easy to verify that it was fixed, but also easy to understand why it had happened. No excuses, but problem solved and power was now restored.

The computer was still upgrading but nearly done, so a few minutes later I got to try migrating again. Same result, naturally, and another call to Apple and this time little more than an apology. The unsatisfactory solution: do a clean installation and manually restore what's important from the Time Machine backup.

Thank you Apple.

It was fairly straightforward, if slow, to restore my personal files from the home directory on the backup, but the situation for installed software was dire. Restoring an installed program, either using the ludicrous Time Machine UI or copying the files by hand, is insufficient in most cases to bring back the program because you also need manifests and keys and receipts and whatnot. As a result, things such as iWork (Keynote etc.) and Aperture wouldn't run. I could copy every piece of data I could find but the apps refused to let me run them. Despite many attempts digging far too deep into the system, I could not get the right pieces back from the Time Machine backup. Worse, the failure modes were appalling: crashes, strange display states, inexplicable non-workiness. A frustating mess, but structured perfectly to belong on this day.

For peculiar reasons I didn't have the installation disks for everything handy, so these (expensive!) programs were just gone, even though I had backed up everything as instructed.

Thank you Apple.

I did have some installation disks, so for instance I was able to restore Lightroom and Photoshop, but then of course I needed to wait for huge updates to download even though the data needed was already sitting on the backup drive.

Back on the phone for the other stuff. Because I could prove that I had paid for the software, Apple agreed to send me fresh installation disks for everything of theirs but Aperture, but that would take time. In fact, it took almost a month for the iWork DVD to arrive, which is unacceptably long. I even needed to call twice to remind them before the disks were shipped.

The Aperture story was more complicated. After a marathon debugging session I managed to get it to start but then it needed the install key to let me do anything. I didn't have the disk, so I didn't know the key. Now, Aperture is from part of the company called Pro Tools or something like that, and they have a different way of working. I needed to contact them separately to get Aperture back. It's important to understand I hadn't lost my digital images. They were backed up multiple times, including in the network, on the Time Machine backup, and also on an external drive using the separate "vault" mechanism that is one of the best features of Aperture.

I reached the Aperture people on the phone and after a condensed version of the story convinced them I needed an install key (serial number) to run the version of Aperture I'd copied from the Time Machine backup. I was berated by the person on the phone: Time Machine is not suitable for backing up Aperture databases. (What? Your own company's backup solution doesn't know how to back up? Thank you Apple.) After a couple more rounds of abuse, I convinced the person on the phone that a) I was backing up my database as I should, using an Aperture vault and b) it wasn't the database that was the problem, but the program. I was again told that wasn't a suitable way to back up (again, What?), at which point I surrendered and just begged for an installation key, which was provided, and I could again run Aperture. This was the only time in the story where the people I was interacting with were not at least sympathetic to my situation. I guess Pro is a synonym for unfriendly.

Thank you Apple.

There's much more to the story. It took weeks to get everything working again properly. The complete failure of Time Machine to back up my computer's state properly was shocking to me. After this fiasco, I learned about the Lion Recovery App, which everyone who uses Macs should know about, but was not introduced until well after Lion rolled out with its preposterous not-bootable installation setup. The amount of data I already had on my backup disk but that needed to be copied from the net again was laughable. And there were total mysteries, like GMail hanging forever for the first day or so, a problem that may be unrelated or may just be the way life was this day.

But, well after midnight, worn out, beat up, tired, but with electricity restored and a machine that had a little life in it again, I powered down, took the machine back to my office and started to get ready for bed. Rest was needed and I had had enough of technology for one day.

One more thing

Oh yes, one more thing. There's always one more thing in our technological world.

I walked into the bathroom for my evening ablutions only to have the toilet seat come off completely in my hand.

Just because you started it all, even for this,

Thank you Apple.