An Update…

Leave a reply

Weird work is hard.

Internally, SiftFu has been undergoing several rounds of failed builds. Each time that I believed I had the technical issues pretty much under control, something would just not work.

My intention here is merely to clarify a situation that would otherwise be open to a whole host of misinterpretations. SiftFu was never an abandoned piece of software – the long period of radio silence was a reflection of total uncertainty as to where or when it would be acceptable to claim release quality, and a desire to avoid the mistakes of Fall 2012, during which several unacceptably bad (minimum way-below-viable not-really-product) releases served purely to damage morale. This is not a case of seeking perfection either – the app has legitimately failed so far to reach basic, privacy-safe usability.

I have learned a great deal from this experience, and it’s not over. I have roughly until summer to produce a working app. At that point the next plan kicks in. I am returning to the study of physics. I plan to document more thoroughly the myriad lessons I’ve learned from New York to Edinburgh to Santiago and back. That said, this post deserves at least a short treatment of a few of the things that I’ve done and seen, and why they lead me where I claim that I’m going.

File:Power Mac G3 B&W.jpg

My latest piece of Apple hardware. It will run SimAnt and some versions of Microsoft Word that came out in the 90s.

I too often start from the abstract, so here I will start with examples. I don’t own any device that runs iOS, or any Macintosh hardware save the old G3 I’m finally about to throw out. I have to push myself to use Twitter – my Twitter feed is honestly pretty useless to me, as it’s full of exactly the sort of vaguely technical easy reading that I’m not supped to let distract me from work and find mostly uninspiring anyway. I love Facebook, but I think it reached my own personal usability peak around 2009 – when they turned all the interests and likes and stuff into brand logos, I felt they had just added their first major antifeature, which might have been justified if it gave them enough revenue. I still update my LiveJournal. I like the long text format and friends-lock capability, which encourage deep, original, personal writing.

File:Angry Birds Land Särkänniemi 11.jpg

Remember when SimCity 2000 came out and was all 3D and modern-looking? These days I have no idea.

I would have trouble starting anything like Twitter, or Tweetdeck, or Angry Birds, Groupon, etc. The ideas would never occur to me, because I don’t use software that way, or for those things. I don’t know if I will ever use an iPhone. The aesthetic is not mine. I cannot deny, though, that this is where the tech industry is headed.

In other words, I am a weirdo. My ideas are weird. SiftFu is weird. This can be a good thing – SiftFu’s video “launch” received some amazingly positive feedback, and I believe that it tapped into a sentiment that is just below the surface of common understanding. It can be a bad thing – some people never understand what I’m working on, because it only makes sense if you can relate to this quirky notion of “relevance” that I am slowly trying to invent. Weird work is hard. Sometimes I wonder if I should have built something more typical. The only problem is that being weird, I don’t find that stuff necessarily any easier. The hard things are easier for me, and the easy things too often too hard.

File:Simple pendulum height.svg

I prefer the quantum version of this thing. Not that the quantum Hamiltonian is necessarily any easier to solve, but I find it more emotionally satisfying. Note: all images so far are public domain.

It’s not a loss, though, as long as I learn to use this as a comparative advantage. I saw the same pattern in physics. Classical mechanics in freshman year was the most difficult transition. About halfway through electrodynamics, things were starting to make sense. Quantum was not only extraordinarily exciting, it felt good. Quantum is weird. That’s why I love it. Then I skipped some prerequisites and jumped straight into field theory and lattice quantum chromodynamics. I want to one day lead weird projects. Quantum computing. Biomaterials. 4th and 5th phases of matter. Kinks in space and time.

Anything worth leading should be worth following. I will try again with SiftFu this spring, despite the unromantic and banal nature of the next tasks. I don’t have proof that it will work in the market, but I have signs. That’s enough for the level of risk I’m about to accept. After that, the industry might not be headed where I want to be. That’s okay. I will do great work in academia, and as I become ready, I will bring the awesomeness of weird work to the rest of the world.

File:Swarthmore Amphitheatre Spring2.jpg

Swarthmore Amphitheatre. From http://en.wikipedia.org/wiki/File:Swarthmore_Amphitheatre_Spring2.jpg. Released under CC-BY-SA by Runneryoshi105. This is a great place to do weird work.

If you’ve read closely, you’ve probably noticed a loose end. Maybe more than one. I have an RSS feed somewhere on this blog if you’re interested in the next chapters.

SiftFu in Preparation for Launch

Leave a reply

SiftFu, the content awareness engine for the real-time web, is nearly ready for launch.  Sign up for beta access if you so please, and watch our intro video!

This is the site I motivated, described and previewed in past blog posts.  I’d describe it here, but I think you’ll do better to watch the video and see the site:

How

Leave a reply

Update: By the way, SiftFu has launched!

 

This post is about the way in which SiftFu accomplishes its goals as outlined in the manifesto. The challenge is certainly difficult at face value: it is a combination of search (finding the right content), comprehension (knowing which content is truly relevant), organization (displaying that content in a meaningful way), and connotation (making it comprehensible). This is not a technical post – it is intended to explain features that everday users will begin to see and understand as the product comes further along.

Buried within these challenges are the inherent limitations of starting a new company with a small team. There is little opportunity to throw a sheer volume of computing power into the mix, as server costs are still significant. Developing new algorithms or techniques that could take years is also out of the question. SiftFu, rather than relying on any sort of new technological power to overcome competitors, is built to be fundamentally different from what’s out there.

Impermanence

This is the 1st big divergence between a sift and a search. A search is a 1-time event – it takes a few terms and gives back matches right on the spot, assuming you’ll be done with it before anything new comes crashing onto the web.

SiftFu makes the opposite assumption. Terms, times, locations, and whatever else you use to look for subjects of interests all stick. They create a perennially present feed of new content that updates whenever things appear. Rather than a momentary curiosity, SiftFu is optimized for a long-held interest that remains relevant even as the available information changes.

Volume and Specificity

What do the following have in common?

  • the sun
  • a strawberry
  • scala (programming language)
  • Barack Obama
  • The Yeah Yeah Yeahs

For me (results may vary due to personalized search), all of these include Wikipedia in Google’s 1st page of results. For what Google is, this is the most sensible option – Wikipedia is highly regarded, has been around for a relatively long time, and appeals to the widest possible user based as it contains content intended for those who have never before heard of the subject at hand. It fits the use case of search well – a user encounters a topic they’ve never seen before and desires to satisfy their curiosity in a single moment.

It doesn’t make sense, however, for more expert users. I know what most of these things are and probably would know to check Wikipedia on my own if I didn’t. Furthermore, once I’ve read a source, its recurring value falls substantially. Showing me the most popular, lowest-common-denominator pages for something like Scala (the programming language I use for SiftFu) or The Yeah Yeah Yeahs (a band I kind of follow) is useless to me, kind of like teaching 1st grade arithmetic to a class of physics PhDs. The content I want in this case is new and specialized – it is not widely applicable to the entire human population, but instead focused on Scala programmers and Yeah Yeah Yeahs fans. It is also most valuable if I haven’t seen it before, which is especially true of new content.

This is the 2nd way in which SiftFu diverges from typical search – rather than favoring the oldest and strongest, SiftFu looks for new content that is the most targeted. In other words, we do not seek the loudest voice in the room, but rather the whisper directed to the listener.

Disambiguation (note: this feature isn’t finished yet, which is why you might not see it in the 1st release)

What is a dog?

Sometimes a dog is an animal. Other times it is someone we don’t like. Then again, it could be an action that makes quite a pest out of the one who does it.

Now what do a dog, a canine, a mongrel, a German shepard, and a mutt all have in common? All of these words relate to a single concept, one that should be well-defined and present a clear mental image to any with good knowledge of the English language. There is no longer any ambiguity when all these words are present – they relate to a single thing.

Search engines run on words. SiftFu runs on concepts. It takes multiple words and from them generates a single idea, which in turn applies to many more words. In this way, it avoids getting caught up on the particulars of a term and instead can focus on those all important nuances of meaning.

Levels of Configuration

Convenience, privacy, control. Pick 1. That’s how it generally works – you can give up more information about yourself in order to gain convenience (letting the system guess better what you want), but you sacrifice privacy and can’t control what it does. Alternatively, you can take the controls and enter all the information yourself – but you’ve just lost any semblance of convenience and are still revealing quite a bit of personal stuff.

SiftFu allows you to specify how much the system guesses, knows and learns. It is possible to start an account with no more than a throwaway username and password, and immediately start a sift over publicly available sources for a topic that is nearly impossible to trace back to its owner. At the other end of the spectrum is the browser extension – this will track and crawl the sites you regularly visit for content that is likely to update regularly and ensure that anything which appears in a feed (RSS, Atom, Facebook, Twitter, etc.) also appears in you SiftFu results, weighted by how much time you spend there.

The SiftFu Manifesto

2 Replies

Note: SiftFu is going to be taking beta testers soon! Please go and sign up!

 

Why

The thing that people seem to miss about not just Google, but also our competitors, Yahoo, eBay and so forth, is that there’s an awful lot of communities that have never been served by traditional media.

 

-Eric Schmidt (Read more at BrainyQuote)

Relevance is good. It’s not always measurably good to the average user, but we know it when we see it. The shift from web directories to search engines didn’t happen by accident. Neither did the jump from walled garden services (AOL, Compuserve, etc.) to the open Internet. Both of these happened for a reason, and that reason is relevance. In one case the filter was better, and in the other the content available was more broad. In both cases, this allowed us to find content that was more relevant, more tailored to our interests. It wasn’t necessarily objectively better content – had this been the problem, the original megacorporations could simply have offered bigger payouts for better providers. It was because the freedom of an open, searched web meant that our browsing adjusted to us, rather than the other way around.

We don’t “browse” the way we used to, however. Instead, more and more of our online experience comes in the form of updates to and from the sites, stories, friends, and followers we have come to check several times each day. Some of us just leave our browsers open, waiting and watching for some interesting tidbit to appear.

On this new Internet, however, we seem to have left the old lessons behind. We’re back to a world of walled gardens, content providers building up “feeds” and “streams” based on what they guess we will want to consume, sometimes based on information we didn’t even realize they were using. We don’t have the kind of control over the process that we were used to in search. We also left behind the open access of the web, now relying on each service or network to curate a feed. The result is a plethora of these update pushers, each providing a piece of what they think is relevant.

But is what we get always what we’re looking for? I personally always like to keep up to date on Scala (a relatively obscure programming language). I check HackerNews, Slashdot, a couple other geeky websites, whatever. Maybe 1 in every 200 posts is actually related to something I care about – yet when something does show up, I probably miss it if I’m not constantly going back to the refresh button.

And what of the feeds we do get? I personally love cats, but not all my friends do. If they don’t want to see the pictures, then they can take my posts out of their feed, but there’s now way to tell Facebook that they can skip cat pictures but probably want to know about about that party next weekend. They could check “only important” on the subscription options, but how does anyone else know what’s important to each of them?

We need a new way to attain relevance. A way that respects our individuality, our personal preferences, and our time. A way that can handle the medium of updates, not just the static web of text. We need to tell the system what we’re looking for and know that’s what we’ll get, with the peace of mind of knowing that it’s keeping track and making sure we don’t miss out.

What

In Chinese, Gōngfu (功夫) is a compound of two words, combining 功 (gōng) meaning “work”, “achievement”, or “merit”, and 夫 (fū) which translates into “man”, so that a literal rendering would be “achievement of man”. Its connotation is that of an accomplishment arrived at by great effort of time and energy.

 

-Wikipedia: Kung fu (term)

Clay Johnson claims we need an “Information Diet,” a new set of habits and entirely new relationship with the information we consume. It’s a good metaphor in some ways – we do get to choose what we read, right?

Only, more and more, the way that we consume our most common form of new information, the update, is kind of like the high school cafeteria. What’s for lunch is what’s for lunch. You can skip meals (turn off your notifiers), try to shop around for the healthiest school (pick better people to follow and sites to read), and max out your variety (balance view points), but you are still stuck to what someone else has been cooking up. And more often than not, it’s the update equivalent of mystery meat – even the content provider isn’t sure what that Tweet actually means.

Reading from the Facebook stream, or HackerNews, or Slashdot is kind of like eating by popular vote. And when it’s a curated stream, whether the CNN website or my favorite webcomic’s RSS, I’ve just outsourced the choice to some stranger.

What we need is control – not just to hold the menu, but to write our own. A way to tell the feed that I am looking for new Scala-related posts and articles, and for my friends to express the fact that they really enjoy partying with me but care less about my cat. And I don’t really want to go on a “diet” – that’s hard and painful. I want something else to do the work for me… kind of like how a search engine saves me the pain of handpicking all those results.

There are many dimensions to relevance now, because the web isn’t just about the old notions of “information.” Scala is an easy case to search for, a pure topic, but that’s because it was an easy thing to handle before. It also happens that I’m a big fan of jazz parties, but only on Monday, Tuesday and Saturday nights, in New York City, and it matters to me that “party” connotes a fun event and not a political organisation. Social media made that kind of thing a possibility, but trying to search it is just a mess.

Enter SiftFu – a website that lets you construct update feeds out of all these criteria. SiftFu is configurable, but it doesn’t require you to spend ages figuring out exactly what and how to look through – tell it how specific you’d like to be in your sources (from “just this RSS feed” to “anything on the web”), what subject(s), time(s), and place(s) you’re interested in, and SiftFu will keep track of new happenings. It’s not a search engine – search engines are designed to find static content, especially the older variety. SiftFu tracks what’s new and organizes it specifically to show you what you probably didn’t want to miss, so you can feel free to turn off notifications for a couple of hours and just let the computer do its job. Additionally, it’s designed to prevent a single source, user or topic from drowning out your view – SiftFu will make sure that you get the whole picture, and maybe even make it easier to see how it all fits together.

How

SiftFu is not a search, but a sift. Instead of presuming that your interests are fleeting and new, SiftFu holds on to what you’ve put into it and gathers new information from the Internet. It’s kind of like an information matchmaker, always looking for new content that just came online and checking relevance. SiftFu isn’t yet big enough to get the whole thing, but it can take a pretty big sample – in order to bridge the gap for now, we’ve created several ways to set up the sources that matter most to you and ensure their careful consideration.

We’re launching an alpha soon, so if you’re interested… well, this blog has a feed you can subscribe to. Hopefully I get the email list working before someone goes ahead and shares this post somewhere important…

Stepping Backwards in Real Time

1 Reply

Think about the early 90s, late 80s, pretty much any time between the beginning of Compuserve and that of Google.  You’d log on through what was probably AOL and find content waiting for you – their content.  The walled garden systems of old assumed that your ISP knew what kind of content you should consume.  They taxonomized services according to the categories they knew and billed by the hour for what they provided.

It took a while for the idea of a free, open pipe to the rest of everything to catch on.  I was late to the Internet and aren’t sure when.  Probably somewhere between the fall of AOL and the rise of search over the web directory (another kind of walled garden in its own sense), people naturally gravitated towards entering a term they made up into a search box to scan for relevant content.  No more trying to self-pigeonhole into the established taxonomy.  Finding what you want became a game of querying and adjusting, figuring out the best way to tell the Internet what you wanted.

There were probably many reasons why this happened, but I’m going to assume that the content was just better.  Not necessarily because it was objectively better (though PageRank probably helped that too), but because a prebuilt taxonomy has some natural limits to its capacity to capture the interests and wants of human beings.  If your interests happened to lie perpendicular to the web directory taxonomy, chances are you’d find results that were maybe 10% or 20% relevant in a related category – but you’d not only be sifting through tons of irrelevant junk, but you’d miss out on everything the curator didn’t specifically decide to put into a category barely relevant to your own.  So if you happened to like, say, Kabuki theater, you’d probably peruse the theater section for anything relevant and be lucky to find an article or two that you really wanted.  On a search engine, however, you could specify exactly what you should get and know that the voracious Googlebot would have crawled nearly every accessible page.

So problem solved, right?  We have Google, Bing, Yahoo, DuckDuckGo… plenty of ways to search.  Old news.

Old news – that’s exactly the problem.  Search is built on the same assumption as browsing – that you’re the one moving around, and the content is going to sit and wait to be found.  So we can search, but when the content stops being perennially valuable, those results are probably stale before we see them.

Now think about what tabs you have open.  I can tell you mine.  Facebook, Twitter, HackerNews, Gmail, 7 Youtube videos (mostly music), a couple Wikipedia pages that I checked before I wrote this, a map, and a Google search.  The Wikipedia pages do their jobs as well as could be expected – I have something I need to look up, and I can find it.  Youtube I’m sort of misusing as a lazy man’s Spotify/Pandora hybrid, but that’s a whole other issue.

My concern is with the feeds – the 4 tabs of news and social media I’m constantly refreshing, plus the 10 or so blogs/webcomics that I generally close and re-open.  I find 3 articles that look interesting on HackerNews.  One of them is relevant to one of my interests (startup fundraising) but not very useful.  Another is emotional linkbait that might’ve been interesting to know, but probably isn’t important to what I’m doing.  The 3rd… same deal.  The rest of the pool are clearly a mix of technical and newsworthy topics that I don’t have any particular interest in.  So not only am I sifting through 30 articles that are mostly, obviously irrelevant, but the fact that I can’t through headlines means I’m bound to get halfway through a post before I really know what it’s about.  Facebook and Twitter are much worse – while HN is probably about 10% relevant on average, the social networks are showing random selections of content taken from people I friended/followed because… we’re friends.  I have 8 “outstanding” requests for games I don’t want to play, the stereotypical cat and baby pictures, a couple checkins to bars in cities I don’t live in, and maybe somewhere in there an event I actually wanted to go to but might not even see.  In theory, I could pick the people I want to hear from by carefully managing subscriptions – but not the topic or quality of their posts.

Furthermore, these updating media represent 4-10 tabs in my normal browsing.  There is no unified way to all the new stuff about Scala (a programming language in which I’ve had recent interest).  I could maybe hand-aggregate a bunch of the blogs and news sites on that particular topic, spending hours on aggregration that could be machine-driven and missing sources I just don’t have the time to find, and then plug them into Google reader.  And what if the topic is more specific or obscure – what if I’m really only interested Scala for a particular feature (such as parallel collections) and don’t really care about all the theoretical debates on programming style?  My RSS feed would be full of crap.

I could of course try to Google the subject.  This takes me back to that static web – I get 3 relevant results on the 1st page – the official website for the language, the Wikipedia page, and the site for the popular Lift web framework.  All of these I’ve read before – generally to the point of boredom.

Then there are Google News Alerts – alternatively trap.it or DailyPerfect.  These work pretty well for finding the news.  I just don’t always want “the news,” which seems to have narrowly become defined as content published by a certain collection of media and journalism outlets.  I want blog posts.  I want Tweets linking to them, interesting new discussions on forums, and release notes when they appear.  And I want this content searched and organized – don’t give me 10 articles on MongoDB scalability because the RSS reader’s regex filter found “scala” embedded in “scalable,” and don’t make me handpick 50 feeds that might have relevance when there are 10,000 of them out there on the web.

There’s a tradeoff.  Freshness, relevance, or automation: pick 2.

The post-2.0 web, that of updates and user-generated content, hasn’t yet caught on to this notion of an open Internet.  We are still in the world of privately walled gardens – Twitter, Facebook, Google+, Bloomberg News, Github, CNN… each with its own content stream, separate from and frequently incompatible with everyone else’s.

We (finally) have basic search on each of these streams, individually.  I can look for hashtags or keywords on Twitter, or on Google+/Sparks.

We have real-time search: Topsy, Greplin, etc. – pick your provider really.  But search isn’t quite right – again, it’s based on this idea that my interest in what I’m looking for lasts for the hopefully-less-than-a-minute span between when I realized I had to look it up and when I found it.  That’s not what I’m trying to do when I “search” Twitter.

What we need is an analog to how search is a portal.  When I go on the internet with a new interest, search gives me a best-case-scenario way of saying what I want and getting it – that’s why it initially beats the crap out of categorized directories.  It’s like somebody is making up a category on the spot, and that category corresponds to precisely what I am hoping to find.

We need to do that totally personalized category thing, but turn it around.  The web no longer sits and waits to be discovered – it happens.  I’m the one waiting.  I want to know when that super-insightful blog entry tells me exactly how to hack Scala 10x better, or when my friend 2 blocks away checks in to that cool jazz bar I’ve been meaning to investigate.  I don’t want to “search” for this information, because it would get tedious – imagine keeping a list of 10 Google searches to make every hour just to know what’s happening.

So instead of searching for new/momentary interests across the web of pages that rarely change, I have relatively constant interests and want to receive news based on them.  So just flip things around.  I enter a “search.”  I leave it.  Items accrue to that search as they happen.  I check back in 5 seconds, 5 hours… 5 days – it shouldn’t matter.  I’ll rely on the system to keep track for me.  It should know what I’ve seen, what I’ve missed, what I want and what I care about.  And when I check again, it should present it like a secretary.

Does this exist?  Can it?  It’s trying to sometimes.  It’s close.  But I while I don’t think it does yet, I think it most eminently can.

I don’t know how much we’re missing out on due to the non-existence of software that does what I just described.  Maybe it’s merely an additional feature on top of the growing notion of “aggregation” – a fusion of Tweetdeck, RSS, and Greplin.  But maybe there’s something more here.  Maybe we are missing not just a few pieces of Internet, but a substantial fraction of the entire thing.  Maybe we have all grown accustomed to consuming that which is 10% relevant, because we’ve never known anything else.

Observations on Writing

Leave a reply

When you want to create something really scary, often the best way is to refuse to resolve tension.  Be it a creepy monster creeping outside the windows or a bad deed about which mom has yet to find out, setting up a danger that will appear later starts a cascade of tension and uncertainty.  Conflict with immediate resolution usually transforms itself into more of an action movie aesthetic, in which the characters progress through the plot in a viscerally noticeable but often empty way.  In contrast, great horror films frequently deny even the most basic progress.  Coverfield, one of my personal favourites, goes through most of the film without a clear view or answer to what is wreaking such havoc.  In another scene (may have been Event Horizon, but I’m not sure), the characters stand at a fork in some tunnels on a space ship as ominous thumping noises suggest the approach of a large enemy.  Just as it reaches its peak, the thumping stops with no visual suggestion of what caused it – one of the characters peers down a tunnel to no avail, and the source never gets a satisfying explanation.  Rather than diffusing, the tension builds as it hangs.

Take the audience into a ridiculous situation without seeming implausible.  To do so is to master the comedic art of retell-able scenes.  One way to achieve this feat is by seeking to fill a hole left earlier, making new events fit perfectly into a reader’s mental map of the story even as they have no business is realism.  One example comes from a college improv show, in which a very early scene made passing reference to the ninja turtles.  Around 10 minutes later, well after the original gag had come and gone, a character who had done some slightly naughty things received the admonishment, “don’t make us call in the turtles!”  Another comes from the movie Dead Men Don’t Wear Plaid, in which consistent references to a certain past incident eventually justify what would otherwise qualify as a Deus Ex Machina.  The ridiculous can seem realistic when it closes up an otherwise loose end.

Another key to comedy is understanding that the audience has a vested interest in suspending disbelief.  We want to laugh – that’s why we bother to watch or read.  It is much the same with any sort of event or situation – let the audience have a good time.  Don’t force it.

Make characters that we can love and hate at the same time.  These are the best characters, the ones that I at least spend the most time thinking about even after the performance is over.  The ones I go over and over in my mind, wondering why he couldn’t have made a different choice in this one place, or what if I had only seen one side.  They may not become too many readers’ favourite characters but will definitely contribute to the choice of book.

As a corollary, don’t make antagonists too crappy, unless you can afford not to rely on character development.  I suspect that a good villain inspires in us the fear of our passions more than our failures.

Take risks, but hedge.  If you offend your audience, they leave.  If you apologize, they bore.  You can get away with more if you’ve already set up having something they want.  You can guard against offense by offloading the situation onto another layer of fictional occurrence.

Have subplots.  These are like the supporting instruments in a song.  Sometimes you can go without them.  Other times, they create the amazing whole.

Finish.  Before you get too tired.

Why I Will Always Be Better at Writing than Programming

There is a certain type of person who enjoys the most technical aspects of software sufficiently to build a life upon them.  This mythical superprogrammer lacks the romantic danger of the blackhat hacker, yet matches his darker counterpart in skill and intellect.  The superprogrammer can outprogram all of his colleagues and disdains promotion to management.  The superprogrammer will routinely shut himself away for days, emerging with with a work of utilitarian beauty only he could ever understand.  The superprogrammer is every businessman’s dream – all of the skill and motivation upon which to build the company and none of that Napoleonic ambition to take it away.

I once thought I was a superprogrammer, albeit with that deadly streak of ambition.  I was a physics major then, so everyone else assumed I had a professorship solidly before me.  I thought then that I was a terrible physicist but not so bad with computers.  I programmed often in spare hours, learning computer languages for the feeling of intellectual superiority it gave me and constantly churning out my next idea for the smallest next big thing.  I remember the night that I first “finished” a software project, a complicated intertwining of threaded AVL trees and gap buffers that would power the secure self-tracking filesystem of the future.  Triumph soon degraded into exhasperation at what I still had left – user interface, system integration, installation…

I never finished that piece of code.  It’s probably still on at least 2 of my backups from God knows when.  That winter was the first time I tried to take the upper level of modern algebra (a great course for those who want abstract, general mathematics) and found myself lotteried out.  A little alteration in the brain shunted me out of the trap, and I took acting instead.  Something about being on stage changed me as a person, or really just brought out the person in me that had been buried under grades and IQ tests.

I was a member of the creative writing club soon later and immediately proceeded to do what I did best and learn fast.  I’d always procrastinated the desire to write until it had some social purpose.  Soon the rejections piled in, from the college program and the relevant competitions and even those closer to me.  I kept writing, because it was too late not to.  I didn’t want to stop.  Ideas just happened, and they wound up in stories.

It took me another year to figure out why I wanted to be a writer.  When I hack out a line of code, there is that compilation->run->test cycle that gives me green lights and tells me that I’m brilliant.  It’s the report card every few months, the Pokemon cards I used to get as a kid in return for my good grades and meek behavior, the robotic authority that constantly says yes with so little effort.  It’s the girlfriend I never had, the father I’d learned to internalize and the God I’d learned to fear all wrapped up in beeping lights.  Writing does not do this for me.  It takes a complete work before validation meets satisfaction, and even at that I should always strive to find someone to read it.  Or should I?

When I write, I often lie to my readers.  I don’t mean this sinfully or confessionally, but practically.  Code is neither girlfriend nor God nor father, but I have written that it is, because that is true in my version of the universe.  My compiler also lies, because it tells me that everything is ok when the business is failing, tells me I’ve accomplished something by wasting more time on features no one will ever use, or tells me I’m wrong when that line should have been legal.  It’s worse than that.  It hands me what has slowly become an empty validation.  The degree offered by this terminal grants no privileges or value in life except that directly related to what I’ve just typed, and yet it has the nerve to light up green and tell me that something was “successful.”   It has not been the first to lie to me this way, but I wish it were the last.

Code is full of semicolons and brackets, or parantheses with fully-defined evaluation order, or maybe function call syntax and overloaded operators.  None of these mean anything to me – they are all boilerplate parts of the language; a decent programmer could look at well-written C with the brackets removed and know immediately where they should go.  Despite the vapidity of these constructs, the compiler insisted on triggering that little reward/punishment mechanic until I associated neither with anything but nihilism and empty symbols.

Writing, on the other hand, forgives.  It does not much care whether what I have written is sound or valid – that’s up to the reader to pass along their prejudices.  Writing lets me branch out and reach the same way that I can on stage.  Writing lets me wander through the city with vague destination.  Writing encourages me to express the momentary and subjective truth that happens to meet my hands on the keyboard; it does not bully me into thinking about its future propriety.  This is not to say that writing lacks grammar, but that on some level, grammar is still a choice.  I have written words made of numbers before, stories without protagonists and long strings of punctuation meant to express the feeling of information without buying a narrative on top.  Writing asks that I choose well, while programming requires that I choose correctly.

I want to reach out with my mind and bend reality.  Even in a virtual world, I still have the urge to send forth tentacles of thought into the abyss and work the shapes and colors into form and void.  I care not for the validating return of properly-placed semicolons; it feels like driving over a speed bump on the road to nowhere.  Some hackers claim not to feel it in their favorite language, but I find this feeling universal to the trade itself.

Sometimes I wonder if the superprogrammer even exists.  A part of me wishes to blame business for this urban legend, but I empirically believe it to be true.  Regardless, the superprogrammer is a societal pariah in his own way – unlikely to partake in the pleasures of the rest of us, the superprogrammer carries the weight of society purely for the joy of labor.

Neophilia vs. Doing Things the Easy Way

When I first started working on Fear of Software, I had enough burnout/business instinct to realize that using fancy new tech wasn’t going to be the deciding factor in the fate of the company… so I decided to start immediate and fruitful prototyping by coding things in a venerable language called Java and then hack-deploying over the wonderfully obsolete method known as Java Applets.   These were little Java programs you could stick into a web page, and they’d have access to the same Java as programs that you need to download and install… in theory.  When I’d first used Java years ago, applets were a mess.  When I started Fear of Software, they’d hardly changed at all.  I also knew that Java would cost me some productivity in terms of being an old fussy programming language, but the support for what I was trying to do seemed to make up for it.

About 2 months later and with almost nothing to show, I decided to port everything into Scheme, an even-more-venerable language that had aged relatively gracefully and was always known for its elegance and appeal to the mathematically minded.  Another few weeks went down the drain, and I was back to Java.

More months, some wasted, others burned.  Finally came the prototype release day, at which point I’d switched over to a special tool called Google Web Toolkit that could turn my outdated Java into a modernish webapp.  Painfully, a release came out.  We got feedback, learned lessons and gathered up what was left of us to restructure.

Of course we switched languages yet again.  This time to Scala, an up-and-coming programming language that appealed to serious hackers and some semi-famous corporations, like Foursquare and Novell.  This was the first time that I felt a programming language actually make a difference.  I get the feeling that the Lift web framework built about 60% of our app for us.  All sorts of things that could have taken hours of learning and doing actually turn out to be included in the framework itself, and they are fast, secure and frighteningly convenient.  I wish I’d known this last year.

Yes, you should avoid neophilia and getting wrapped up in technology, but getting better tech is like night and day when your company runs on it.  It will make your life easier.  It could mean the difference between trying to explain your idea and having a demo.

Lessons learned:

  • Market is still first… don’t obsess over platform.  Use mockups and videos whenever you can, and don’t be afraid to shelve code and make mockups one or two months in… this will save you the other five months you could have spent chasing sunk time.
  • Using a crappy platform really does screw over your business.  If you don’t know what a crappy platform is, ask someone who does.
  • When you do choose a platform, spend about a day researching what you think your project is most likely to use, and then choose a good platform that offers the most of that.  Don’t get caught up in language popularity contests or hipness – you want to be ahead, not with the pack.  Think about the coming months but recognize that requirements change.  When in doubt, choose the fanciest and best technology you can get for free and then get back to worrying about your market.

Insomnia -> Invention

Some days I throw off my sleep schedule so thoroughly that I never quite find it again.  Tonight marks the aftermath of one such misadventure, and while I had every intention of making tomorrow an early start, my mind is alight with flows and webs of ideas.  I cannot help but connect topics I’d been waiting to see for weeks.  It’s a strange sort of dance between brilliance and idiocy.  One moment I have connected three topics of curious uncertainty, and then next I have forgotten what one of them was.  My introspecting senses flit around, darting between objects and concepts.  I slip easily into fantasy, a price I pay for letting the wanderings take me into newer places.

Why now?  Is it the mix of caffeine and off-ness of not sleeping when I should?  Did I discover something subconsciously today that must rise before I can sleep?  Maybe.  I once did not eat any of the fried, processed or sauced food offered in Chinese take-out for probably two years.  When I finally decided to splurge on an incredibly tasty dish, the MSG sent me into creative overload for hours.  I could almost see the flashing lights spark through to writing.  Today is a different kind of creative energy – one that prefers long walks through fog and thoughts made at the gateway to sleep.  I suppose it is more relaxed but no less tame than the sparking fusion of earlier attempts.

I spent hours today tracking down songs that were semi-disappeared from the Internet and all other widely available media.  I found it easy but boring to skim texts on web architecture.  I nervously await the trials of tomorrow, all the responsibility for getting visas, customer development, website administration… big words I don’t care to use anymore.

I look forward to sleep.  While difficult to come by under these circumstances, I suspect it shall be most glorious.  The ideas have risen, and they now beg for solidifying rest.

Cognitive Dissonance in Project Setbacks

Certain kinds of projects, be them the software engineer’s Mythical Man Month or the impressively delayed Sydney Opera House (as mentioned in The Black Swan), violate all typical intuition about effort and its relationship to rate of completion.  In software companies especially, once a project has delayed a few of times, it will almost inevitably delay again, sometimes to the point of never finishing (see the story of Duke Nukem Forever for an extreme case).  Numerous theorists and observers have tried to explain the perpetual project, be it as a spectre of the information age or a natural extension of chaos in overly complex systems.

I propose there is at least one simpler mechanic at play: cognitive dissonance.

When a project delays and then continues past its intended deadline, morale probably takes a huge hit.  The team will inevitably feel they have screwed up, even if management does its best to assure them that the failure will be corrected.  The original prize, the dream of all weeks past, is already lost.  The incentive structure is now centered around damage control and avoiding blame – this alone presents a significant challenge.

By telling a team to continue after a missed deadline, management implicitly sends the message that such delays are a standard part of the company’s functioning.  The effect compounds after multiple reschedulings as each one reduces the weight of the next.  I saw this in school, where students who failed to graduate the first time around tended to hang around for several years after, despite the rather trivial nature of what was technically left for them to finish.

I don’t have volumes of data or experience on which to confirm this hypothesis, but I can make a testable prediction: pressing deadlines earlier may actually increase the average time it takes to get things done. Conversely, setting realistic deadlines may actually speed up progress holistically, even in areas unrelated to those particular deadlines, as it creates a culture of closure rather than of protracted struggle.  These principles should apply to life as well as business if they hold correct.  Maybe the secret to speeding up is trying to slow down.