Between My Ears

Solving Causes' Levenshtein Distance challenge in Python, the Sequel

2012-01-29T09:47:00.000Z

The 3 faithful readers of this blog have probably seen my previous attempt at cracking Causes' Levenshtein distance challenge. It all went well until Adam Derewecki of Causes commented with the following:

...Pretty good solution though, about 15s on our benchmark machine. Record is 11.3s if you're up to the challenge :)

At first I was like "Yeah right, mate, you're not roping -me- in with that one, I have a startup to run." But the predictable engineer's mind just couldn't let it go. How could someone have done about 30% better in Python? What was I missing? So, I started hacking at the code again. Turns out (surprise!) I was missing quite a bit. Let's start with putting the original code up for you to see:

import string
w = set(open("00wordlist.txt").read().splitlines())
f, nf = set(), set(["causes"])

#from b, yield all unused words where levdist==1
def nextgen(b):
    for i in range(len(b)): #for each index in b
        for c in string.ascii_lowercase: #for letters [a..z]
            if c != b[i]:
                #substitute b[i] with c
                if b[:i] + c + b[i+1:] in w:
                    yield b[:i] + c + b[i+1:]
                #inject c before b[i]
                if b[:i] + c + b[i:] in w:
                    yield b[:i] + c + b[i:]
        #remove b[i]
        if b[:i] + b[i+1:] in w: yield b[:i] + b[i+1:]
    
    for c in string.ascii_lowercase: #for letters [a..z]
        if b + c in w: yield b + c #append c after b

while len(nf):
    cf = nf
    nf = set([j for i in cf for j in nextgen(i) if j not in f])
    w -= nf
    f |= nf

print len(f)

First, Adam's suggestion was very good by itself. Why write this:

nf = set([j for i in cf
        for j in nextgen(i) 
            if j not in f])

when you can omit the intermediate array and just write this:

nf = set(j for i in cf 
        for j in nextgen(i) 
            if j not in f)

But it gets better. Since I subtract nf from w, from where the values are sourced, why even check if j not in f? No reason. So, we end up with the much more palatable:

nf = set(j for i in cf for j in nextgen(i))

After improving that line, I noticed that I had a line above that was doing absolutely nothing whatsoever:

cf = nf

This line simply betrays my uncertainty about how python's comprehensions work. It turns out the next line can be simply written as follows, with no need to ever declare cf at all.

nf = set(j for i in nf for j in nextgen(i))

Next up, let's look at the little optimisation I had in line 9:

if c != b[i]:

Here I used a whole line to check that I wasn't going to be doing any useless checks. Even though I was aiming for small code. Even though Python has O(1) membership testing. When I looked again at the code and doubted my own premature optimisation, the results were damning: The test cost more time than it saved. Removing that line yields a speed improvement.

All these improvements were small. They saved 1-2 seconds over the total of 25 seconds it takes on my laptop. The big improvement came when I tried the technique seen in this stackoverflow answer. Interrupting the program while running for a few times indicated the culprit. The constant use of the slicing operation was not doing me any favours. For every given letter and every position in a string I did operations like this:

if b[:i] + c + b[i+1:] in w:
    yield b[:i] + c + b[i+1:]

That's 4 slice operations, and actually this is done twice for a total of 8 per letter. So I decided to do the slicing only once per position, assign the results to variables and use those for each letter. That sped things up enormously. It brought runtime from slightly under 23 to well under 17 seconds.

UPDATE: After some impromptu after-work tinkering with my co-founder Pagan, we realised Python iterates over lists faster than over strings, which means that adding the line

letters = list(string.ascii_lowercase)

to the setup part of the code speeds things up by a cool 4%.

All these improvements add up to 1/3 of the total running time. Since Adam said that my programme ran for 15 seconds on the benchmark machine, while the best Python they had ran at 11.3, I suspect this may be enough to beat the frontrunner. Now I just have to get Adam to test this one again.

Another change I did is improve the horrible variable naming I had last time around, and also add a few more comments. I also was very strict about keeping lines under 65 characters in length. So here is the resulting program:

import string
words = set(open("00wordlist.txt").read().splitlines())
frnds, newfrnds = set(), set(["causes"])
letters = list(string.ascii_lowercase)

#from word wd, yield all unused words where levdist==1
def freefrnds(wd):
    for i in range(len(wd)): #for each index in wd
        wd_upto_i,wd_from_i,wd_after_i = wd[:i],wd[i:],wd[i+1:]
        for char in letters: #for letters [a..z]
            #substitute wd[i] with char
            if wd_upto_i + char + wd_after_i in words:
                yield wd_upto_i + char + wd_after_i
            #inject char before wd[i]
            if wd_upto_i + char + wd_from_i in words:
                yield wd_upto_i + char + wd_from_i
        #remove wd[i] from word
        if wd_upto_i + wd_after_i in words:
            yield wd_upto_i + wd_after_i

    for char in letters: #for letters [a..z]
        #append char after word
        if wd + char in words: yield wd + char

while len(newfrnds):
    newfrnds = set(j for i in newfrnds for j in freefrnds(i))
    frnds |= newfrnds #add newfrnds to the frnds set
    words -= newfrnds #remove list of newfrnds from words

print len(frnds)

A 3-rectangle 17x17 grid

2011-11-03T16:28:00.002Z

I've made no secret of my obsession with the 17x17 challenge. I started working on it in November 2009 and went straight at it for six months. At that point I had to stop to code/write up my PhD but started working on it again as soon as I was done. This problem has given me a reason to learn so many amazing things in both math and programming, that I would be happy to have worked on it even if I never produced anything worthwhile. It's now a weekend project given that I run a startup, but, after almost 2 years, I have something to show the world: A grid with 3 rectangles.

To be clear, this is not a solution. It would need to have 0 rectangles to be a solution. But it is the least broken solution I know of. Bill Gasarch posted with the problem a 4-rectangle grid by Rohan Puttagunta, but this has not been improved on since. Here is a leader board of the best grids known so far.

This solution is less impressive than that one in that it's missing two cells rather than one, but it does have fewer rectangles when extended to a full colouring, which is what makes it interesting. Without further ado, the solution, with the rectangles marked out:

If anyone has code they want to use to analyse this, here it is again in a machine-friendlier format:

4,2,1,3,1,2,3,4,4,1,2,4,2,1,4,3,3

2,4,2,1,3,2,1,4,1,4,1,2,4,3,3,4,3

1,2,4,1,3,1,3,4,3,1,4,2,3,4,2,2,4

3,1,1,4,3,1,4,3,4,2,3,2,2,4,1,4,2

1,3,3,3,3,3,4,2,2,4,1,4,2,1,1,2,4

2,2,1,1,3,4,4,1,2,3,4,3,4,2,4,3,1

3,1,3,4,4,4,3,4,1,1,2,3,1,2,3,2,2

4,4,4,3,2,1,4,3,3,3,1,2,1,2,3,1,1

4,1,3,4,2,2,1,3,2,4,4,1,3,1,2,3,1

1,4,1,2,4,3,1,3,4,1,4,3,2,2,2,1,3

2,1,4,3,1,4,2,1,4,4,3,3,3,3,2,1,2

4,2,2,2,4,3,3,2,1,3,3,1,4,4,1,1,2

2,4,3,2,2,4,1,1,3,2,3,4,1,4,1,2,3

1,3,4,4,1,2,2,2,1,2,3,4,4,2,3,3,1

4,3,2,1,1,4,3,3,2,2,2,1,1,3,2,4,4

3,4,2,4,2,3,2,1,3,1,1,1,2,3,4,3,4

3,3,4,2,4,1,2,1,1,3,2,2,3,1,4,4,3

In case anyone hasn't noticed, this solution is symmetric, i.e. colour(x,y) = colour(y,x) if you start numbering from the top left.

I have a lot more to write about the solution and method, and I'm not yet done with this approach, but if I start writing up I may never publish this, so I'll just stop here for the moment.

An accidental survey of the Hacker News ecosystem

2011-10-02T22:33:00.000+01:00

So my little rant a week ago made a bit of a splash, briefly occupying the top of Hacker News. Looking at the inbound traffic, what surprised me was the variety in the HN ecosystem, particularly the alternative front ends that people use to access the HN stream.

Writing a HN front-end is an uncertain proposition: You only get to talk to your audience once - at launch. You don't have a viral loop, so users will most likely be within that initial spike, plus whatever you can get from word of mouth. There are a few online listings, but then the question is how many will get to see those without a static link from HN. This is my attempt to bring a little more attention to these very cool but underappreciated projects.

What I saw in the logs, is that at least 3% of the HN traffic came from alternative front ends. The number may seem small, but zooming into those visits reveals the surprising variety of ways people consume Hacker News. At this point I'd like to apologize to the coders of native apps: I can't see you in my logs, because there is no referrer in the traffic you sent my way. I know that 20%+ of the traffic I got is unaccounted for (and it damn sure wasn't 3600 people dropping in to see what's on my blog), but I can only guess where it came from. With that said, here's the situation, if my analytics dataset is to be trusted:

The most popular alternative HN front-end is hckrnews by @wvl, sending 69 visitors over. hckrnews competes with the default front end head-to-head by offering multiple convenient ways to access the submissions. It is also well integrated with the Hacker News plugins made by the same author for Chrome and Safari.

iHackerNews targets an unmet need, namely the lack of an official mobile website. iHackerNews is built by @ronnieroller and also offers something else much requested of HN: an API. 51 visitors were iHackerNewsers.

The hardest to measure HN-alternative of sorts is @cperciva's Hacker News Daily. The idea is simple: A feed with each day's 10 most upvoted submissions. It covers the need to know that even if you're gone for a day or two, you can still see the best posts when you come back. The reason it's hard to measure is that only a fraction of people go through the website, that sent 31 visits. I would expect most to consume this through the feed (as I do too). Google Reader tells me that Hacker News Daily has 1061 subscribers. That's significantly lower than the 36.564 subscribers to the main HN feed, but I suspect readers are much more likely to be engaged, since they get only one feed item per day.

Another mobile-friendly front end is hn.gethifi.com. As far as I can tell it's been made by @JoelSutherland, @KrisJordan and the gethifi.com team. While not as feature-complete as iHackerNews, I do prefer the look and feel which looks more tailored to a mobile device. 13 visitors came that way.

12 more visitors came through another mobile-optimized site, iCombinator.com. The selling point is the instapaper integration as well as its optimization for iPhones by using iUI.

For the fans of a more traditional reading experience there is Hacker Newspaper by Giles Bowkett. As you might expect, it renders the stories of the HN front page in a newspaper format. I imagine this would work quite well with a tablet. Hacker Newspaper sent 10 readers.

An interesting take on the HN frontpage is hnsort.com, which allows you to sort by points, comments, domain, submitter, and age amongst other things. 4 hnsorters showed up.

Another 3 came via one of my favorites, Hacker News Reader, which is a serverless app. This means that you download a static file that when executed on your browser, fetches the HN frontpage, parses it, and presents it in a different way. No server is involved, which means that if login was implemented, your credentials would never need to touch the developer's server. HN Reader is also installable as a Chrome Web App. While feature-limited, it looks optimized for mobile and also rocks instapaper integration. Definitely one to keep an eye on.

Then there is the long tail: The alternative front ends that barely registered by sending a single user my way. Since there are so many projects in this list, I am sure I may have missed as many if not more that didn't send a user to my post. There's fuhn.tk, the only project that explains its mission with a rage comic.

Then there’s HN overload, which nicely organizes each day's top links in a clean format. It scores by aggregating hn points, reddit karma, and number of retweets. Definitely tempted to use this one more.

There's Hackerslide.com, which organises hourly snapshots of the HN front page using an etherpad-style timeline. Genius.

hnvue.com promises to optimize your HN reading experience by allowing you to see both the front page on the left, the article page on top, as well as the comments page at the bottom. A very interesting experiment.

Another very clean and mobile-friendly site can be found at yhack.net. I like the minimalist well spaced layout as well as the refreshing blue colour scheme in a sea of orange competition.

As is obvious there is a lot of variety in the HN ecosystem. I hope at least one of these apps convinced you to give it a try, even if just for a day.

Besides the dedicated HN front-ends, I really should also mention the aggregators: websites that pull together an HN feed with feeds from other relevant sites. The main ones are jimmyr.com, popurls.com, and hackurls.com, in this order. Aggregators sent 94 user agents this way.

Another benefit of being able to dive into the referrer logs, is to see from what part of HN people came through. If you're wondering what that means, read on. There's /best, where articles rise and fall much more slowly, giving you access to great articles you may have missed over a few days of absence. There's /classic, which applies a different sorting algorithm. I believe it has something to do with usng only veteran users' votes. Then there's one I didn't know existed: It's /over. This allows you to set the threshold of points you want to see articles, well, over. Going to /over?points=100 will show you the latest(?) articles that have over 100 points. As for the two people who came with news.ycombinator.com/rss as the referrer, you sirs, are gentlemen and hackers.

Since we're talking data, another lesson I learned is that these 'follow me on Twitter' links you see at the end of some blog posts? They get clicked. I added one to my previous post a few hours after the initial spike, and I estimate it brought about 1.5 new followers per 1000 reads. Not earth shattering, but not too bad, either.

What do those links look like? Well, something like this:

If you read this far, why not follow me on Twitter here.

Why Facebook's 'Frictionless Sharing' violates HTTP

2011-09-25T16:40:00.000+01:00

Facebook has this new feature, whereby the act of simply reading a web page, under certain conditions, gets it posted to your news feed, for your friends to see. Here's how ReadWriteWeb puts it:

With these apps you're automatically sending anything you read into your Facebook news feed. No "read" button. No clicking a "like" or "recommend" button. As soon as you click through to an article you are deemed to have "read" it and all of your Facebook friends and subscribers will hear about it. That could potentially cause you embarrassment and it will certainly add greatly to the noise of your Facebook experience.

Facebook calls this 'frictionless sharing'. This has raised all sorts of ‘creepy’ flags, and rightfully so. A big reason for this is that it breaks a fundamental contract of web interaction, in place since the beginnings of the web, that users have come to rely upon.This contract is the fact that merely browsing a webpage (Executing a GET in HTTP talk) should not cause effects that you, the visitor, are responsible for. Posting to your news feed is a side-effect, is a direct side-effect of your reading the article. You take no extra step to authorize this.

This violates a convention that is not there by accident. The HTTP Specification defines GET as a ‘safe’ operation, with certain guarantees. This line has been skirted for a very long time, but never by a company of this size, so publicly, and so blatantly.This is what the HTTP Spec has to say on the matter:

9.1.1 Safe Methods

Implementors should be aware that the software represents the user in their interactions over the Internet, and should be careful to allow the user to be aware of any actions they might take which may have an unexpected significance to themselves or others. In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe".

[…] Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.(emphasis mine)

I don’t think it gets any clearer than that. It’s as if the HTTP committee had looked into the future and was personally addressing Mr. Zuckerberg. Now, the HTTP spec has no teeth. There is no enforcement body that goes around and metes out fines and punishment to the violators. It is a gentlemen’s agreement and the contract that good citizens of the web should keep. As such, I think it merits at least a mention when large companies find new and ‘frictionless’ ways to undermine the foundation upon which they (and everyone else) is building on.

Update: A number of people are pointing out the fact that the user authorizes the side effects by installing the app on facebook. However, I assume Facebook also agrees to the HTTP Spec by implementing it. Does getting user authorization allow you to violate HTTP? I don't see any such language in the spec. I think the safeness of GET is one of those rights that you shouldn't be able to give away, even if you wanted to, as doing so undermines the web for everyone else.

If you read this far, consider following me on twitter

Three times Google’s ‘strategy’ got in the way of success: Skype, GDrive, Google+

2011-09-18T21:34:00.000+01:00

I just finished reading Sriram Krishnan’s excellent post 'Don’t be so f*king strategic' and couldn’t stop thinking that this must have infected Google as it had Microsoft.

Here are three times when the public learned of missteps of Google that were somehow related to a grand strategy of the company:

Skype

This is documented in Steven Levy’s book ‘In The Plex’ and the author has a more specific blog post on the issue. What is comes down to is this: Someone at Google thought, and was able to convince the higher-ups, that peer-to-peer is old technology, not consistent with their cloud model, so Skype was worthless to them. The fact that this someone was from a product group that would have to compete with Skype internally goes unmentioned, but what is important is to see that Google in 2009 passed up the opportunity to buy Skype for a fraction of what Microsoft paid for it in 2011. Skype is now integrated with Facebook.

GDrive / Dropbox

Drew Houston worried in his YCombinator application that Google would launch GDrive any day. It turns out Google didn’t and Dropbox is a billion dollar company today. Why? In The Plex has this to say (http://googlesystem.blogspot.com/2011/05/how-google-docs-killed-gdrive.html): The Google Docs team was able to convince the higher ups that files didn’t make sense in the cloud. File systems were a thing of the past and so GDrive was abandoned almost complete, the engineers sent to work on Chrome. It turns out that files aren’t quite as dead yet, and Google Docs itself now allows you to upload them. Recent rumours say that GDrive may have been resurrected and is getting launched, this time in a much more crowded space with credible independent competition.

Google+

The latest news is that Google+ is showing signs of decline. The causality here is not as strongly established, but the early demographic has not been happy with their real names policy. When I found out they were enforcing this policy, I was in disbelief. Surely when you’re entering a new market, you want to be friendlier than the competition, you want to be welcoming to those who were disenfranchised from your competitor. Google proceeded to shoot itself in the foot by affecting other services that the blocked users were on, shutting out ethnic groups that did not have names with a western structure, people who are known by a name that is not their legal name, as well as those who preferred to remain anonymous for security reasons. The statements coming out of the GooglePlex were to the effect that 'Google+ is not for everyone', and that they can't fight all the battles all the time. This is bizarre behaviour on its face, and I've learned that when smart people behave in ways which appear outright incompetent, there's usually higher level considerations at play. It turns out that Google sees Plus as an ‘identity service’, a part in a grand strategy we’re not privy to (but can make guesses about). To put this in plain terms, Google is jeopardizing their bet-the-company move to attack a competitor because they have some masterplan that may or may not be what users want in the long run.

This is three times when Google let their ‘strategy’ get in the way of success as far as we know. I’m sure more are known to the insiders. I hope this has documented what I’ve been seeing watching one of my favourite companies display a fondness for footguns. Here's my unsolicited advice to Google: Stop being so f*cking strategic and just focus on building the world’s coolest technology, what people love you for, before you end up boring like facebook.

Solving Causes' Levenshtein Distance challenge in 28 seconds. (Python)

2011-08-22T12:20:00.012+01:00

Causes has recently put up a coding challenge for prospective engineers who want to work with them. Here is their description, short and sweet:

Two words are friends if they have a Levenshtein distance of 1. That is, you can add, remove, or substitute exactly one letter in word X to create word Y. A word’s social network consists of all of its friends, plus all of their friends, and all of their friends’ friends, and so on. Write a program to tell us how big the social network for the word “causes” is, using this word list. Have fun!

The corpus size is 264061 words. A friend wrote a C solution that solved it in sub-300 seconds and challenged me to do better. Looking at it, it seemed like it would benefit from using Python's amazing set() data structure, so I gave it a shot. After some horrible bugs, I got the right answer: 78482 (including the original word 'causes').

My Python solution is in 28 lines, including comments. It also runs in 28 seconds on my antiquated Core Duo. Someone on Hacker News found that pypy also speeds this up by 70%. Not bad at all.

Other solutions I saw online take longer, which is why I decided to put this on here. Other attempts, (claimed runtime in parentheses): Ruby (50 sec), Ruby again (?), PHP (1.5 hours), Python (4 hours, 1 hour), Python again(?), Python redux(?), newlisp (?), C++ (1 min).

The secret behind my version's performance is Python's incredible set(), which does membership testing in O(1). I suppose the approach of generating all possible Levenshtein distance 1 strings and testing for membership also helped performance while keeping code size down. It's much easier to generate the small number of next steps rather than testing each of 260,000 options to see if they are a next step.

Without further ado, the code:

import string

w = set(open("00wordlist.txt").read().splitlines())

f, nf = set(), set(["causes"])



#from b, yield all unused words where levdist==1

def nextgen(b):

	for i in range(len(b)): #for each index in b

		for c in string.ascii_lowercase: #for letters [a..z]

			if c != b[i]:

				#substitute b[i] with c

				if b[:i] + c + b[i+1:] in w: 

					yield b[:i] + c + b[i+1:]

				#inject c before b[i]

				if b[:i] + c + b[i:] in w: 

					yield b[:i] + c + b[i:]

		#remove b[i]

		if b[:i] + b[i+1:] in w: yield b[:i] + b[i+1:]

	

	for c in string.ascii_lowercase: #for letters [a..z]

		if b + c in w: yield b + c #append c after b



while len(nf):

	cf = nf

	nf = set([j for i in cf for j in nextgen(i) if j not in f])

	w -= nf 

	f |= nf



print len(f)

Using Dropbox as a Host for Static Websites.

2009-05-01T21:18:00.004+01:00

I have been a huge fan of dropbox for a while. What they have acheived is nothing short of amazing. I recently had an epiphany that may make the implications of their synchronization paradigm even more interesting.

What if I could use dropbox as a host for a static website?

Dropbox has a feature that allows you to make some files public simply by dropping them on your 'public' folder. It turns out, that placing interlinked html files works just fine. Case in point:

http://dl.getdropbox.com/u/117449/page1.html

I presume images, etc. would work just fine as well.

What makes this observation interesting is that combined with the synchronization infrastructure of Dropbox, this means that you can keep your static website right in your desktop, edit it with your favourite editor (WYSIWYG if you have to), and have it instantly available online. It may not be the best idea for everything, but for a number of basic uses, it might just be a massive step forward. One of the most difficult steps for beginers to master, is the need for FTP. Dropbox-as-a-host completely sidesteps that and lets new users easily build their websites.

Of course, once a nice little hack is found, the next thought is to consider what can be improved and streamlined:

Can we incorporate dynamic elements to this scheme?
Can the ugly URL be replaced with something better?

I gave myself very little time to write this post, opting to write it now rather than postpone, so these and many other potential questions, my dear readers, I leave as excercises for you.

The Open Business - An Oxymoron or The Future?

2008-09-10T03:54:00.003+01:00

I routinely hear that businesses are not open enough. Even companies that were once renowned for their openness, say, Google, bump up to accusations of intransparency eventually. Then again, in a world where the openness poster child named Wikipedia is accused of being run by a secretive elite, who can be safe?

I happen to believe that these accusations have merit. Google is not as open as it should be. It withholds its search algorithm and its advertising revenue distribution formula. Is this for a good reason? Perhaps. But if you were to grade google on openness, these would be black marks on the score chart. How about Wikipedia? Well, its secret discussions in unofficial IRC rooms and mailing lists are no longer rumours. They are real. Are they serving a real survival purpose? Who knows. That is the problem with intransparency. Nobody knows what happens behind closed doors.

There are many voices that claim openness is a solution to many of the world's problems. The claim is that by increasing openness, all kinds of creativity are released and 'the community' provides solutions. On the other hand, attempts to build businesses on top of open foundations invariably bump up against a trade off. How much openness can we sacrifice without losing the community? How much do we need to sacrifice in order to become profitable? It seems the trade-off is exchanging profit for openness.

So my question is the following: Can there really be a completely open/transparent business or is this a contradiction in terms?

In order to answer this question, we need to answer some more elementary questions. What does an open business look like? Let's take things to extremes. Let's say all conversations between founders are publicly available. Let's say that all company e-mails are also available to the public, along with all chat logs etc. All employees are encouraged to have a blog and of course have no restrictions on what they can say. All code is of course open-source. (AGPL v3 or BSD?) Essentially everything the business does and thinks is out in the open. Anything anyone could imagine to ask about the company is already available. Every involved party of the business has active conversation with the community. It feels a bit like a reality show, but this is how a business designed with transparency as the #1 goal would look like. Nothing more for anyone to ask. Of course, while the past and present are open for anyone to see, future actions are still the monopoly of the owners. We are not discussing a democratic or crowd-sourced business. The public can communicate with the owners and contribute in ideas and code, but ultimately the leverage of initiative belongs to the owners. This of course is just my idea of an open business and not something complete or final. Let's consider it a working hypothesis.

The next questions are: What are the pluses and what are the minuses of a business like this? This is a much larger discussion. I would assume community input and support as well as trust would be much greater. Assuming the employees and owners are competent, this would be obvious to everyone who follows the dealings. The drawbacks are also many: Inability to surprise the competition, a lot of pressure by the community at all stages of development of an idea, difficulty in communication when everyone feels watched, etc etc.

However the killer question is this: Is there even one, small, tiny, extremely slim niche that a business like this could be successful and even advantageous? Remember, this is a for-profit business so profitability is still the goal. The open business just tries to accomplish this without maintaining the secrecy requirements. If a single idea, with a single business model can be found, where an open business could be successful, then the open business concept has a lot to teach us.

The concept is still in its infancy, it has just been expressed for the first time, so it will probably go through a lot of evolution. I am eager to hear your feedback. What is your definition of an open business? Would you become its customer? Do you believe it could ever be profitable?

Tribute to Alan Turing

2008-07-08T00:09:00.004+01:00

There are few people I would consider myself a fan of. The Beatles are great but it is demeaning to call myself a fan. Ditto Radiohead. I just can't see it happening. But there is one person whose fan I can easily declare myself to be.

What if I told you that someone laid the foundations for all modern computers, set the stage for artificial intelligence and also was the most important scientist in breaking the German cryptography in WWII? Doing any one of these things would surely be an accomplishment beyond the imagination of us mortals. But all three? Where do I sign up and get the t-shirt?

Unfortunately his life was not all fun and games. Convicted of having sex with a man, he was asked to choose between imprisonment and probation conditional on undergoing 'hormone therapy', which caused him several side effects including breast enlargement. As homosexuals were considered a danger for entrapment by the soviets, he was also stripped of his security clearance and therefore his ability to do the work he loved. He was found dead at the age of 42, apparently poisoned by a cyanide-laced apple.

Some say he was reenacting his favourite fairy tale, Snow White. Others say he was making a reference to the forbidden fruit. Others still, say he wanted an ambiguous way to kill himself so his mother would not believe he did. I just see the parallel with another great man who was led to suicide for the crime of being too far ahead of his own society, Socrates. Yet this being so close to our time, I cannot contain the level of ingratitude he received. The thoughts of his last moments, abandoned and disgraced, pondering his last moments touch some string deep inside me, one not commonly vulnerable in everyday life. This tragic failure of human society is so shameful that i cannot help but be filled with pessimism for this ungrateful race of ours.

This scar and thought and picture will always be with me, to what end I do not know.

The Advertising Immunity is upon us

2008-06-03T01:58:00.000+01:00

So the BBC ran an article recently claiming that web surfers are getting more 'selfish and ruthless'. In disbelief, I read the article in search of the proof of our increased/increasing selfishness. Apparently we click on less ads. And we do what we want faster. I kid you not. Being efficient, immune to distractions (such as ads) as well as being more aware of how the web works and hence wasting less time is proof of our selfishness. Next thing you know, we will be clicking on the bloody banners as an act of charity to prove our selfless nature. Now, I wanted to write this article for a while, but then I saw this post in a blog somewhere which put things into perspective. Aside from the silly title of the BBC article, it seems to be pointing to a reality. Advertising is based on the assumption that we cannot control our focus, that if you put something in front of us users/viewers/passers-by, we will pay attention to it, even if it is not what we were looking for initially. However besides the tools that we can use to block advertising on the web, it seems that ads on social networks are not having much success selling ads either.

So what happens when in this arms race, the human brain responds with improved focus? It seems that whenever a flaw in the mind is found and exploited to death, the next generation grows up with a predisposition to avoid this pitfall. There will always be a vulnerable minority, but the mainstream masses seem to handle themselves against narcotics, gambling and other shortfalls in human perception, if not by pure genetic ability, then by learned cultural means.

So, will the advertisers simply increase their dosage on more potent ingredients such as, say, sex? Oh I wish they do.

...

No, no, you misunderstand me. I mean, I wish they do so we can finally develop an immunity to sex advertising as well. Given a few years of course. Objectification of women and men as a means of increasing sales is simply classless. But if it can be done, someone will do it. It is the way of the market, and the way it should be. The good thing is that our brains and culture will also similarly evolve to distinguish commercial sexuality from the real thing and dismiss the former as mere noise, which it is. In a way, for some people, it already has. I am hoping in more of that for more of us.

What about personalization? That's the next big thing, right? The premise is simple: If they sell us ads that we _want_ to see, we will _want_ to see them. The catch is that in order to judge what we want to see, they have to learn quite a bit about each one of us individually. And that reveals the ugly twin of personalization, which is surveillance. And these twins are joined at the hip. In order to personalize, they have to know who we are and to know that, they have to keep an eye on us. Now, assuming people may not mind some surveillance by a commercial entity, even disregarding that too personal can become creepy, there is another problem with the core argument for personalized advertising. When the advertisers say "Advertisements you want to see are almost like actual content", they are missing the part where someone has paid to have their advertisement displayed. If that was the best thing for me to see, I would have found it myself. I would have paid to have it found for me even. The fact that someone is making an expenditure means that there is some distortion happening. And this distortion will be corrected itself by the same mechanisms that have dealt with the other distortions mentioned previously. If these hypotheses hold, then the advertising game is a dead end.

What next you ask? Well, it just so happens that advertising is the engine of the professional web. Most professional websites are competing for what they call 'eyeballs' which is what they sell to advertisees. Even subscription based publications are abandoning that model and turning to ads. But it is not just content that depends on advertising. Google itself is a company that is built on advertising, which brings in 99% of their revenue. Take the advertising away and the world is back to a very primitive web, one that either has no professional input, or one that depends on subscriptions, or micropayments. Surprisingly, Google seems to be aware. Of course it should be noted that it is not the only option. Organizations such as Wikimedia (parent of Wikipedia and other wonderful ventures) are surviving just fine without advertising so that should be a very viable model as well. For the ones that are not charitable foundations though, there is one very good avenue to replacing them: decentralization.

Here is where I of course offer the usual disclaimer: Take my words with a grain of salt. This is what I want to see, so calling me biased is an understatement. Take my words only as seriously as my arguments permit you to. I welcome you to challenge me in the comments. If you have an element of knowledge that I do not, if you have noticed a flaw in the arguments or if you have attained an insight that eludes me, do not leave me in darkness, I beg of you.

My Content Mess

2008-05-24T18:58:00.000+01:00

So, I was making a diagram of all the services I use online, just to help myself better understand how things are connected. To start with the fact that I need a diagram to make sense should be cause for concern. I am a programmer and that should mean I am quite good at holding things and connections in my head. Still, I needed a diagram, so here it is:Now, while making the diagram, I made a number of observations:

For one thing, I use more services than I thought I was. There are 13 services listed here, and this is not everything.

Also, their interconnections change relatively fast. I repeatedly had to update the connections in the days I was making this, so I don't expect it to stay current for a very long time.

Another observation: I have followers in many of these services. Aside from my blog readers, my Google reader friends can see my shared Items, I also publish my starred items feed, I have a lot of connections in Facebook that can see my activity there, and also some connections in Friendfeed. In fact, since all these services offer open feeds, there are people who could be following me on any of them. There may also be overlap in these sets of people, and I try to maximize it by sharing content between services as much as possible.

Although this all seems a bit complicated, it is better than having one provider do everything for me. Yes, I could do most of these things with a combination of Facebook and Google Reader but it would not be nearly as easy, effective or enjoyable as it is now that I use the best of breed tools in each category. I do have a number of problems though:

1. Too many passwords and accounts. Please someone start using OpenID so I can log in everywhere with single set of credentials. I know it's not perfect, but it's the best we have.

2. Facebook is selfish. What can I say? In the diagram, you can see the little arrows going in, but nothing going out. Why? Because Fb does not let me take my data out of their walled garden. Apparently, what happens in Facebook, stays in Facebook. That's their choice, but that's also the reason I am moving my activity elsewhere. For me, Facebook is turning into a place where stuff gets posted automatically for my old-fashioned friends to see. I avoid posting original content inside Facebook. My cool friends are moving to Friendfeed, that apparently trusts its users more and traps them less.

3. Overlap. What if I write a blog entry and then share and star it on Google Reader, and Facebook, and FriendFeed? Well since I have linked all these together, It may end up 3 times on FriendFeed and 5 times on Facebook. But it is the only way it can get to the widest possible audience, all the people who are following me on all these services. Someone want to do a smart filter that merges all the identical entries? I have heard talk about something like this on Friendfeed, let's see.

4. Not only can I not merge overlapping articles, I also cannot aggregate comments. So what if I post something here, then share it inside Facebook and Friendfeed? There is a possibility that independent comment threads will start and I will have to follow them in three places. There are a number of new ideas coming up in this area, I will look into using them and I hope they generally catch on. (Of course, keep in mind #2)

5. Friendfeed and recently Facebook, are acting like aggregators. This is cool. I like it. But they don't let me integrate with any site I want. They offer me a choice. For the moment, FriendFeed offers everything Fb offers and more, but this may change. My problem is that I have other websites where I am or could be active. Examples are Zotero, Threadless and DeviantArt. The first website that makes it easy for its users to write the adapters themselves, is the winner. Less work for them, more capabilities for us. My money is on Friendfeed, but we'll see.

6. Status messages: there are just too many! I have status on Facebook, Twitter and Google Talk. They generally offer me the same utility. Someone make it easy for me to keep them synchronized? (Of course, keep in mind #2)

7. Bookmark managers. Del.icio.us and Google bookmarks each have their own advantages. Del.icio.us has a great firefox plug in and a nice social aspect, but Google Bookmarks makes my Google searches more relevant. So I want them both to have access to my favourites. I have found a way to transfer data from the one to the other, more or less, but I would like them to be synchronized. I also would like them to be synchronized with foxmarks which I use to keep my browsers at work and at home synchronized. Any ideas?

8. And now for my favourite complaint: Everything is centralized. Waaaaah! Why doesn't anyone build something where I can keep doing my work as I like without being tied to a single entity? Or many single entities for that matter. Perhaps then all the other problems would be moot since I would be able to solve them myself. Oh well, I guess my PhD work is relevant here, in the long-long term.

Overall my little content network is complicated but works mostly Ok. I wonder how many of these will be solved by next year. Maybe there is a ray of light on the conventional horizon called SwitchABit. It's still in development but sounds interesting.

Yet another blog post that I thought would be small but turned out huge. Oh well. I need more experience blogging.

A Spam-less e-mail system

2008-05-20T16:02:00.000+01:00

The last two posts have been quite different to each other. While the first one was an organized, principle-establishing and mature but rather basic article, the second one was more of a braindump, thinking out loud kind of post. Seeing this pattern that is forming, today I would like to introduce yet another category of posts I want to make: Ideas. When I say ideas, I mean technical concepts that can be turned into products or services. You may ask why I do not use these ideas myself, to make these products and profit. Why publish them in the open web to be 'stolen' by anyone? The truth is, and I don't mean to be pretentious, I have far too many of them for the amount of time I can expend on developing them. So while I am working on a big one for my PhD, I might as well hand the other ones out for adoption and hope to get visiting rights once in a while. And of course there is valuable discussion that can be made, and technical points that can be raised and a great educational experience for me all around. I do not pretend that all my ideas are valuable or that they do not need any correction. So without further ado, I present to you, thespam-less e-mail system, let's call it v-mail.

To set the mood, let me talk about the social network messaging systems. Many prefer them over e-mail. Why? because they have verified everyone who can send me a message and therefore have a very high signal to noise ratio. In fact, almost all the messages that reach my inbox are valuable and directed towards me personally. So it is easy to see why people may prefer this method of messaging. What are the problems with this system? Well there are quite a few. For one thing, centralization. Everyone needs to have a facebook orwhatever-book account in order to send me a message. Another is the difficulty in adding people on the go. If I meet someone and tell them to mail me, I do not want to have to go home and add them to my social network first. Also, I may have my mail address published somewhere and people I have never met before may want to e-mail me. This is impossible if I require prior approval for someone to send me e-mail. So how do we exit this seeming catch-22?

My answer is, by applying the best of breed solutions to each problemseparately. What I mean: Let's start with the social networking messaging system. So we have a mail provider, let's call them v-mail.com, that allows you to create an account and add a list of other addresses that you accept messages from. They may give you a standard e-mail address (say, alexandros@v-mail.com) but any e-mail from a source you have not added to your list gets dropped and never reaches you. So this would be an implementation of the basic social networking messaging system. In fact it is one step further, since it allows for multiple servers to communicate, much like the current e-mail system. All you need is an account with any provider of this new mail system. It does not have to be the same as mine, in fact, nobody is stopping you from hosting your own provider and becoming completely self-sufficient.

Next problem: Adding people is hard work, especially if you have to remember to add people later. For this we can use an sms-service that you can use your mobile phone to add people with. So when you add a new person, you may send a simple text message (e.g. add someguy@x-mail.com) to your provider. This would instantly solve that problem. This person can now send a message that will get through. Of course to start your list off you could import contacts from various social networks, address books and e-mail applications.

How about anonymous people that want to send you messages? Well, this has been solved already. Go to any website that wants to get contacted by its readers. They will not publish an e-mail address for fear of spammers. What they do is expose a contact form with a CAPTCHA (these strange numbers you have to type in so that they know you are a human). This contact form could be available at my mail provider's website so that anyone can contact me as long as they provide a valid v-mail ID and prove that they are humans by typing the CAPTCHA. This could also be used to request that someone adds you to their allowed list.

So by merging social networks, e-mail, mobile phones and the web, we can create a composite system that guarantees we only get verified e-mail. Let's go over some of the benefits:

You could use a typical e-mail program and e-mail standards. Additional functionality such as list management would be nice but notnecessary to start with. In fact you could use this new e-mail program to communicate with traditional e-mail addresses, as long as you add the people you correspond with to your allowed list, which could be done automatically when you send an email to someone. In a sense, it's backwards compatible.
The user is back in control. If someone on your list starts spamming, you can simply remove them and never receive another e-mail from them.
Leaking your email is no longer a problem. The information of your e-mail address is useless to a spammer since they cannot send automated anonymous messages anymore.
This solution can work well in a decentralized environment. no need for the massive processing power required by spam filters, anyone can do it on their home computers.
It's good for the Internet since spam is a massive waste of bandwidth.

Ok, have to admit I enjoy creating all these scenarios and exploring the ideas in my head. So here it is, for your viewing pleasure. Let's think about it: Would such a service be useful to you? What downsides can you think of?