Category: interdubs

“It usually takes about 12 to 18 months to build a new center,” she said. “We’re cutting that down to less than a year.”

from a NY Times article about Microsoft and Google and their respective data center operations.

It is interesting that after years in corporate culture people start saying this kind of thing and feel that there is nothing wrong with it.

The article poses the question whether google benefits from looking at each level of the technology stack and inventing where needs are. It does not come to a conclusive answer. I think it is rather obvious: Google was able to reduce its capex spending simply when it felt oppotune to do so. To my knowledge and own experience there hasn’t been any noticeable impact on the Google useability by this reduction in spending. I would guess google simple turned down the pace innovation while the influx of new equipment was slowing.

On a -in comparison- microscopic scale I experience the benefits of looking at the entire technology stack first hand. Part of what runs INTERDUBS is of the shelve, and other parts are enhanced, customized and severely optimized. Some we even actually build ourselves. We constantly look at the running service and identify room for improvement. Be it, in the user experience, or how efficient internals work. Having an understanding of the entire system on all levels lets us identify clearly where enhancements should be made. Each of these steps might only add a couple of percentage points. Having metrics and detailed information about all aspects of the system at all times not only give us visibility into which areas are to be tunesd and enchanced next. It also reveals, that all those little optimizations add up into a configuration ,that is faster by dimensions than the un-altered and generic one would have been.

Having this culture of change and constant optimization is allot of fun. I was plain scared having to do this on live data and a running service. But the goal was that INTERDUBS is available 24/7. And it turns out that technology – used in the right way – is able to do this now. It is literally flying the airplane and rebuilding it in the same time. You start in LA in a 707 and land in New York on a A380.

The main purpose of the API for INTERDUBS was to let my clients use it programmatically. Their system control INTERDUBS in a way that fits best into their workflow. That is what the API was made for, and it works. Every day.

Interestingly there are other, somewhat more surprisingly, benefits to having an Application Programmable Interface to a system too. This week I had a discussion with a client where they outlined specific needs in respect to their data that they have in INTERDUBS. They could have updated things manually, but that would have taken a very long time. By using the API myself it actually took me not that long to implement their needs. I spent only slightly more time on it than the actual discussion took that we had about their needs. This was very nice to see. And actually quiet unexpected. If the next client needs something similar I will be able to solve this in five minutes.

Let’s see if people are interested in getting updates about INTERDUBS via twitter:


http://twitter.com/interdubs


Having seen meme’s come and go together with their tools I am extremely neutral on any new technology that comes up. Does not stop me from seeing if it could be useful.

As usual I try and then see what happens. Easier to ‘do’ and then see instead of having endless evaluations and discussions about something.

Four days of intense work for less than a percent of improvement sounds not like a great use of my time. But I am actually very happy about the outcome: I was able to increase the success rate of clip meta data detection in INTERDUBS by 0.8%. This is great since it went up from 98.8% to 99.6%. Or looking at it from the other end: two third of all flawed detections were and will be corrected with the improved code. One of the benefits of having 100,000s of clips online is to be able run simulations and stats while improving the code. There is a wide variety in what people like to use as their encoding and file format. I’d rather do some more -invsible- work on the backend than to lecture my clients on how exactly they should encode their files. There are recommendations. Sure. But why fail if you don’t have to?

Even though this application of Grubers Broken Windows is seemingly invisible, in the end it certainly is not: A well running system just needs less support per client. Actually so far I was able to decrease the total time spent on support. Despite the fact that the client based tripled last year.

The code management system I wrote for INTERDUBS happens to also count the number of updates that I publish to my clients. It just hit 1,000. Of those about 10-20% were cosmetic updates. Like typos or smaller changes in the html to make things more readable etc. Luckily less than 5% were bug fixes. I code in small chunks and those extensively. And maybe it is also a matter of routine. I hope I know what I am doing, and where changes would jeopardize the system. Of course stating this is inviting trouble. A thousand times I changed the running INTERDUBS. While it was in use by clients and admins. And: nobody ever noticed. Flying the airplane and fixing it. I really love this part of the project: Somebody has an idea. I look at it, and can tell them right away how doable it is, or in the best cases the reply email is as terse as “good idea! done”.

The fast majority of all the good things that came in those 1,000 updates were actually customer ideas. The people using the system know best what they need. It is really great being able to listen to them and to implement what they want.