Tuesday, March 31, 2009

Web development is an organic process

I read an article on Joel on Software where Joel advocates hiring developers who can understand pointers. The idea behind that is that using pointer necessitates having a good abstraction ability and therefore it is a good test if you want to hire the best developers around.

In the particular field of web development, I don’t quite agree with this. To be more precise, I do agree, but since I haven’t used pointers in ten years and wonder if I still could, I’ve got to find some reasons to disagree ;).

Web development is more about a volume of knowledge you need to have, rather than an expertise on one special part: you need to be aware of an insane number of technologies, know how to code fluently in many languages, and know how all these technologies work together (or sometimes don’t, by the way). And you realize this when you debug web applications… haven’t you felt like Gregory House sometimes? The bug you’re hunting is the product of four bugs, like Russian puppets, and actually only happens when you don’t watch because when you do watch, you affect the system and another bug hides all the others… Sounds familiar?

It’s an organic process, a bit like medicine: you need to have a vast knowledge base, good technical expertise to write the tools you need, but above all you need an ability to think globally and understand how the whole system works… A bit like a human body: if your feet hurt, there is a chance that the pain causes you to tense your shoulders. Will a shoulder massage ease your pain? Perhaps, but it’s wiser to think globally and remove that pebble from your shoe.

That’s why I’d rather hire the MacGyver type in my team. I need people that can think globally, that can think different (buy Asus, huhu), that are curious and not afraid to try new things, not technical experts who read the Linux kernel at bedtime. I don’t care if you are not an exceptional coder as long as you understand what we do globally and can start working on any part of our web application. More precisely, I don’t care if you are not an exceptional coder, if you know what you have to learn in order to be exceptional at the task I’m asking you.

Strangely enough, the persons that have these qualities often can use pointers. Go figure.

Monday, March 30, 2009

MySQL 5.1 is out !

MySQL recently released a new version. Among the changes, one feature particularly got my attention: data partitioning.

One of the problems with MySQL has always been huge tables. It often creates problems because they are hard to alter, hard to backup, and because they get slower as they grow, even with the use of carefully placed indexes.

In one of my previous jobs, we partially solved the problem, splitting huge tables in MERGE’d tables. We could then backup them one by one etc, worked pretty nice. And it was faster too, since we could choose which particular table to query depending on our needs (the main table was split by month).

Data partitioning actually is the same feature, only more powerful. Imagine you have data that is organized by date. You can specify when creating the table that you want your data split by month. Additionally, you can even specify that each partition is on a different disk to optimize disk I/O. Requests in a particular range will only scan the necessary tables thus reducing the volume of data to read.

Imagine all the possibilities :)

Thursday, March 26, 2009

SQL Optimization 101 – Part 2

A naming convention for database schemas.

I am often surprised to note that many projects and companies don’t have any naming convention for the fields in their database.

You can for example have a schema as follows:
  • Table user
    iduser_nameuser_passaddress_id

  • Table address
    idstreet_namecitypostal_code

The address_id field in user is a foreign key: the primary key of table address.

I hate when schemas are designed like this. Why can’t I use the same field name when requesting over address_id? Moreover, if there is another table with IP addresses for example, it can spawn confusion.

Finally, when I see the schema of the table, I don’t know the datatypes at a glance.

I have used the following naming convention for a long time and it as proven very clear, if a little verbose, maybe.

Table name: PROJECT_TABLE.
For example: FORUM_USER

Field name: TABLE_NAME_TYPE_FIELD_NAME
Where TYPE can be:
  • i for integer
  • d for date types
  • t for timestamp
  • e for enum
  • s for string types
You have to name the foreign keys exactly as they are named in their own table.
We sometimes use trigrams to shorten long names.

The schema presented above would then look like:
  • Table EXMPL_USER
    user_i_iduser_s_nameuser_s_passaddress_i_id

  • Table EXMPL_ADDRESS
    address_i_idaddress_s_streetaddress_s_cityaddress_i_pcode
It is more verbose, but you see at a glance the global types of your schema, and the relationship between tables. And whatever the table you are in, the address id of an address will be address_i_id, making it a no brainer to write requests.

Amaury (who was the first to use this norm in our team) advocates the use of trigrams. I find it sometimes confusing, so I like better long names. The poet in me, probably.

Tuesday, March 24, 2009

Leadership in video games

Some study leadership reading Sun Tzu’s Art of War. Let’s be a bit funnier and study leadership in video games. Mom, read on, it’s serious.

I played a MMORPG a few years ago. You know, World of Warcraft, this kind of games. The game was Dark Age of Camelot. One of the main interest of the game was that beside making raids to kill monsters, you played in one of three realms that waged war against the other two, leading to events in the warzone where armies of up to 200 players would fight for strategic assets. Think of Braveheart, with 600 players in the warzone.

If you want 200 players of a realm to coordinate, you have to have a leader. And I was amazed to encounter in a video game a handful of individuals who were able to lead masses of people with great skill. I knew one particular guy who was so respected that he only had to call for help and hundreds of players interrupted what they where doing and came to him. These leaders where not necessarily the most hardcore players, or the best players: they just had a knack for leadership.

I actually was kinda fascinated, and started analyzing what made them so exceptional. There were actually a few points:

  1. They respected people and obeyed the golden rule.
  2. They where good players with good play skill, who had the respect of even the most hardcore gamers.
  3. They had awesome communication skills and the best communication devices, using for example teamspeak besides the in-game chat.
  4. They had a plan and had done their homework, testing it beforehand.
    "Destroying this tower takes 4mins, we have 2 minutes to evacuate and ambush here. Go."
  5. If the plan encountered a problem, they made "vital" decisions in a split second.
    "Everybody Kamikaze on this gate, it’s our only hope of creating enough confusion to sneak the Relic through."
    Everybody died, except the relic holder. Nobody actually believed we pulled it off.
  6. They said "No" a lot to demands of individual players.
    "No we are not making a stop to this monster, the aim is to kill the Dragon!"
  7. They knew how to delegate, delegating complex tasks to hardcore gamers,staying with the casuals in order to inspire them and teach them.
    "Go and take this tower, you have two hours, your diversion is the key of the plan, do not fail us"

  8. They assessed people very fast (not easy via an in-game chat), knowing instinctively what objectives to give to whom.
  9. They had fun!

I think the key of their ability to raise a realm laid in point 1. The key of the success of the events they lead lay in the other points.

Isn’t it funny how well this translates to leading a software development team? This translation is left as an exercise to the reader ,).

Friday, March 20, 2009

Why Outlook is awful as a collection tool.

As I said in a previous post, at work, I use Outlook as my information gathering process.

Why Outlook?

Because I don’t have the choice. I also chose not to install third party software to complement Outlook. I just wanted it to work out of the box.

As you might have remarked, I do not use some of GTD’s principles: the Context for example. I just don’t need it.

Also, when in the Act part of my mailbox, the mails are sorted by status, not by project.

After a bit of fine tuning, I just figured that I would rather have a few big buckets with stuff in it that I can review at a glance, than a lot of tiny very organized buckets, that I have to review one after another. It just takes me less time, and I do less routing errors.

And this is something you have to address when implementing a GTD system: you have to trust your collection process, but ultimately, even if it is very reliable, you will make some errors when collecting, and put stuff in the wrong buckets. It has to be simple enough to minimize errors when collecting.

My system, however simple, has nonetheless some serious drawbacks due to Outlook limitations.

As you might have remarked, an email represents a line of my to-do list. But if it is a conversation, then ten mails can represent the same action. Or two actions, or more.

It can be partially solved by the threaded view mode. I just don’t like it though. Never have, never will.

Another big problem is that a single mail can contain info about more than one project. Where do I put it in my reference part when I finish the actions? I often try to forward myself the mail to archive it in another folder, but to be honest it’s a pain.

Last but not least, Outlook search is miserable, to say the least, and in huge projects, I sometime find it hard to find a particular reference mail.

Maybe you begin to see where I’m coming to? How could we collect email efficiently in a GTD fashion?

The answer is Gmail. And it’s not a small improvement over Outlook. It’s a quantum leap. And used with firefox and GTDInbox, it's just the simplest and yet most effective solution I know. Stay tuned ,).

Loïc's golden rule of management

I don’t know where I got this motto, whether I read it on a book, or if I got it from one of my mentors. I really can’t remember, but this rule has become the first absolute golden rule in my way of managing people.

“When the team succeeds, it’s the team’s success. When the team fails, you fail.”

Not respecting this rule, in fact not having this rule completely internalized (it has to be a way of life, doesn’t work if you constantly have to remind yourself of it), will make the team loose respect and confidence in you. And if you don’t have this, you can’t go anywhere.

I have seen managers take all glory of a successful project. The result was the team felt expendable. On the next project, they had no interest in working till they dropped: they were not working as a team for the benefit of everybody: they were numbers, assets that could be replaced.

On the other hand I have a little story about myself: I once screwed up big time as a project manager. When facing the client, I was prepared to take the blame, and I deserved it. But my program manager stood up in front of me, took everything on his shoulder and spent two hours getting humiliated by the client. This had several consequences: I was feeling like shit, and this helped a bit to climb up the slope. I also felt I had betrayed my manager, and I tried to produce the best work ever in order not to let him down again. And last but not least, I instantly respected him. He had earned my loyalty, and I would have done anything he requested from that point onward.

And the fact is: this rule is so true!
If the teams succeeds, it’s not because you are a top-notch manager. It’s because the guys were in the trenches, worked overnight to meet the deadlines etc. When the team fails, it is always your fault, because you haven’t detected there is a problem and thus failed to solve it. It’s your job!

Be the oil in the motor: if the motor is good it’s because it has amazing pistons and the best injectors. If the motor breaks, it’s because there is not enough oil in it.

Thursday, March 19, 2009

Organizing your time: How I chose to implement GTD

David Allen strongly suggests that you empty your brain and dump it on a trusted system, which enables to free your mind and enables you to focus on well, getting things done.
By emptying your brain, he actually means doing this both for your personal life, and your professional life.

I just don’t function like this. I needed a way to handle information overload at work, and that’s it. My personal life is not that busy. Well it is, but it’s not like I’m receiving 300 mails a day about my grocery list.
If I forget to buy shampoo, big deal. I don’t really care, and neither does my wife. I guess it’s a personal choice on how to live your life. I have to be so organized at work, that I’d rather just feel like I’m on holidays when I get home. There are of course exceptions, more on that later.

Actually, I don’t like the term GTD. I didn’t need a method to get things done, I’m getting things done all day long, thank you very much. I’d rather call GTD: “Handling information overload” and that’s what I’m keeping from the method: guidelines on how to gather data efficiently and retrieve information very fast when I need it.

I have to use Outlook at work (and it’s a pain, btw), and most of my information comes in one way or the other in my inbox. I used to set up a complicated filter system to sort email and redirect it in the appropriate directories. Didn’t work. I had to scan multiple directories, could not track the status of a particular mail, it was just a mail. I actually created folders based on who sent the mail. I wonder why I did this, and why everybody does this. It’s just plain dumb.

My setting looked a bit like this:



GTD actually made me realize that I had to sort email by project (duh) AND by status, and this was a lot less obvious.

The final implementation looks like this:


I practice the 0 inbox philosophy.
Every mail is either purely informative and goes in the appropriate project, or actionable and goes into the Act section.

  • NextAction is what I have to do today. Typically has no more than ten mails in it.
  • Action are the actions I have to take, but not today. Roughly 20 to 30 mails in it.
  • InProgress basically explains itself, can grow pretty large.
  • Incubate is the special section when I can’t decide what to do. This basically gets cleaned every 3 month ;)
  • Reminders is a shortcut to the mails I use all the time, like procedures and so on.

Please note that I change the normal behavior of Outlook, and display the number of emails in each folder instead of the unread count.

I don’t have a Finished folder. Every finished action goes in the appropriate project, and I spend ten minutes every morning doing a daily review of all these folders. A weekly review, as advocated by GTD isn't enough for me given the fast-paced environment I work in.

Since information comes to me primarily in the form of mails, this works perfect for me. I email myself with the info that I get by other mediums.

This post is getting long, so I’ll explain later why Outlook is not optimal and what would be my perfect setup.

Organizing your time: how I discovered GTD.

As a manager, I very quickly ran into the problem of information overload, which was the primary reason I got interested in David Allen’s GTD.

It’s funny how at school you learn to be a good little soldier, full of math, theories, concepts, but you don’t get any lessons about how to organize yourself.

I hear you thinking “But if you got through an engineering school, you have to have learned to organize yourself”. Sorry Mom (yes I know you read me), no. You can’t compare school to being a program manager who handles 5 projects, 15 team members, 20 clients at the same time. The scale is just different, and the amount of information you have to process is greater by several orders of magnitude. I got to a point when I felt my work was just keeping up to date with the information coming at me: if I started doing actual work, I just lost track of what was going on, and had to stop working to catch up. I felt I was a data miner, a little dwarf with a pickaxe, digging through an information age Moria. I got to the point where I encountered the Balrog, and I had to stand up and fight back.

I was stuck, not knowing how to make my information gathering process more efficient. I didn’t even had the time to stop and think about how to revamp my system. This is when I discovered GTD. Please, do not think “This is another GTD fanatic”. I am not, and in fact I am very critical of the method, and have adapted it a lot to fit my needs. But this is another story.

My first encounter with it was through an audio book, and let me tell you, I enjoyed the ride immensely. David Allen is an amazing speaker, and I actually laughed out loud listening to a seminar about how to organize yourself. If you had told me this, I wouldn’t have believed it. I also bought the book, which, while not as funny, is definitely worth reading. The benefit of this discovery was almost instantaneous. Ideas on how to solve my problems started to come to me in truckloads. I was Jester the White, and I could take the Balrog.

I will continue this series by presenting what I like and what I don’t about this method, and how I adapted it.

Wednesday, March 18, 2009

Jester Management

I finally found a name for this blog.

Why Jester management?
Well Jester has been my online name for quite a long time but it’s not the only reason.
I am a strong advocate of the motto “Work Hard, Play Hard”, specially in a start-up like environment. More on that later.
I also like the metaphor of a project manager having to juggle between clients, priorities, problems etc. so I think it kinda fits :)

Welcome here. I hope you’ll enjoy reading and that we can share our views on management, tech, and maybe get geeky at times ;).

edit: thank you to amaury for this title!

Tuesday, March 17, 2009

Management : delegating

A little preamble to the management posts on this blog.

I am still a young manager: I have managed people for 4 years, and it’s been my only duty for about two years. As my tech posts, they may seem naïve, but this is where I am in my head.

On to delegating :)

Delegating is in my opinion one of the key skills of a manager. Knowing what to delegate to whom, and when, is of utmost importance, and very hard to grasp.

When I began to lead people, I had a strong need to control everything. I felt I needed to know everything my team was doing, to do all the important tasks, to oversee everything that was done.

I was a control freak. Call this step 0 of management learning.

This didn’t last long, because after a few month, I was on the verge of nervous breakdown and realized I HAD to delegate in order to free myself some time.

And at this time I thought delegating was only needed to free myself some time. If I had 48 hours days, I wouldn’t need to delegate. I was still a control freak in my head. Call this step 1.

After a while I realized that delegating served an altogether different purpose.

The first reason to delegate is that as a manager, you are not paid to do the job of a trainee. Whatever work can be done by somebody else on the team should be done by them, else you’re paid doing their job, and you are overpaid for it. This is a very trivial yet valid reason.

A more profound reason, that I realized only recently, is that delegating is a way of empowering people.

By trusting people in doing work, you make them feel like they matter. You make them learn and grow. You make them take charge, and they feel like they are an important part of the team. In a few words, they both learn, which is good, and get motivated.

And this should be your number one priority.

A control freak on the opposite will strip out any sense of belonging to the team. People will do their work, period, and wait for the clock every single day to free them from his grasp. The projects don’t matter: it’s his projects not their. And brilliant people will do low quality work, because they are not working for themselves, but for him.

If you cleverly delegate work, people will work for themselves, because it’s their project, not yours. They’ll stay until it’s done. They’ll produce top quality work because they take pride in it.

Delegating is not a way to free you some time. It’s a way to empower the people. And empowering the people is one of the best way I know to build a competent, self-motivated team.

Some people stay stuck in step 0. Some in step 1, which is a bit better.

They wonder why they have a team of incompetent, non-motivated people. Sadly, the answer may only lay in their management skills.

Friday, March 13, 2009

A random day

Funny, yesterday I wrote that note about how I wanted my team members to learn a scripting language.

Today, one of my guys comes to me because he has been struggling with a SQL request for most of the morning.
After a few minutes of struggling with him, I just got fed up with his huge jointure that what taking ages AND not working for some reason.

I wrote 30 lines of perl to compare the results of two simpler requests, found the missing rows we were looking for desperately, needed two minutes to understand why we couldn't find them with our SQL and fixed it.

It turns out that NULL fields don't equal other NULL fields and neither of us were aware of that.
NULL is never equal to anything, not even NULL. Well, it makes sense actually, but we had not thought of that.

Learn a scripting language.

Thursday, March 12, 2009

The things you should now how to do – Part 1

I was a bit surprised when working for a company with a strong Java culture that many juniors, although awesome in Java, didn’t have a global picture of the world they worked in.
They sometimes lacked the most basic knowledge, and to me this is clearly not acceptable.

Below are the points I consider to be known when dealing with any engineer developing web application. I don’t necessarily request proficiency, but you have to know this exists, or know how to learn to do what I ask for.

  1. Write a command that traverses a whole Linux file system looking for a file, and send a mail telling where it is.

  2. You need to be able using the command-line. Period. Passing this simple test makes you able enough. Further knowledge you’ll be able to pick-up easily.

  3. Write an HTTP request using telnet.

  4. Might look stupid and odd, but it actually teaches me two things. You have basic network knowledge, and you understand how HTTP works. Some people seem to be dumb-stricken when I ask this.
    Additionally, when you don’t have fancy tools like Firebug or Wireshark, this is going to be your only way to debug: you will always have telnet.

  5. Be proficient in one scripting language.

  6. Two reasons for this. First of all, I think that learning an uncomplicated scripting language enables you to discover lots of things, from network programming to database programming, without the language itself being an obstacle.
    Second, on some servers all you’ll have will be either the shell or a scripting language. I would personally rather use Perl to do mass regex operation on files than a shell script, I just find it easier and more powerful.
    Third, learning any other scripting language should be painless once you are fluent in one.

    Which scripting language should you learn? Well, I would tend to say Perl.

    I actually love the language for a lot of reasons but the most important to me is it’s regex engine. You’ll learn regexes as an added benefit of learning perl. You’ll have to, since it’s so integrated with the language. And having a quite good understanding of regexes will definitely save your ass sometime soon.

  7. Using tcpdump

  8. Hey, we are working on a connected world. You have to be able to tell me what goes on on the wires. I can’t even tell you how many people working in our industry I met that didn’t even know what tcpdump was.

  9. Know what a LEFT JOIN is

  10. Even a marketing guy knows how to do simple SELECTs these days. Please demonstrate that you have more knowledge than a marketing guy when it comes to SQL.

MySQL optimization 101, part 1

One of MySQL’s key performance factor is disk I/O, and this is particularly felt on big tables where there is lots of data on disk to be scanned. You have to be careful when creating your tables. If you don’t pay attention at this step, you’ll end up having rows bigger than necessary.

You need to make your schema as compact as possible from the start. This is the first absolute rule. It’s like cleaning your teeth. You can’t have white teeth if you don’t clean them. Well you can, but it’s going to cost much more. This said, if you are a smoker, it doesn’t guarantee that you’ll have white teeth.

If you double your row size just because you use default types without paying attention, you won’t notice it for ten rows. However, with a 50 million rows table, that wheighs 3 Go, the overhead will start to be felt.

If your MySQL table is twice bigger on disk than needed, you can assume that a query will be twice as long (well, maybe). And additionnaly your sysadmin will hate you for making him buy new disks all the time.

A little anecdote to prove my point, that I read on MySQL’s website.

A company used VARCHAR(1) fields to store flags, like “Y” or “N” (yes or no). A VARCHAR field uses 1 byte to store the length of the field (if the size is below 255, two bytes otherwise).
A CHAR(1) field uses only one byte, no need to store it’s length since it’s fixed.

If you follow me, each flag used two bytes, instead of one. Taking into account that they had several of these flags in each row, and that they had gigantic tables, this small overhead started to really matter, and they converted all of their tables to use CHAR in order to optimize both size and response time.

Please, keep this in mind, but don’t go for extremes all the time. We once designed a table without foreseeing how much it would grow. At one point, a year later, our platform crashed. The reason was that we had reached a field’s max capacity (65k). Your schema should be able to go on for ten years being unattended.

Next post will continue on schema optimization.

A little foreword on the tech postings on this blog.

A little foreword on the tech postings on this blog.

I started working in a start-up environment in 2002. With a degree in Thermoenergetics, I had basically no knowledge of the internet world, but since I didn’t want to sell fridges all my life, I was motivated to learn, and fast.

I was the first employee of a company that grew from 0 to 3M$ revenue in the course of four years. We developed solutions to publish mobile websites.

I learnt a great deal there, and particularly had a very good view of the trial and error process of building something from the grounds up, and of how a platform scales (or sometimes… doesn’t) when you start with zero page views and reach 150k page views a day.

My company was then bought by my present company, which was actually younger than ours, but backed up by a bigger company.

Technically speaking, it was a bit of a setback for us. The new platform we had to work on was only 1 year old, and they clearly lacked the experience we had in terms of how a platform behaves when traffic increases tenfold. There was clearly a big philosophy of developing products fast (which is good) but with a lack of insight of how to do this cleanly enough to minimize work in the future.

Another point was that the technical team was younger and although very skilled in their area of expertise, lacked the global experience we had. They had not make the mistakes we made before yet (or did make them, but didn’t realize it), but as far as I could see, where happily going to make them.

This is the reason why I started writing articles for my team, trying to pass on some of the things that appeared to be important to me. Some of these articles may seem naïve to expert developers, but prove to be interesting for most juniors.

I wish you a happy reading, and please do not hesitate to comment. As every developer, and geek, my main passion is information, and I’m always willing to learn.

Wednesday, March 11, 2009

Why?

I wanted to write a blog for a long time, but somehow never had the time.

I am now about to get married, and will move to Seattle to follow my wife in a month.

I guess it's the perfect time to start this blog and write down both our progress in this journey, and all the articles that are lurking somewhere in my head.

Search Results