When a Tandem, or HP NSK program crashes, it creates a memory dump file called a SAVEABEND file. When my brain crashes, it adds an entry to this blog.
Sunday, December 10, 2006
The Parable of the two Programmers
Once upon a time, unbeknownst to each other, the "Automated Accounting Applications Association" and the "Consolidated Computerized Capital Corporation" decided that they needed the identical program to perform a certain service.
Automated hired a programmer-analyst, Alan, to solve their problem.
Meanwhile, Consolidated decided to ask a newly hired entry-level programmer, Charles, to tackle the job, to see if he was as good as he pretended.
Alan, having had experience in difficult programming projects, decided to use the PQR structured design methodology. With this in mind he asked his department manager to assign another three programmers as a programming team. Then the team went to work, churning out preliminary reports and problem analyses.
Back at Consolidated, Charles spent some time thinking about the problem. His fellow employees noticed that Charles often sat with his feet on the desk, drinking coffee. He was occasionally seen at his computer terminal, but his office mate could tell from the rhythmic striking of keys that he was actually playing Space Invaders.
By now, the team at Automated was starting to write code. The programmers were spending about half their time writing and compiling code, and the rest of their time in conference, discussing the interfaces between the various modules.
His office mate noticed that Charles had finally given up on Space Invaders. Instead he now divided his time between drinking coffee with his feet on the table, and scribbling on little scraps of paper. His scribbling didn't seem to be Tic Tac Toe, but it didn't exactly make much sense, either.
Two months have gone by. The team at Automated finally releases an implementation timetable. In another two months they will have a test version of the program. Then a two month period of testing and enhancing should yield a completed version.
The manager of Charles has by now tired of seeing him goof off. He decides to confront him. But as he walks into Charles's office, he is surprised to see Charles busy entering code at his terminal. He decides to postpone the confron- tation, so makes some small talk then leaves. However, he begins to keep a closer watch on Charles, so that when the opportunity presents itself he can confront him. Not looking forward to an unpleasant conversation, he is pleased to notice that Charles seems to be busy most of the time. He has even been see to delay his lunch, and to stay after work two or three days a week.
At the end of three months, Charles announces he has completed the project. He submits a 500 line program. The program appears to be clearly written, and when tested it does everything required in the specifications. In fact it even has a few additional convenience features which might significantly improve the usability of the program. The program is put into test, and, except for one quickly corrected oversight, performs well.
The team at Automated has by now completed two of the four major modules required for their program. These modules are now undergoing testing while the other modules are completed.
After another three weeks, Alan announces that the preliminary version is ready one week ahead of schedule. He supplies a list of the deficiencies that he expects to correct. The program is placed under test. The users find a number of bugs and deficiencies, other than those listed. As Alan explains, this is no surprise. After all this is a preliminary version in which bugs were expected.
After about two more months, the team has completed its production version of the program. It consists of about 2,500 lines of code. When tested it seems to satisfy most of the original specifications. It has omitted one or two features, and is very fussy about the format of its input data. However the company decides to install the program. They can always train their data-entry staff to enter data in the strict format required. The program is handed over to some maintenance programmers to eventually incorporate the missing features.
Sequel:
At first Charles's supervisor was impressed. But as he read through the source code, he realized that the project was really much simpler than he had originally though. It now seemed apparent that this was not much of a challenge even for a beginning programmer.
Charles did produce about 5 lines of code per day. This is perhaps a little above average. However, considering the simplicity of the program, it was nothing exceptional. Also his supervisor remembered his two months of goofing off.
At his next salary review Charles was given a raise which was about half the inflation over the period. He was not given a promotion. After about a year he became discouraged and left Consolidated.
At Automated, Alan was complimented for completing his project on schedule. His supervisor looked over the program. With a few minutes of thumbing through he saw that the company standards about structured programming were being observed. He quickly gave up attempting to read the program however; it seemed quite incomprehensible. He realized by now that the project was really much more complex than he had originally assumed, and he congratulated Alan again on his achievement.
The team had produced over 3 lines of code per programmer per day. This was about average, but, considering the complexity of the problem, could be considered to be exceptional. Alan was given a hefty pay raise, and promoted to Systems Analyst as a reward for his achievement.
Wednesday, November 29, 2006
Driving in Jeddah
Some of the common things you will always see are:
* Making a dozen lanes in a three lane street at traffic lights.
* Not seeing lanes at all while driving.
* Some very very wide stretches of roads without lanes at all.
* Stopping at the right lane on the road, and then making a U turn at the lights. crossing two or more lanes who want to go straight, and one or two who want to make a left.
* Who stops at roundabouts? Who has the right of way? I still did not figure this out, but it seems that whoever can stop, should. In most cases it is those at roundabouts who stop.
And all of that behavior is perfectly normal. very few honks are ever heard while doing any of this.
It is in other words, "organized chaos".
But one cool thing is, you can do all of that too, and get away with it! Just like everybody else on the road, others will generally take care for your wrong moves. You might hear some horns, but you will probably give some to others too. Just relax, and blend in.
Two major things that are proven by looking at the streets here are:
1. Allah does exist.
2. HE is very forgiving and merciful.
Otherwise there will probably be no man alive in here. I still recall that guy crossing a 3 lane congested highway, without even looking, hearing no breaks or horns at all, and him reaching the other side safely like nothing happened. By all laws of probability and physics, he should be dead meat. But Allah Raheem.
Happy driving if you do it in KSA :-)
Wednesday, November 08, 2006
Moving
Yesterday we went to get our one-way ticket. Then I went to finalize selling my car, and realized I still have another 2 years instead of just one of loan payments. We went back home, and my brother came to get some stuff. That stuff was a spare desk, and my PC desk. Both desks occupied most space in my "operations" room. I had to pack my PC. And then my operations room was almost empty, except for some boxes.
Now I feel it. We are moving, we are leaving all of this behind, and starting a fresh start.
I am not feeling sad or afraid for the move at all. I am excited, but I will miss all things, and most importantly all people here.
The different thing in this move from all the others I made previously is that this time, I will take the most precious person with me. That makes it easier :-)
Thursday, November 02, 2006
7ajafaan
My family, have passion for laughter. One of their talents, and joys, is remembering sentences, sayings, and miss-pronounced words and creating our own proverbs and sayings.
Today's word is "7ajafaan". This is pure family heritage. It's not even a word! Yet we say it so often, that it is part of our daily vocabulary. We share it with our friends and relatives. It's also a sort of passcode that when any one of my family hear it, they will know you are part of the family. Sort of get's you in the circle.
And the story for this word happened way before I was born. During a gathering that included my uncle, many others and a very old man having a heart problem. The old man did not look well that night and was asked what was wrong. He said "Madri, yumkin 3indi 7ajafaan fi al qalb" 7ajafaan instead of khafaqan (pulse). One of my uncles heard it, and loved that word. And so the word was born.
And from that day for anything that is not okay, and the reason is unknown, we say "fee 7ajafaan", or "having 7ajafaan".
Examples.
"What's wrong with your stomach? You don't seem well.". to which you answer. "madri, fee 7ajafaan".
"What's with this program? a bug?" "Yeah, this program fee 7ajafaan".
Sunday, October 29, 2006
The Mont Blanc Pen
See the many fine rings it has? The black and silver? The two black rings?
I take simplicity to the extremes I guess. I like simple, clean, straight designs. Nothing fancy at all. "Less is More" is one of my favourite proverbs. And this applies to design too.
So, as much as I appreciate the gift, I just had to exchange it. I want to have a gift that I like. It still seems unethical to me, but I proceeded. I went with my wife to Mont Blanc in Seef Mall with the Pen. I didn't have a receipt. So I did not know if they would accept an exchange. I just told them I got it as a gift two days ago, and I want to exchange it. They gladly exchanged it for another.
So now, I have a gift from my dear friends and colleagues, and it is something I like. A clean, straight, and simple, yet sophisticated Mont Blanc.
I also like the fact that this pen has a cap that you need to screw to open/lock it. It sorts of lets you think before you use the pen. Not just any cap you get from a Bic. You make sort of an effort to use it. After all you are using a Mont Blanc :-)
What do you think?
Saturday, October 28, 2006
Prada
I was reluctant at first. I knew the movie was about fashion, main character is a she, so I thought it's a chick flick. It's not like that, although it does show a lot of chicks and a lot of fashion. I did like the movie and the points it was trying to make.
The movie is about Andrea, an educated smart girl, who is not into fashion, not that pretty, or at least thinks and acts as if she is not. She has no other opportunity but to work as an assistant to the editor of Runway magazine, Miranda. Miranda is so bossy, so over-the-top and a goddess of fashion. I will not spoil any more of the movie, but it is very good.
For me, the main point in the movie is that it doesn't matter if you have something that a million other persons dream to have, what matters is do you have what you want? And even if you think you don't have a choice but to do what you have to do, you always do have a choice, even if it is simply walking away.
Tuesday, October 17, 2006
MySQL and index use
I have a transaction log file that contains a million plus records. Two essential fields I almost always use are TranDate and TranTime, CHAR(6) and CHAR(8) respectively. They contain the date and time in YYMMDD and HHMMSSss.
After the log table contained a million records, my queries were starting to take longer and longer, even though I had an index on TranDate and TranTime columns called DateTime. I needed to search for transactions between Date Time (YYMMDDHHMMSSss). The query had the following syntax:
SELECT *
FROM log
WHERE Concat(TranDate, TranTime) BETWEEN '061001' AND '061010'
I initially thought that the DateTime index will be used in the above query, but an EXPLAIN command on MySQL showed a full table scan. My index was not used!
I went digging in, and surely enough, MySQL will not be able to use the DateTime index, because the TranDate and TranTime are NOT used directly in the WHERE clause. The query basically tells MySQL to take the TranDate and TranTime columns in each row, combine them, and only return those rows between the given values. MySQL is no magician to know that the combined values are indexed already.
So, the solution was very simple. Create another column, TranDateTime that is the concatenation of both fields, and then modify the DateTime index to index that single column. The query will also change to replace the Concat function with the TranDateTime column.
Another EXPLAIN command, and the DateTime index is used, cutting the execution time to less than a second instead of few minutes.
The rules of index use in MySQL (and probably other databases) are very simple:
1) The columns that in the WHERE clause will be subject to index use if they are part of and index.
2) An index on (col1, col2, col3) can be used to search for any of the following combinations of columns: (col1), (col1,col2), (col1, col2, col3). The same index cannot be used to search for col2 only. Just think of the index as a sorted array of string, and you should figure out why.
3) The index used will be the one that returns the least number of rows.
Moral of the story?
1) Ask the database engine what is it doing. MySQL may not be the most sophisticated database, especially with multiple tables, but it does the job. Imagine what a database like Oracle, Sybase or one of the big guys will do.
All you have to do is ask.
2) Understand why is the engine doing what it is doing. Why is that seemingly suitable index not used? Is the right column present in a query WHERE clause? is that column used in a function that makes the index unusable? A LIKE '%value%' clause will not use any index.
3) Try to create a better index suitable for that query, or try to fine tune the query so the best index can be used.
Re profile, and continue until you get fed up.
Remember that you cannot solve your bad query problems by getting better hardware!
Monday, October 16, 2006
MySQL Crosstabs
Term Resp Count
ATM01 APP 100
ATM01 DEC 10
ATM01 REV 5
ATM02 APP 250
ATM02 DEC 23
Listing 1
something like the following is much easier to look at, and graph:
Term APP DEC REV
ATM01 100 10 5
ATM02 250 23 null
Listing 2
There is no built-in crosstab or pivot construct in MySQL, but this is much easier than it seems. After you create it once, it will become second nature. I am using it very frequently to create all sorts of statistics and graphs.
So, let us assume you have the above data in a table like Listing 1. You need to transform it, or create another query that will generate to you the result in Listing 2.
Here is the Query:
SELECT Term,
SUM( IF(Resp = 'APP', 1, null) ) as APP,
SUM( IF(Resp = 'DEC', 1, null) ) as DEC,
SUM( IF(Resp = 'REV', 1, null) ) as REV
FROM mytable
GROUP BY Term
The trick is using a SUM of 1 or null. 1 will be added when the condition in the IF applies, and null otherwise. The beauty of this, is that you can have any arbitrary complex expression in the IF statement to group records that must be added in the APP column. For example let us say you need to have only two columns for APP and OTHERS. Your query will be:
SELECT Term,
SUM( IF(Resp = 'APP', 1, null) ) as APP,
SUM( IF(Resp <> 'DEC', 1, null) ) as OTHERS
FROM mytable
This gives you much more flexibility in constructing your crosstabs. It May get a bit complicated if you need many columns, or if you need a single column per value. If you have this, then probably the best thing is use the right tool for this. Use Microsoft Excel. Or use any one of several techniques that build the SQL for you. Just do a Google search on MySQL crosstab, and you will find some good ones. One good article on MySQL site is here.
Thursday, October 05, 2006
Genetic Algorithms with Ruby
One of the nice sites with demos is here.
So I thought this will be a very cool exercise to practice my newly acquired Ruby skills. It took me a few hours to come up with a simple "framework" to represent a general GA.
I will keep updating my GA framework and add more tools and functionality to it. Ultimately I want to be able to modify my sub Chromosome class only, and just add a fitness method, then fire my simulation and let the search begin.
Basically the framework consists of a Chromosome class, and a Population Class.
The Chromosome Class
This class represents a Base Chromosome. It is basically a sequence of genes that have a fitness, and possibly represent a certain value. The Base class knows how to marry, or crossover. The Base class crossover is a single point, and no mutation is done. A subclass will be created that can have two crossover points. The framework is still in very early stages, and will be re-factored. But basically, that is all there is to it.
The Population Class
This class represents a Population of Chromosomes. It holds the entire population and is responsible for Selection and Generating offsprings. The Base class does a simple Roulette Wheel Selection based on a fitness value. To make the initial code simple, the fitness value must be higher for fitter solutions. So, a chromosome of fitness = 10 will be 10 times likely to be selected that a chromosome with fitness = 1.
Genes?
I did not create any classes for genes. I wanted to make things simple, so the sequence itself will represent any kind of genes. It is tested as a Strings in this initial version. The main thing is finding a way to represent the fitness of the chromosome. I will see if representing binary numbers, integers as sequences, or real numbers will require any changes in the sequence attribute or require the creation of a Gene class. But since Ruby is so dynamic, I doubt this will be needed.
The Code
Anxious to see the code and try it out? Drop my a line, and I will send it to you or publish the current version somewhere. It is still in very early stages, but works for simple problems.
Saturday, September 09, 2006
Java + Ruby = JRuby
Well, I did several small applications in Java, and it was my language of choice for some time now. Java is Free, and lots of IDE's are free, like eclipse, and netbeans. Java has millions of free libraries for almost anything. Java can be used on the server with jsp, j2ee and on the client with Swing. So it's an all-in-one language.
So, having a Ruby implementation in Java, you can leverage your Java code, and use the elegance and beauty, and fun of coding in Ruby. You can extend your Java classes using Ruby, and you can automate some of the high level work for your applications using Ruby.
Another GREAT news is that JRuby is now endorsed, and officially supported by Sun. The core JRuby developers are paid by Sun. Way to go Sun!
There are some nice tutorials in the JRuby site. I'll try that out myself and see how far I can go with JRuby.
Thursday, September 07, 2006
Ruby Oneliners
The one liners can be found here.
If you are running on Windows, then substitute type for cat.
As an example, Here is a handy oneliner
# delete leading whitespace (spaces/tabs/etc) from beginning of each line
$ cat| ruby -pe 'gsub(/^\s+/, "")'
Note that the command
ruby -pewill execute the Ruby commands that follow it on each input line, and sends the output to STDOUT. To capture the output you can use the Linux/Unix or DOS > char.
Another useful oneliner is the following:
# print line number 52
$ cat| ruby -pe 'next unless $. == 52'
The above oneliners depend on the $ Ruby global variables. Here are some of the most commonly used:
$! error message
$@ position of an error occurrence
$_ latest read string by `gets'
$. latest read number of line by interpreter
$& latest matched string by the regexep.
$1, $2... latest matched string by nth parentheses of regexp.
$~ data for latest match for regexp
$= whether or not case-sensitive in string matching
$/ input record separator
$\ output record separator
$0 the name of the ruby script file
$* command line arguments for the ruby script
$$ PID for ruby interpreter
$? status of the latest executed child process
Have fun.
Wednesday, September 06, 2006
The Story of the Mexican Fisherman
An American investment banker was at the pier of a small coastal Mexican village when a small boat with just one fisherman docked. Inside the small boat were several large yellowfin tuna. The American complimented the Mexican on the quality of his fish and asked how long it took to catch them.
The Mexican replied, "only a little while."
The American then asked why didn't he stay out longer and catch more fish?
The Mexican said he had enough to support his family's immediate needs.
The American then asked, "but what do you do with the rest of your time?"
The Mexican fisherman said, "I sleep late, fish a little, play with my children, take siestas with my wife, Maria, stroll into the village each evening where I sip wine, and play guitar with my amigos. I have a full and busy life."
The American scoffed, "I am a Harvard MBA and could help you. You should spend more time fishing and with the proceeds, buy a bigger boat. With the proceeds from the bigger boat, you could buy several boats, eventually you would have a fleet of fishing boats. Instead of selling your catch to a middleman you would sell directly to the processor, eventually opening your own cannery. You would control the product, processing, and distribution. You would need to leave this small coastal fishing village and move to Mexico City, then LA and eventually New York City, where you will run your expanding enterprise."
The Mexican fisherman asked, "But, how long will this all take?"
To which the American replied, "15 - 20 years."
"But what then?" Asked the Mexican.
The American laughed and said, "That's the best part. When the time is right you would announce an IPO and sell your company stock to the public and become very rich, you would make millions!"
"Millions - then what?"
The American said, "Then you would retire. Move to a small coastal fishing village where you would sleep late, fish a little, play with your kids, take siestas with your wife, stroll to the village in the evenings where you could sip wine and play your guitar with your amigos."
Monday, September 04, 2006
Click, Fast farward, rewind
I was talking about the same subject after lunch with my family before going for the move. We talked about how some people struggle so hard to "get a better life for my kids", but forget that they are losing their own life. Time can never be brought back... One should enjoy the journey, not just the goal.
In a job like IT, we tend to work so hard, and "fast forward" through life, missing many of it's moments that we can never get back.
Get your priorities right. Mine is my family.
Wednesday, August 30, 2006
My Database Design Rules
Database design:
It is easier to think in terms of Entity Relationships. Entities then become tables. Entity Attributes then become columns. ER can be mapped to classes very easily.
Uses consistent names for fields. If you can, stick with full words or abbreviate very long, but well known columns and tables.
Table names are plurals. Classes, Messages...
Use underscores to separate words.
When naming columns or tales, start with the more general entity, then narrow down. This makes searches and lists much easier. Case Attachments, Case_Notes, User_Roles, User_Groups and so on.
There is no need to prefix tables with tbl or the database name. Have one database or catalog for each application. This is just my personal preference. Since most GUI tools group tables and views, why replicate this? What good would it make to prefix eveything with tbl or v?
Normalize, normalize and normalize even more. However you may need to copy totals, and some user entered or varying data in some tables and denormalize or duplicate. An example is for some critical applications, e.g. financial apps, the lookup data may change, and it may be needed later. E.g. lookup of a location of a terminal may be copied to a transaction record, even though the terminal ID, which contains the location is present in the transaction record. In this case copy the lookup data to the table.
This is related to above point. To make your queries simpler, denormalize some tables by creating some detail views that lookup the master data from the master tables. You can then simplify alot of your queries and reports by not linking the master tables every time you need master data in detail report.
For enterprise applications, it is best to have two columns in every table as: USER_ID and SYSTEM_DATE. Name these two columns anything you want, but use the same name in all tables. I used CREATED_BY and CREATE_DATE sometime. These two columns must be set by triggers and must get the values from the system. If the tables are critical, have the triggers update audit tables (TABLE_NAME_AUDIT) with before and after data for the main table. The point is, these USER_ID and SYSTEM_DATE columns, and the audit tables must be maintained at the database level. This way the application never needs to care about updating them, and the database will contain correct data about each record, no matter what the source of the change is. Come on, even MySQL has triggers now, so there is no excuse.
I had a very strong argument with my Masters Degree DB teacher about this one, and I finally convinced him: If you have many-to-many relationship, you did something wrong. The relationship should be broken down to another entity (table) and then you will see that this link table probably has some more attributes to it. SO... it is not a many-to-many relationship after all. This many-to-many relationship is a new entity.
About Primary Keys:
You must have one column called ID for each table. This must be the primary key for the table, and if you can, have it auto-increment. Even if it does not seem needed at first, it will be later. What if you need to delete or update that single record from the table? you have it's ID.
Use this ID, and only this ID, as link in foreign tables, and prefix the foreign key column with the foreign table name, and an underscore. Always use database constraints for foreign keys. For example, you have one to many relationship between cases and attachments. In Cases, there is ID column. In Attachments, you have ID (Prikey) and Case_ID. The name of the foreign key column must be the singular form of the foreign table.
Never EVER EVER use user entered data as a primary key. If user entered data must be unique, then use a unique key constraint. This way all your links will not be affected if a user enters the wrong "key" and everything must be in sync again. It may seem overkill, but trust me on this.
Sunday, August 27, 2006
Ruby Scripts - Simple SQL dumps
Here is the sample Ruby code:
require 'DBI'
# make an ODBC connection
conn = DBI.connect('DBI:ODBC:','[USERNAME-HERE]','[PASSWORD-HERE]')
sth = conn.execute("
[SELECT STATEMENT HERE]
")
# If you need column names, then you can use the below statement to get them
# cols=sth.column_names
sth.each do |row|
puts = row.join("\t")
end
sth.finish
Nothing except the Ruby Installer was needed to run this.
Thursday, August 24, 2006
A bit of history
Back in the good old days of the Commodore 64, I typed pages and pages (probably 30+) of numbers from my Programming The Commodore 64 book just to create an assembler. Then the fun began with Assembly Language. Things were really fast in assembly when you have an 8-bit 1-Mhz processor with three registers, and a full 64K (almost) of RAM! I developed my first compression routine when I was probably 16. Later I knew it was called RLE (Run Length Encoding). I had my first "games" with moving sprites and changing colors.
So why did I do that? No internet for me at that time, no help from anyone. Just my book, my C-64 and lots of reset button presses (there is no debugging in assembly). Why? Because I simply loved doing that. Still do. Programming for Programming's sake. Born to Code.
Wednesday, August 23, 2006
Take One
Stay tuned.