John Linton
.......in this industry it's difficult to keep tabs on it.
I was saddened to read this earlier this morning:
http://nighthacks.com/roller/jag/
as Exetel's whole business operations are based on MySQL and since the Oracle takeover there has been a lot of doubt as to whether, possibly the most important open source software on the planet, was going to be chewed up and spat out as some over priced 'Oracle Light'. James Gosling, the developer of Java, leaving Sun (recently bought out by Oracle) is one more sign (he wasn't the first) that the MySQL development team is being 'let go' and that is not a good sign for the future of MySQL. Maybe that will take years to happen, if it happens at all, but I think that a company that spends $US1 billion on buying up a company (MySQL) is going to need to justify why it spent the money and why it can justify spending more money continuing to develop it. Perhaps the justification that it is an anti-Microsoft measure is enough but you would have to doubt that.
It may seem to be a insignificant issue for someone who has a whole range of more directly threatening issues to worry about but, one thing being in business teaches you is that it's often problems from 'left field' that cause you the most grief as you are totally unprepared for their impact because, like right now, you become wholly immersed in the key day to day problems you have to deal with.
All yesterday we had to have the key technical resources of our company trying to resolve a 'bug' that had 'suddenly developed' in our VoIP services which manifested itself in some calls from some customers (and our own VoIP PBX) being 'one way audio'. This was sporadic (perhaps one call in four either inbound or outbound) in our own operation and some different parameters in our customer's usage. As we had made a very minor hardware change the previous Monday we assumed that we had somehow made an error in that minor upgrade so the first thing we did was to 'roll back' the changes we made - it didn't help. By 4 pm we had found what we thought was the error and changed a key component which appeared to resolve all the customer problems but not our own so we concluded, logically but wrongly as it turned out, that, some how, we had induced an intermittent error condition in to our Asterisk based VoIP PBX.
The hours ticked by with several engineers getting increasingly tired and increasingly frustrated as we had made no changes to our own internal VoIP hardware and software but were 'convinced' that the problems must be related to the hardware changes we had made in some way. Like the proverbial 'bleeding obvious' it was yet another example of Sod's Law which was finally traced down using our Asterisk engineering/programming resources in Sri Lanka, Chile and Australia as a defect in one of our voice carrier's services that didn't only affect us but affected all traffic that followed a particular transit path either to or from our and our customers end points.
The problem was finally resolved a little after 1 am this morning and the carrier advised of what we believe their problem was. While I received confirmation that the carrier has 'accepted the ticket' and is looking in to the reported problem we can't be certain that was the root cause of the troubles we and some of our customers were experiencing but switching that traffic to another carrier eliminated the issue and was thoroughly tested. So that's a 'left field' type of problem that after 3 years of fault free operation comes 'out of the blue, potentially was affecting almost 20,000 customers and prevents Exetel reliably receiving and making telephone calls for almost 24 hours.
While I was greatly relieved to receive the news earlier this morning and feel a lot better than I did yesterday it was a salutary reminder that no matter how well something has gone (in this case for over three years) anything can suddenly happen that changes your operating scenario so completely you are completely shocked....and, as in this case, all the redundancy of alternate carriers, duplicated hardware and standby hot spares made not the slightest difference.
So 'death and destruction' in the commercial world can come 'out of the blue' from events as varied as a huge US company taking over your source of all your automation and possibly changing your cost structure 'overnight' to a key service that has performed faultlessly for as long as you have had it in operation suddenly failing for no apparent reason with no-one able to diagnose what has gone wrong - dealing with sudden changes in competitive marketplaces or supplier policies is child's play compared to being confronted by events that appear randomly and over which you can exercise no control and have no ability to provide input/assistance to their resolution.
Is today going to be a better day?
10.........