Wednesday, July 29, 2009

Security (Bug Fixes) is now a Major Industry

Have you noticed that Microsoft and other software vendors think bugs are a natural common place?

What I mean by that is that they EXPECT USERS to EXPECT BUGS in their products. I've been a software producer for nearly 30 years now, selling and supporting customers over the years and I don't expect BUGS to be a natural part of the process.

That said, yet, it is well understood bugs do happen and no software is 100% perfect, regardless on how great your are. While some products are managed by highly trained engineers (as myself) and QA is very important, there are times where a bug is discovered by others. "Shit happens!"

The problem is that BUGS have become so pervasive in the industry and in mindset of this generation of software developers, there is a whole industry developed around it, especially in the security area or bugs that cause security problems.

Its become so bad, companies like Microsoft has a regular scheduled BUG FIX update, like every month - and they have trained the mindset of the industry to accept this modus operandi of software product development,

Another problem is that vendors, especially Microsoft use the "BUGS FIX OF THE MONTH" schedule to ship and include new technology they wish to enforce on the user base. And these new technologies are not always top quality and will be part of future Bugs Fixes updates!

Its a tough situation because as a software product producer, we can understand the situation. But we are not one to play this poor QA game to begin with and that is my beef.

Why did it happen?

Well, its hard to point fingers but I have my take on it:
  • Open source market,
  • More complexity,
  • Higher Dependency on 3rd party software,
  • Smarter IDEs,
  • Scattered support process,
  • Developer are less trained as Engineers - problem solvers,
  • And too much FREE software!
Overall, when you have less control of your software development, libraries and tools, you are prone to suffer the issues that are within the software you less understand and did not developed.

For example, for the Wildcat! Interactive Net Server product line, there are a few millions lines of code. We developed nearly 99% of it all in house. We hardly ever use 3rd party software, and in the rare case we do use 3rd party technology, we only do so with 100% complete understanding of what it does and what issues it has. Doing the design process, we know what every function call does. There are no surprises or unknowns. Its not a guessing game in how a function, blackbox or component is going to behavior. When you don't fully understand something, the risk is higher for problems.

So much of the software products out there are open source, and worst based on other open source. So if one part is broken, all of those products using the broken open source, is broken as well. A good example of this is WebKit which many browsers use. If webkit is broken, all of these browsers are broken. And this is just one small example. There are thousands of examples out there.

There are good and top notch open source software, so that is not a big part of the issue.

A bigger part is the complexity, less documentation. Since you didn't write it, you will spend a lot of time figuring it out. Good developers will roll up their sleeves and figure things out. Unfortunately most do not because of the complexity and when help is needed, many times help and support is unofficial, based on user-support forums and extremely time consuming. For example of this, most vendors have Newsgroup Support Groups and if you ask the question in a way people can understand, many times you get help. You may have to wait a few days and its a pop luck if you do get the help. That's because the vendors have push the support process to the users - users help users. A good low cost idea to help solve vendor QA problems!

It has become so complex, you need smart Interactive Development environment to assist you in putting together applications. Its like a puzzle. Take many parts, including open source, logically put it all together, press a button and viola - you have a product executable!

For many, that's good enough! they do a little testing, package it up and distribute it. It is far cheaper today to let users find problems and report them to you. After all, its no chip off your shoulder. Users expect it.

Only the good developers will use the feedback and solve the problems. Others will use either ignore it or wait for others to complain. After all, its free! How dare you complain and expect fixes!?

But overall, it happens because many developers have lost control of their design, development, quality testing, documentation and production - the traditional five stages in Software Engineering! The smarter IDEs allow kids 10 - 16 years old to write great software - but they are generally BUGGY AS HELL!

If you are not getting paid, they is less incentive to fix problems - yet, the software is "good enough" to release to the world for others to use and propagate the issue!

The industry has been molded to accept this practice!

Hector Santo, CTO
Santronics Software, Inc


Havard said...

The "good enough" mindset is pervasive, not merely in the free software world, but the commercial and business in-house-only world as well. Software should be developed with the mindset that someone's life depends on it. As a matter of fact, the developer's life, or at least livelihood, depends on it. In my personal projects, developed for my own amusement, bugs are unacceptable. If I'm going to let someone else use my code, I'm at least 99.9% certain there are no obvious bugs. Then again I try to make software lean enough that any bug would be obvious. Subtlety and subterfuge have no place in software development.

As for using components developed by others, it's an excellent way to avoid reinventing the wheel. Then again, look at the car wheel and tire industry. There are plenty of companies that make obscene amounts of money reinventing the wheel.

Hector said...

True regarding "reinventing the wheel." I think the key difference though, there is a higher standard like for a care tire and more importantly, there is product liability - higher QA involved. Reusing 3rd party code is good, but in the past, you generally had solid vendors behind them or at least those getting paid for it.

I think we are starting to see this more with open source, free to use, not free to get support. So maybe in time, the QA will be higher. Just the other day I downloaded a very popular open source MP3 tag reading library. I found a bug that would intermittently FAULT the application. Found a solution but it didn't fix the real problem - the solution simply added a NULL check. The result was no tags extracted for the valid MP3 file. I fixed it after I sat down and first learned the MP3 tag format and structures. It was suppose to check for 4 vs 5 bytes for the non-encoding vs encoding TYER tag. It was always expecting 5 characters. The fix allows for tags to exist now in these valid MP3 files. The point? Where was the simple QA testing and proper understanding of the format?

Marry Starry said...

Very enlightening stuff.