My Views on Tech, Security, Life

   I have been an administrator for a Host Intrusion Prevention System for a little over a year now on a network.  I have been using the McAfee HIPs  and ePO system and while for the most part I like it a lot, there are a few major downfalls of it that I feel that in the long run, the current path that vendors are following will not work in the long run.   I feel that the idea is fundamentally flawed and I will explain why.

   First, let me explain what HIPs is.  HIPs is an idea that stands for Host Intrusion Prevention.  The idea behind Network Intrusion Prevention has been around for a while, but Host Intrusion Prevention is about moving this same capability to the host.  Instead of just monitoring network traffic most HIPs product monitor system calls, and the flow of applications.   If the HIPs product sees something it doesn’t like as defined by the signatures it will try and prevent it.  For example if Internet Explorer all of a sudden spawns a child process and makes the system call to exec “cmd.exe” you are probably owned through a browser exploit.  The idea is that if we can control applications to not be able to perform system calls that are harmful to the system we can then create a safe environment for applications to run.  The idea sounds great, but falls flat in two areas, transperency and testing.

   Transparency is the first problem that any HIPs type application has.  This has more to do with the Windows ecosystem then the HIPs itself.  When an application is creating events that the system is blocking all you are often given is an event name and an event description.  Sometimes the event description is helpful, but this is very rare.  Is this event normal?  No idea…. You could try to create a known clean box and try to recreate the situation which is both time consuming but still doesn’t give you an idea what is happening.  I have seen one particular situation where HIPs was preventing users from adding a signature to their emails in Outlook.  The event described a vulnerability in Visual Studio.  How are these two related? I guess you could say Outlook was probably made with Visual Studio?  No idea.  This was affecting every machine on our network, so I assumed it was “normal”, I used a fresh built machine to test this and it still was doing this event, so what could I do?  I disabled the event.  There was nothing else I could do.   Did I just open up our systems to some large gaping holes?  Maybe.  You really have no way of knowing.  The nature of closed source prevents you from viewing what all your applications are doing.  Unlike Network Intrusion Prevention Systems where I can see all the traffic, create a tcpdump of it and then study it in wireshark, I don’t have this luxury with HIPs type products.  The closest I have found to this is using the SysInternals Suite to try and monitor what the system is doing (excellent FREE! product by the way). 

   To make matters worst, I still haven’t figured out how I can even view the signatures themselves!  Unlike a product like the Snort NIPs product, where the majority of the rules are just plain text with a very clear and documented standard, I still haven’t found a way to do the same with McAfee HIPs (no idea if other HIPs products are the same).  How can I make decisions as an IA professional on what is malicious and what is not when I don’t have access to the underpinnings of the system?  That to me is crazy!  Security products need to be transperent.  Every day IA professionals are making decisions as to what is normal and what is abnormal, if we have no access to see the underpinnings then we can’t properly make these decisions and the product loses its value.

   The next issue that I have with HIPs is testing.  Any large company who creates a quality product is going to make their programs undergo rigorous testing.  Organizations buy these products because they product has already been developed and properly tested.  The organization has decided that buying product X would be cheaper then building their own.  This is a simple IT concept and is why companies like Microsoft, Oracle and others have such successful products.  The problem with HIPs is all that testing becomes useless once HIPs is introduced as vendors application’s paths are altered which can cause large issues that aren’t easily testable.  HIPs products contends that the users need to test their applications with HIPs.  This is done to a certain extent, but can an organization be expected to hit every single doohicky in every single application that they have deployed?  Heck no!  Microsoft Office alone would take months!  So what do organizations do?  They either turn off all events that are being triggered without properly investigating them or they just let things break, harming availability and making users frustrated, and fix issues as they come up.  No test lab is ever going to be able to recreate everything possible your users can do.   

   If the purchaser of the product can’t properly test the products they use with HIPs who should the burden be on?  The vendors of the application?  Problem here is there are a large number of HIPs products.  It isn’t Microsoft’s responsibility to their applications with every HIPs product and every single configuration of these products.  Maybe the vendor’s of the HIPs products need to start testing their products with all the available applications on the market… Of course this is never going to happen as there are probably millions of applications out there, not counting all the custom applications that are available.

   The reality is there is no good answer to this question with how current HIPs applications work!  I have seen HIPs break applications after patches are applied.  I have seen HIPs prevent patches from being applied.  I have seen servers brought  down (even though they worked fine for months) creating losses in availability for hours and more.  HIPs comes at the expense of software availability and software reacting reliably every time. 

   What can be done to fix this issue?  Well the first thing is to create software that doesn’t need HIPs.  HIPs is really just a band aid solution to a larger problem.  HIPs assumes software is going to have security vulnerabilities and that it needs to be monitored and controlled in such a way that prevents these issues from further damaging the system.  If the software didn’t have holes this wouldn’t be required.  Sounds great in theory, but is a pipe dream with the complexity of today’s software and the current tools available to create such software, so we are still stuck with HIPs.

   The next thing that can be done to fix this issue is to standardize HIPs.  If there was a standard to HIPs signatures and how their systems reacted software could be tested with HIPs enabled to verify their software works reliably.   Opening up the signature sets and defining how HIPs products act with these signatures could fix the issue of testing as well as help the issue with transparency (not entirely because I doubt the software will ever of open source).  Creating a single central repository that is shared by all vendors would lower the number of products that a software maker needs to test their products with as they could test it with one HIPs application and it should pertain to all other third party systems using the same signature set.

   The other fix could be to have one vendor dominate the market similar to how SNORT is pretty much the undisputed champion of NIDs and NIPs devices (a lot of commercial products use SNORT as their base).   This would allow vendors of software to only have to test one HIPs product to ensure their software doesn’t break at their customer site.  It would also allow the vendor to draw up detailed documentation on how to turn off or tune specific signatures to allow their products to work (similar to how vendors usually specify the ports to open on the firewall).  One way I could see this happening is if Microsoft were to integrate a HIPs type component into a future version of Windows.  Not sure how great of a solution this would be, but it is a possibility. 

  If vendors want their HIPs products to be more widespread and create more happy customers then they need to fix these two issues.  If you have any other suggestions let me know, I would be interested in hearing them!

An article I had written was published in Hakin9 magazine and is available at: Hakin 9 June.  If you are not familiar with hakin9 it is a great magazine that features a ton of information every month that is in depth and useful.  This is probably the geekiest magazine out there for all things security related!

This article from SANS about how the McAfee issue affected a communities incident response plan is excellent.  Like many other organizations the communities , they were hit with the latest AV vendor mess-up which brought their entire ER center down to a halt.

The real issue here isn’t that McAfee made a mistake, it is that AV has been failing and is quickly losing its effectiveness.  Virus definition updates used to be weekly, which gave AV vendors plenty of time to update the definitions and sufficiently test them.  As viruses spread faster and worms took hold of networks AV vendors started to update on a daily basis.  With today’s AV products many vendors are moving to hourly updates!  How do you test against that?  You can automate it to a certain extent, but it is impossible to test for every scenario that a user may encounter.  It is really a bad situation to be in.  Someone needs to come up with a better solution to AV.  The current methods are failing and will continue to fail.  Heuristic checks (how a virus acts instead of the hash of a file) was a step in the right direction, but false positives a lot and has every more likelihood of causing a major error like was seen with McAfee.  Application whitelisting, which is going through and marking off each and every executable allowed to run, is great if you have a very controlled network and don’t mind a management headache.

I would like to start seeing signed executables by everyone.  This way, as an IA professional I could say that I trust this vendor and now all binaries signed by this vendor will run.  This reduces a lot of headache of going through and approving each and every application.  Better yet, if Microsoft were to get a PROPER package manager they could do a lot of really nice items with it.  Imagine you wanted a program from vendor Acme Inc.  You add this vendors URL and public key certificate to the new Windows package manager and then up comes a nice user friendly application that allows you to download the software you want.  Because the binaries are signed, and you decided to trust the binary, it is safe to run.  Any application not in the package manager list will not run.  Managing an enterprise in this fashion would be great.  As if that wasn’t reason enough to do this, now you can update much easier.  You can run your own repositories for your own company to manage updates (always a plus for security).  Users wouldn’t even really know about the packet signing because they would get so used to installing software in this fashion, and would love the benefits it would quickly become the only way that they want to do it.  The fact that it runs only signed binaries is an added plus.

The only drawback to this method would be the fact that now you need to train users to not add junk to their sources list.  This would be hard to do in a home environment, but really isn’t a large issue in a corporate environment.  I have long wanted Windows to implement a package manager.  It would make one of my major gripes with it disappear.  Here is to hoping it is a feature in Windows 8.

Recently, as an Incident Handler I have been seeing a large upswing in malicious ads.  These are advertisements that are served on legitimate sites that normally you wouldn’t think twice about trusting to not have malicious content.  These sites aren’t directly serving the content, but instead are using ad networks that the site has contracted with to serve ads while you are viewing the information.  While this isn’t a new idea, I have recently been seeing a large upswing in this trend and I have talked to other security professionals who have echoed this statement.  Some of this malware is of the 0-day variety meaning patches won’t help you.  I have seen malicious ads that, at the time of being dropped on a box, did not get detected by AV and other defensive mechanisms.  As information security professionals we can’t tell people to be careful online as they are often just visiting their favorite sites that serve legitimate and useful content.  The only way to defend against these attacks is to either not visit sites with ads which makes it impossible to use about 99% of the Internet or install an application like ad block which brings us to the point of this article.

I understand the value of these ads. Many of these site are in fact funded by these networks.  The sites cost a lot to maintain from the writers who give us great content to the machines that power their web sites.  For the most part, micro payment experiments have failed (We will see if Murdoch’s experiment pays off).  Arstechnica had an excellent article about how services like ad block were hurting the sites that we use every day. From a purely capitalistic stand point I understand the need for these ads, and honestly, I don’t really mind them that much even if they are yelling “hit the monkey for a free ipad” and distracting me while I am reading (I still really can’t stand the ones that slide over what you are reading, those just piss me off and make me not want to buy from the company).

I would be more then willing to view these ads under one condition, I could be assured that they wouldn’t contain any malicious content.  By allowing these ads you are essentially deciding to trust the ad agency that is running these ads on the sites you view just as much as you are trusting the sites them self.  The ad agencies so far have proven to be grossly incompetent at protecting their users and therefore I am forced to take matters into my own hands and say that to protect your self, the only responsible thing to do is to block advertisements on sites.  The ad agencies that I have seen allow javascript and other executable code (possibly malicious) with what appears to be little validation.  What is worst that they will often allow the ads to pull from other sites that they don’t even control.  This means that even if the ad agency does their due diligence on the code that is given to them for the advertisement and checking out the site that is linked to it, there is no reason that the attacker can’t later change it to something malicious.  This is irresponsible and unacceptable and it is time for the advertisement networks want us to use their ads it is time for them to start taking responsibility.

Until the ad agencies begin to take responsibility and their customers (the sites serving the ads, not the users viewing the sites!) begin dropping ad agencies on the first occasion that malicious content is served this trend is going to grow worst.  Ad agencies provide a perfect vehicle for hackers to deliver their content as traditional user education no longer works of don’t visit shady sites you don’t recognize, click on shady attachments, etc.  If the ad agencies aren’t careful though, many users are going to begin blocking these ads on sites (more then already do).  Often after an incident has been traced back to an ad agency it isn’t uncommon block that particular site at the proxy server which prevents all users on their network from seeing these ads.  I know of a few organizations that are considering or are blocking ads all together.  Popular web filters like Websense now blocks and categorizes advertisements meaning organizations can block these ads with one easy click!  If the ad agencies want to put an end to this problem before the problem gets out of hand here are my suggestions:

1. I personally wish they would only allow jpgs and animated gifs, but I also realize that this isn’t going to happen.  Therefore and executable mobile code (javascript, flash, etc) must be reviewed and in NON obfuscated form (this will make my life a million times easier looking at traffic captures).  This code must be approved before it is submitted to the ad networks
2. No external sources for ads.  All content is to be hosted on the ad networks themselves, no 0×0 iframes, no pointing to external javascript libraries, etc.  You can’t verify code that you don’t own.
3. Begin following security best practices and understand that you are a liability to your customers and their users!  This one companies may be following I have no idea, but to my knowledge there is no auditing regulations for these companies, though I could be wrong on this.

The point is, until the ad networks begin to show they are being responsible I will continue to fight ads which may be malicious.  The battle has only begun and I am more then willing to surrender when the ad networks start to maintain a better track record.

The problem, is I understand that this probably will not happen.  Ad agencies want to make it easier for their customers (the people hiring them to display the ads) to submit their content not harder.  They don’t want to create a one month waiting period for the information to be approved.  They also don’t want to be significantly more expensive then their competitors as they begin to have to build in the costs of auditing code.  This is why  I don’t foresee this happening anytime soon.  That is, not until the industry starts to fall apart and then their users may realize that adblock is really convenient and they don’t want to go back to annoying ads that blast music over your speakers when you accidentally mouse over some image.

With all the talk of ringing up a large bill with Obama’s proposed spending bills I, as a tax paying American citizen and someone who works in federal IT, would like to give advice on how Obama can find some pennies in the cushions.

My first proposed plan is to make all major software projects open source, unless there is a reason otherwise due to the sensitivity of the software itself.  An agency that creates software must open source the software.  If they feel the need to keep the source code secret, then the agency needs to go through an approval committee of some sort.  It is important to understand that rarely the software in a classified environment needs to be classified.  For example, a database can hold all sorts of classified information that should not ever get leaked, but the software that stores this data is not sensitive by itself.   I can think of situations like the software to control a missile, where you would not want the source code to be leaked, but I would bet that this would be the exception and not the rule for most federal IT software projects.

OSS is already used heavily by the government.  Software like Linux, MySQL, various APIs and more are all used by the federal government already.  Open Source software is nothing new to the government.  Often the federal government will choose to go through an OSS vendor like Red Hat or Sun in order to have support with their product.  What I do not see a lot of is the government heavily investing in working on OSS.  I would like to see a time where the federal government begins to use our money to contribute to the community as well as to other agencies.

I am not arguing that the federal government be mandated to use Open Source software, I am only talking about projects which the government directly funds.   Oracle can still keep the code of their database if they please.

Here are some potential counter-arguments that I see towards my debate:

Communism!- Before anyone screams that we don’t have communism in America, I need to remind the reader that the money for the software project comes from the tax payers, not from the government itself.  I would not make this argument if it were not for this fact.  I don’t feel that any corporation is obligated to open source their software (though I would submit it may not be a bad idea), I do feel that software funded by the US taxpayer needs to be put in the hands of the US citizens, and not locked behind closed networks.

Think of the security- This has been shouted all over in the argument between open and closed source products.  If we open source the code, then hackers will be able to look at the code and break into our systems.  The problem with this argument is that the security vulnerabilities are still there in the closed source software and just aren’t being found.  I would submit that this is a bigger issue for the federal government then it is the commercial software industry due to the nature of the governments attackers.   Windows is installed on millions of computers throughout the world.  Attackers of Windows vulnerabilities are rarely looking to break into an individual machine, but rather looking to “own” as many machines as they possibly can in as short a time as possible.  Botnets and malware are great examples of this.  Rarely does the attacker aim at a particular person, all they want is a machine and if it happens to have a steady Internet connection, then they will be particularly happy.  Attackers which attack software which is specific to the federal government aren’t looking to attack as many machines as possible, instead they are looking to attack a specific machine.  The Federal Government is going to be worried about terrorist and militant groups a lot more then regular users.  The reason this matters is that the shotgun attack is much more likely to be detected by someone.  Someone is going to figure out that their machine is acting funny.  Someone is going to notice that their web server has been opening strange ports, etc.  Anyone aiming to attack the federal government will be aiming to do so discretely.  The likelihood of the attack being detected with a targeted attack will drop significantly.  This increases the risk of security vulnerabilities as they are less likely to be found.

The attack surface is larger and therefore people are more likely to spot anomalies.  While I don’t doubt that the government doesn’t employ many people to find vulnerabilities in software, there is no way it can be more then the millions of people who may use it around the world.  By open sourcing the products not only does the community help you find security flaws.  Interest from the good guys grows which means the ability to find security flaws grows as now they may be wanting to use your software and may want to be involved.  It will also be possible to keep many contractors in check through this process.  Often the government hires contractors to do their software development work.  Rarely do they as the government perform a total code review towards the end.  Instead they opt for checklists which do nothing to increase software security.  If a contractor knows that their software will be open sourced they will do all that they can to ensure it is of the highest quality, because if they do not, it could lead to be an embarrassment to the entire company.

A perfect example to this would be Diebold and how when the software was leaked to the public, it quickly became apparent that their software was garbage and should never have been trusted with something as critical as our vote.  Contractor and government accountability is increased by this process.  Anyone who argues on a basis of security is starting on shaky ground.

Agencies will never work together- I am not going to try and argue this point here – agencies will never work together as well as they should.  There are many examples of agency A creating software and then agency B creating the same software that does almost the same exact thing.  Duplication of effort will never be eliminated. Agencies seem to often forget that they are not in competition with one another.

There are a few advantages to forcing federal software to be open source.  First of all, forcing the agency to publish their code will allows another agency to find this.  There have been attempts to recreate this spirit of collaboration within DOD with forge.mil, but so far the projects have not been active and I wouldn’t deem forge.mil as  a raging success, but the idea is there.  If the software was open to people outside of the DOD and the  government then the successful software that is created from this collaboration will become more known.  I would bet that a software write up in eweek is more likely to be learned about by senior management then forge.mil.

Accountability of software creation development efforts will be increased as well.  Newspapers will begin covering why the CIA is working on software that is almost identical in nature to what the FBI is working on.  No agency wants to have a negative write up in the newspaper and agencies will be cautious of duplicating too much work across networks.  The American public will know when the government is wasting our money working on challenges that it already has faced.  In addition this will promote the use of other open source software saving more money.  Why is the US government creating software which is available for free already?

Open Source Software is abandoned all the time- Look on Sourceforge and see all the projects that haven’t been touched in years.  Open Source Software is abandoned on a regular basis.  People lose interest in their software and if no one else picked up the project it may lie dormant forever.

The problem with this argument is that these same things happen in the federal government all the time.  I have seen numerous occasions where a software blows up in some strange way that needs someone to look under the hood to fix.  When you go to try and find the original developers you quickly find out they have been placed on new contracts and really have no time for this existing software.  Now someone is forced to solve the error from ground up which can often takes weeks.

Will open sourcing the software eliminate this problem? No, not completely, but I will submit to you that this problem will be lowered.  Even if the federal government abandons a project (never a good idea while you are still using the software) there is a chance, if the software is popular, bugs will be found by other users of the software.  Software may also be picked up by other agencies increasing the user base.   The larger the user base, the more likely the code will be understood by more developers and more people will have a vested interest in finding bugs in the software.

The economy is bad enough, now you are talking about more lost jobs- I hope not as I am employed by the government!  I don’t think that jobs would be lost and I would think that contracting will continue to be lucrative.   I would bet that more projects would be spawned from this arrangement because the government is saving money.  Another side benefit would be that the American People may be more willing to pickup the bill if they are able to see benefits from the software.

Will this hurt the commercial software industry, probably would hurt many of the companies that aren’t ready for it.   Red Hat, IBM and many others have proven that you can be successful working with others through Open Source.  The only software companies that would be hurt would be companies who are unwilling to change.  Companies could flourish from small projects that the government has decided they were critical and would like to see more work.  If a small group of people is creating software in their spare time,  and the government decides they want more work from this group.  All they need to do is hire these developers themselves full time.  Often Open Source Software is created by people in their spare time, imagine how much quicker and better software would be developed if these passionate people were paid to work on it full time!  Being paid for this work is every OSS developer’s dream.  I would think that the federal government would have a much easier job of finding skilled developers using this method of development.

Open Source Software isn’t Common Criteria Certified for FIPs compliance- Currently software companies need to pay for CC evaluations on their own for the “privilege” of selling to the US government.  Open Source software rarely can pay for these expensive and time consuming tests.  The US government should fund NIST to evaluate software for free.  OpenSSL has been FIPs certified and is now used extensively throughout the government.  An example I can think of is desktop encryption software.  All of the DOD is required to encrypt all data stored on removable media regardless of its classification level.  This requires basic software that encrypts data securely.  Currently, the only software that can do this is commercial software which costs money.  Look at all DOD machines that now need this software and even if you can get the software cheaply, it still will cost a lot of money.  With this money the DOD could have easily gotten a program like TrueCrypt approved.  If the software doesn’t integrate into their current PKI infrastructure, there is nothing stopping them from adding it.  Once the DOD develops this software any agency can now use the software throughout the federal government.  The money comes from the same source anyway, the American People!

Software created by the government is rarely generic enough for use by anyone outside of the government- I would hope that the government is implementing best practice principles and utilizing Object Oriented software design!  I am sure that no matter how obscure software is in the US government at least some of the code will be used by other agencies as well as other people outside the government.  Even if the software is looked at by someone interested in the software for school, it has gotten use.

Open Source software is of inferior quality- Depending on the software you are looking at, this may be the case.  Open Source software, like commercial software is of varying quality.  In instances where the government wants to build off of an existing open source software project.  This will happen in some cases.  The software may be incomplete, buggy, insecure, etc.  With this in mind, the government should rarely then come to the conclusion that we will build their own software.  I am fine with the government deciding that commercial software is better.  If they choose to go that route, that is their prerogative.  The commercial software does not need to be open sourced because of its use in the federal government.

There are numerous software products that have been created within the government  that duplicate functionality of many popular Open Source packages, and quite often the software sucks.  In one instance I saw a group that had created software which allows you to deploy custom software to Linux boxes that was written in python.   The software allowed you to install and remove software on boxes. While the idea sounded great, it completely duplicated the functionality of RPM.  I highly doubt that this software was better then the venerable RPM, which has been used and abused for years!  There are far too many projects like this, if the open source software isn’t perfect, they will choose to build their own custom software.  If the software isn’t of proper quality, then it will usually be easier to fund someone to make the software better then to build it from ground up.

Open Sourcing a software package could lead to more work- This may be true if no one outside of the government is working on project.  Public documentation needs to be created, a website to host the software needs to be added, etc.  This will increase the work of one agency, but if the project is worthwhile and a good product, then this burden will be lifted by an other agencies, Open Source programmers, companies, universities, etc. looking to help with the project for their own interests.  The total cost savings across the board for these projects will easily out weigh the benefits.

More »

Welcome to my new blog.  I feel like I have so much up in my head that if I dump it on paper then someone will eventually be interested.  We will see if that is the case.

For my first brain dump I am going to look at the following article I found online: link

First off, I think that this is the most ridiculous thing I ever heard.  This is someone who is looking at system design from a very limited point of view (the usability designer).  Obviously this guy either:

a) doesn’t work in an office

b)  doesn’t ever leave his house and lives alone

The reason for the asterisks is very simple, it prevents someone looking at your screen from reading what you are typing.  This is called “shoulder surfing” in the security world.  Even if you only used the asterisks for sites that need to be “secure” you run into the problem that a lot of people use the same password for all sites.  I could go on about the idiocy and short sightedness of this, but I am not, instead I am going to discuss something else.

This brings out an interesting point, security is at odds with almost everything.  Usability takes a hit with passwords, that are a pain in the rear to memorize.  Look at Microsoft Vista’s UAC,  so many people have complained so much that it became the punch line of an Apple commercial (I have other issues with UAC, mainly that it is useless that it doesn’t tell you what is actually happening).  System security architecture slows the process of building a system down, security takes a while to get right.  So is it all worth it?  Why do we do this?  Why is security needed at all?

Security is always tricky line between making a system secure and still allowing the system development project to succeed.  The safest system is the one that is disconnected and buried in the ground, of course it isn’t very useful in this state.  Security will always be at odds with everything, and this is the important thing to keep in mind when designing a system.  The security professionals job is to work with the project and ensure it’s success while still doing their job.  On the other end though, the people in the project need to understand and be willing to accept that security will come with compromise.  Their will be decisions from both sides that the other will not like.  It is the job of the security professional to educate the project on why security is needed and ensure that security becomes in-grained in the entire project.