I received
quite a few comments this past week following the publishing of California IT Policy Letter 10-01
which
formally establishes "the use of Open Source Software (OSS) in California state
government as an acceptable practice." While many of my security
colleagues offered words of caution following the announcement (and even a
couple of "are you crazy" comments), most were pretty enthusiastic with
remarks like, "Finally, enlightenment" and "It's about time government
joined the 21st century."
As a
security guy, I've been on both sides of the OSS fence at different times but
I've come to the conclusion that anyone who doesn't think OSS has a place
in today's business or government simply hasn't been paying attention.
While it should never be a casual decision, the organizational
choice to adopt an Open Source Software policy should be made based on issues
such as business need, reliability, ease-of-use, ROI and yes, security.
Being too cavalier can dangerous but it only means you've got to do your
due diligence homework just like when you buy COTS.
I'm not
saying that COTS shouldn't be part of our IT environment just that it's time to
acknowledge the OSS elephant in the room. We need COTS but
should we really trust all COTS software just because it comes with a license
from a reputable vendor? Think about the regular (and irregular) patch
cycles we go through before you answer that question. Is there any
question that the Linux OS, Firefox web browser or Apache web server are mature
products delivering real value? Of course not! In fact they are the
de-facto standards in many organizations? In addition, there are
dozens of excellent OSS security tools that many organizations depend upon to
monitor and identify vulnerabilities within their IT environments.
Nessus, Snort, Nagios, Metasploit, OpenSSH, PuTTY, Nmap, and Wireshark are some
of the OSS security gold standards but there are many, many others.
Over time, the open source community has proven to be somewhat self-policing where the best products get adopted and widely used while the stuff that doesn't meet standards gets a well-deserved funeral. It sems to me that thousands of developers and hackers beating up on open source code is a pretty efficient and transparent way of identifying software bugs and vulnerabilities in OSS. Kind of like software market Darwinism where the strong survive.
There are arguments against using OSS but I've heard the "there's no
guarantee of future support" line so many times it makes me want to
cry. How many times and how many endless hours have you
spent on-hold with tech support without getting the help you needed? At least with the open source community one
of the nice things is the worldwide support
available almost any time of day. So while there are some criticisms, there's also
some valid business rationale for using OSS.
I'm still
accused of being overly paranoid (it's part of the job description), but in
these challenging economic times, all of us need to be on the lookout for
savings and OSS is a very logical option. While not obviating the
need to determine our own security risks, when large organizations like the
Federal government and Department of Defense make policy decisions to use OSS,
aren't we being overly irrational by saying we're too good or too important
that we can't consider the same thing?
I'll bet there are a lot of personal and professional thoughts on OSS from my security colleagues so let's hear them! While you're at it, tell me what are your favorite open source security tools and why? Use this forum to share your insight and experience with other government security professionals.
Michigan has had an internal policy supporting the use of Open Source Products by the Department of Information Technology since 2007. This policy references a standard that defines specific technology categories and the Open Source products identified in those categories that may be used. This is very similar to the process for identifying and selecting "traditional" commercial products. The review and evaluation of products selected for an IT Solution is addressed through the Enterprise Architecture Solution Assessment process.
Good for you, especially since some of the commercial software is drek, this type of level headed thinking will force the poorer quality products to clean up their acts to "compete" with open source. I think Snort is an important tool especially with someone that can write content rules to serve as a poor man's DLP solution. And since you do not have a massive investment you can afford to implement slowly such as starting looking for Credit Card Numbers and SSNs so you are not overwhelmed with data.
SANS has been using SPAM Assassin for years. Finally, almost everyone is using Wire Shark anyway, now it can be legal.
Great job on the blog, it is really nice to see California change direction and become more open and accepting of OSS. I believe that OSS, used properly, could greatly benefit anyone who chooses to adopt it for the reasons you have listed above. With the right skillset and talent to support OSS within California State service, I believe that we will have much more flexibility to do things cheaper and more efficiently. As a security professional I have relied on a number of OSS tools to do my job effectively. You have listed a few of my favorites in your blog, but there are a few more I really like that prove to be very valuable for specific types of audits or assessments.
1) Kismet- A flexible feature rich 802.11 scanner. Kismet will work with any wireless card which supports raw monitoring (rfmon) mode, and (with appropriate hardware) can sniff 802.11b, 802.11a, 802.11g, and 802.11n traffic.
2) Backtrack- Linux Live CD (Bootable) that contains most of the gold standard tools listed in your blog. Probably the most complete live CD for security professionals.
3) OWASP Live CD- Great for Web App security testing.
4)Network Security Toolkit- Contains many of the same tools that Backtrack does. The majority of tools published in the article: Top 100 Security Tools by INSECURE.ORG are available in the toolkit. An advanced Web User Interface (WUI) is provided for system administration, navigation, automation and configuration of many network and security applications found within the NST distribution. The WUI is helpful for those who are not as familiar with many of the *NIX commands that are necessary to drive tools found in other OSS.
As a veteran of a state government organization for many years now I can attest that open source tools have made our lives easier and our work more effective.
The tools that you and the other commenters have mentioned are all in my toolkit as well. We are currently assessing Spiceworks for some limited use.
With little or no funding for security or more bodies we have turned mainly to open source to roll our own solutions and in the process have learned quite a bit from both our mistakes and successes in their implementation.
COTS software is certainly not going away and there are plenty of great products in their ranks. But if the main benefit of COTS use is an "official" support number to call, experiences with sub-par support techs often lead me to back the open source community.
In fact, between you and me and the Internet: a fair amount of my success in working through IT and security issues has been my reliance on the fact that some other poor schlub has already had the same problem as me, figured out the fix and documented it online.
There's no substitute for the hive mind.
You hit this one out of the park. OSS must be a part of the equation and I am happy to see someone in your position talking about it. What makes me thrilled though is these words in your article "Business need", "ROI", "Due diligence". Any time you are making a decision these words need to be front and center and I can't tell you how much I appreciate hearing them from a security guy. That label "Security guy" no longer fits. You are an Executive and thoughts and articles like this only prove that point.
I've seen OSS projects mature over the past 15 years. Out of those, some projects such as Apache have definitely matured to an enterprise quality level. Others such as MySQL have had their ups and downs. I understand that MySQL is partially commercial and partially open-source, but let's not debate that here. The point I am trying to make is "can we evolve some criteria to score the maturity of an OSS project?". Ubuntu, for instance, has done a fantastic job of Linux distro maintenance. They promise two releases every year and have charted out goals for every release. And we've also seen projects such as Nessus, which were once open source, and then got swallowed into a closed-source commercial model. Metasploit may be following the same path.
Before adopting an OSS project it would certainly help to look at its "maturity, stability and continuity criteria"
What should this criteria include?
I'm open to collaboration.
"Danger, Will Robinson, Danger!" - OSS will be more demanding on your organization due to the vast capabilities to implement custom solutions. I can't tell you how many times I've heard (in State government) that "we're understaffed" or "we can't attract talent". It seems to me that moving to OSS contradicts that concern. Will organizations really take into account higher recruiting costs, increased staff salaries, and the impact that migrating to these new solutions (or worse, maintaining multiple solutions) will have? Will they just replace the cost of software licensing with these other hidden, long-term costs? Or will the move to OSS platforms / solutions be approved but not the staff / training budgets? *Yipes!* It's not prudent for organizations to undertake this change based on "tough economic times" alone - in fact, the odds of securing the appropriate (soft) resources would likely be easier to do under better economic conditions.
Risk management needs to be a key part of this decision. Although I have philosophical issues with the licensing morass that is to be read out of the various flavors of public licensing, I tend to recommend Commercial Open Source Software to my clients in favor of Free Open Source Software in order to mitigate some of the risk associated with this strategy while still reaping the benefits of interoperability, configurability and lower licensing costs inherent with OSS.
I agree with you Mark that we in State Government should keep an open view regarding this topic. Many OSS products have been actively developed for years and actually become the backend for many commercial ones; Nagios is a good example. I think one point that is often missed when discussing open source are the upfront gains. What I mean by this is that a need often develops into a project. Frequently (as in government), those projects, in a Project Management framework or not, lack well defined scopes or an overall understanding of what a technology can fully provide (or not). It has been my experience (in general) that this issue has lead us to multiple overlapping, little or unused, costly technologies some just sitting on the shelf. Though open source does often require a certain skill set in some cases to get installed and in production, it provides a much larger benefit to risk with an education of what a similar commercial technology could do without a (often) high upfront cost (including Maintenance and Support). In addition, one could then build a very descriptive scope and need based on their open source experience without the commercial cost and guilt if it doesn't work; then go to a commercial version if needed. Open source provides an opportunity to get it right when performing a technology gap analysis without having too much upfront investment. Personally, I think any System Administrator would jump at an opportunity to reach out into this area and if they come upon a lot of troubleshooting (to a certain extent) the more to them and the better off their education and experience is for them and your location.
In the OSS vs. COTS debate it's important to make the distinction between "using" OSS software and "designing" an OSS solution.
If you run an IT shop you're using OSS software. As you stated, Linux, Apache, MySQL, Firefox, etc., have become the de facto standards but beyond that you'll find OSS in almost every product available. For example, if you use SSL or SSH you're almost certainly using libraries from the OpenSSL project. If you run Cisco gear, the new Pix OS and IOS-XR use a number of the GNU OSS utilities. Solaris has now incorporated the same GNU utilities along side its' own old school Unix utilities. If you buy a vendor appliance it's most likely running Linux as its' OS. COTS vendors are using the crowd-sourcing model to borrow from the OSS community instead of employing programmers to reinvent the wheel in house.
Designing an OSS solution is a different matter. In this case you're relying on the expertise of your IT staff to find the appropriate free and open source software then build and configure a custom solution. There is little or no up front cost, but support and maintenance and thus the viability of the solution, is in the hands of your in house IT staff. As budgets deteriorate and unemployment increases this becomes the more viable option. System administrators will be easier to find and cheaper to employ while funding for new software support contracts will tougher to get.
The OSS vs COTS question is older that Jim Jupiter himself and no need to rehash the pros and cons of it again. OSS has played a far more critical role in the commercial and government environments than has been generally recognized. I have seen SCADA and Control infrastructures depend on OSS applications for security strategies, software solutions, and infrastructure support. In the current economic times, it is refreshing seeing the State of California moving forward and adopting, as a policy, the use of OSS to support the mission of the State.
Having worked on all sides of the fence, I find the high expectations of COTS risible. I've done software development for COTS vendors and frequently the in-house development process is a joke: haphazard use of revision-control software, little testing, deployments with known bugs/vulnerabilities, backdoor passwords, etc. And Mark touches on support issues with "How many times and how many endless hours have you spent on-hold with tech support without getting the help you needed?" Too often the COTS vendor outsources support to script-following lackeys that can't provide any more help than the equally weak documentation. Even if you do manage to get a knowledgeable support tech, their ability to actually help you often depends on their access to developers. Lastly, when the COTS vendor decides to terminate features we use, or decides to change the interface at a whim, or we need a custom feature, we are at the vendor's mercy. And for this, one pays dearly.
Compare that to my F/OSS experiences where best practices generally rule; the entire code-base is usually available via a public repository; all known bugs are available for public review & comment in a public bug-tracker; backdoors are readily spotted in the source; support mailing lists, web forums, and IRC usually give direct access to the developers and a community of other users who use regularly use the software. Documentation excels because without quality documentation, the project flounders. If the core developers decide to take the application in a different direction, nothing forces me to upgrade and future patches can often be cherry-picked if needed while maintaining the status quo. With the source code, we can add features we need locally even if nobody else needs them. All this for free usually, though often available with an additional support option for those that need a number to call.
Given the choice between the two, F/OSS wins my support almost every time.
-Tim