From general-return-26493-apmail-incubator-general-archive=incubator.apache.org@incubator.apache.org Fri Sep 17 11:32:45 2010 Return-Path: Delivered-To: apmail-incubator-general-archive@www.apache.org Received: (qmail 2555 invoked from network); 17 Sep 2010 11:32:41 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 17 Sep 2010 11:32:41 -0000 Received: (qmail 39181 invoked by uid 500); 17 Sep 2010 11:32:41 -0000 Delivered-To: apmail-incubator-general-archive@incubator.apache.org Received: (qmail 38577 invoked by uid 500); 17 Sep 2010 11:32:38 -0000 Mailing-List: contact general-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@incubator.apache.org Delivered-To: mailing list general@incubator.apache.org Received: (qmail 38566 invoked by uid 99); 17 Sep 2010 11:32:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 11:32:37 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 76.96.30.17 is neither permitted nor denied by domain of jim@jagunet.com) Received: from [76.96.30.17] (HELO qmta10.emeryville.ca.mail.comcast.net) (76.96.30.17) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 11:32:31 +0000 Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta10.emeryville.ca.mail.comcast.net with comcast id 7nQa1f00117UAYkAAnYBfb; Fri, 17 Sep 2010 11:32:11 +0000 Received: from [192.168.199.10] ([69.251.84.64]) by omta13.emeryville.ca.mail.comcast.net with comcast id 7nY81f0041PGofZ8ZnY9qT; Fri, 17 Sep 2010 11:32:10 +0000 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Apple Message framework v1081) Subject: Re: Real-time communication (was [VOTE] ALOIS to enter the incubator) From: Jim Jagielski In-Reply-To: Date: Fri, 17 Sep 2010 07:32:07 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <53290BE1-AC74-4C3D-8252-ED604E6625D1@jaguNET.com> References: To: general@incubator.apache.org X-Mailer: Apple Mail (2.1081) I still don't see how that gets around the perception, and the reality, that development is being done outside the list. So I don't see that proposal as helping out at all... On Sep 16, 2010, at 3:27 PM, Scott Deboy wrote: > I understand the concern raised by the use of real-time communication = for > Apache projects - that decisions may be made off-list, and that folks = who > aren't a party to the real-time communication do not have the = opportunity to > benefit from or impact the decisions that result from the real-time > communication. >=20 > The proposal does offer what seems to be a reasonable compromise: 'we = would > send the logs daily to the mailing list.' >=20 > Daily chat logs posted to the dev list, coupled with good mentoring = and > guidance that decisions need to be made on the mailing list, would = seem to > minimize the risk. >=20 > I'm interested in what others think of their proposal for supporting > real-time communication, and curious what others are doing, if = anything, to > support the growing interest in real-time communication between = project > participants. >=20 > Scott >=20 >=20 > On Thu, Sep 16, 2010 at 10:49 AM, Craig L Russell > wrote: >=20 >> Hi Urs, >>=20 >> My only concern is the request to have a chat channel. There's wide = use of >> chat channels in Apache (the periodic board and members' meetings = make use >> of them, and infrastructure uses channels to advantage). >>=20 >> But for an incubating project, I'd strongly discourage use of chat as = a >> communication channel. >>=20 >> +1 >>=20 >> Craig >>=20 >>=20 >> On Aug 26, 2010, at 9:09 AM, Urs Lerch wrote: >>=20 >> Hi, >>>=20 >>> I would like to call a vote for accepting "ALOIS" for incubation in >>> the Apache Incubator. The full proposal is available below and on = the >>> proposal wiki page (http://wiki.apache.org/incubator/AloisProposal). = We >>> ask the Incubator PMC to sponsor it, with Scott Deboy volunteering = as >>> Champion and Mentor. Additional mentors are warmly welcome. >>>=20 >>> Please cast your vote: >>>=20 >>> [ ] +1, bring ALOIS into Incubator >>> [ ] +0, I don't care either way, >>> [ ] -1, do not bring ALOIS into Incubator, because... >>>=20 >>> This vote will be open for 72 hours and only votes from the = Incubator >>> PMC are binding. >>>=20 >>> Thanks, >>> Urs >>>=20 >>>=20 >>> -------------------------------------------- >>>=20 >>>=20 >>> =3D Preface =3D >>>=20 >>> ALOIS is a log collection and correlation software with reporting = and >>> alarming functionalities. It has been implemented by the Swiss = company >>> IMSEC for a customer about five years ago. GPL-licenced, implemented = in >>> Ruby and completely based on other OSS-licensed components, it was >>> designed for the open source community right from the start. Now = that >>> the software has shown its functioning over several years in = production >>> with the one customer and one IMSEC-internal installation, it seems = to >>> be the right time to open it to a wider community. >>>=20 >>>=20 >>> =3D Abstract =3D >>>=20 >>> ALOIS stands for =84Advanced Logging and Intrusion Detection System=93= and >>> is meant to be a fully implemented open source SIEM (security >>> information and event management) system. >>>=20 >>>=20 >>> =3D Proposal =3D >>>=20 >>> While almost all other SIEM software, be it closed or open source, >>> concentrate on the technological part of security monitoring, ALOIS = is >>> aimed to monitor the security of the content. It intends to be >>> pro-active in the detection of potential loss, theft, mistaken >>> modification or unauthorized access. ALOIS works on log messages and >>> thus contains all the basic functionality of a conventional SIEM, as >>> centralized collecting, normalizing, aggregation, analyzing and >>> correlating of all log messages, as well as reporting all security >>> related events. Therefore it can be used as any other SIEM. >>>=20 >>> ALOIS consists of five modules interacting to ensure a scaleable >>> functionality of a SIEM: >>>=20 >>> * Insink is the message sink, which is the receiving entry point for >>> all the different log messages into ALOIS. It is partly based on the >>> syslog-ng software. Insink listens for messages (UDP), waits for >>> messages (TCP), receives message collections (files, emails) and >>> pre-filters them to prevent from message flow overload. >>>=20 >>> * Pumpy is the incoming FIFO buffer, implemented as a relational >>> database tables. which contain the incoming original messages (in = raw >>> format). In a complex system setup, there may be several insink >>> instances, e.g. for a group of hosts, for specific types of = messages, or >>> for high-avaliablity. >>>=20 >>> * Prisma contains logic to split up the text of log messages into >>> separate fields, based on regular expressions. Actually, "prisma" is = a >>> set of "prismi", each one prisma for one type of log message = (apache, >>> cisco etc. Several prismi can be applied to the same message. This >>> allows for stacked messages, i.e. forwarded log messages contained = in >>> compressed files contained in e-mail messages. The data retrieved = form >>> the log messages is stored in a database called Dobby. Due to prisma >>> being written in Ruby, prismi can be applied interactively (when = having >>> system access). >>>=20 >>> * Dobby is the central log database. It should be separated from the >>> Pumpy database for availability and performance reasons. The current >>> implementation is based on MySQL. >>>=20 >>> * The Analyzer contains the two sub-systems Lizard and Reptor. = Lizard >>> is the analysis engine and user interface of ALOIS, implemented in = Ruby >>> on Rails using AJAX. It allows for interactive browsing through the >>> collected data, exclusion/inclusion/selection of data, data sorting, >>> data filtering, creation of views, ad-hoc textual and graphical >>> reporting. Reptor allows for automatic activation of views and >>> comparison of these views' results to a predefined result (pattern >>> matching). In case of mismatch, Reptor sends the result to = predefined >>> e-mail addresses. >>>=20 >>> Its modular design guarantees ALOIS to scale from little to large >>> organizations. Since there exists a Debian package, it's easy to = build a >>> test system or even a productive system for small environments. >>>=20 >>> Although the software has been in productive use for a few years, = there >>> is still a lot of desired functionality missing. The plugability of = new >>> connected systems is given, but needs some revision. It is a given = goal >>> of the project to allow modules in other programming language. >>> Furthermore, it has been discussed if parts of the existing >>> implementation may be replaced with other proven open source = software, >>> e.g. the correlation engine or the web frontend. The other way = round, it >>> has been discussed that the filter creation engine would make a good >>> tool for any kind of structured data, and thus could be separated = from >>> ALOIS and standardized as a stand-alone tool. >>>=20 >>>=20 >>> =3D Background =3D >>>=20 >>> It's not simple to know what happens in a bigger network. There's a >>> multitude of applications, services and appliances working together. >>> Many of them provide some kind of events or state information. The >>> network administrator needs to get hands on all of them. But they = come >>> in many different flavors and multiple canals. Therefore, it's hard = to >>> get the big picture. Furthermore, we have learned that it's = impossible >>> to protect a system against all malicious attacks and to keep all = the >>> possible faulty handling away. A monitoring of the systems to = guarantee >>> a pro-active handling is therefore needed.. >>>=20 >>> Therefore, more and more organizations collect and analyze all = logfiles >>> in a centralized system, called a SIEM (security information and = event >>> management). The technology provides two major functions for = security >>> events from networks, systems and applications: log management and >>> compliance reporting (SIM =96 security information management) and >>> real-time monitoring and incident management (SEM =96 security event >>> management). >>>=20 >>>=20 >>> =3D Rationale =3D >>>=20 >>> Why another security information and event management system? It's = true, >>> there's already plenty of them. While the proprietary software is = way >>> too expensive for smaller to mid-sized companies, we find that the = open >>> source solutions are either too simple or not completely open. For >>> example, behind each of the well known systems =93OSSIM=94 and = =93Prelude=94, >>> there is a company that either closes central functionality for its = own >>> business or has dual licensing and therefore asks the full copyright = for >>> all contributed code. >>>=20 >>> ALOIS is aimed to be totally free and open for all contributions. = The >>> openness provided for other programming languages is certainly proof = of >>> this. The plug-ability - yet to be further developed - is meant to >>> guarantee that individual needs can be realized without stressing = the >>> whole system too much. In our opinion, the Linux kernel is a good >>> example that this can work very well. >>>=20 >>> Since we are in accordance with =84the Apache way=93, we would be = very >>> pleased if ALOIS could become part of the Apache community. In = Addition, >>> the Apache Logging Services would be a perfect home for the = software. >>> Furthermore, it's not the intention to compete with the already = existing >>> log viewer and analyzing tool =84Chainsaw=93. Since Chainsaw is a = relatively >>> easy tool, it meets a rather different need. Nevertheless, if the = two >>> projects use synergies, both can profit. >>>=20 >>>=20 >>> =3D Initial Goals =3D >>>=20 >>> When this project started ins 2005, there was no proven SIEM open = source >>> software and the commercial tools were way too expensive for the = needed >>> environment. Therefore, we decided together with a customer of ours = to >>> implement an open source SIEM tool from scratch. Now the software = has >>> run in a production environment for several years and has proven its >>> functionality and reliabilty. >>>=20 >>>=20 >>> =3D Current Status =3D >>>=20 >>> =3D=3D Meritocracy =3D=3D >>>=20 >>> As already mentioned, ALOIS is already in production use in two >>> organizations. All the code has been written by two persons of the = same >>> company in a paid employment relationship. It is obvious that this = is >>> way different from the open source approach within Apache. But >>> nevertheless, the two developers have always worked as a team and = the >>> decisions were made in consensus whenever possible. But it is no = secret, >>> that these developers have to learn to behave in an open community. >>> Understanding this potential problem, they already got support by a >>> freelance consulter, who has the corresponding experience and = knowledge. >>>=20 >>> =3D=3D Community =3D=3D >>>=20 >>> Until today there is no real community, because the project hasn't = been >>> published officially, although it had been completely published on = the >>> web site for a couple of months (until a server relaunch). Convinced = by >>> the concept and design of the software, we are open and hope to = reach >>> many contributors and users. We think that it is realistic, because = the >>> SIEM issue has yet not been resolved in the OSS space. >>>=20 >>> =3D=3D Core Developers =3D=3D >>>=20 >>> ALOIS was developed by Simon H=FCrliman and Flavio Pellanda, both = employed >>> by the company IMSEC. Concerning Design and Architecture, Marcus >>> Holthaus, owner of IMSEC, gave his input as security specialist. = Since >>> the beginning of this year, Urs Lerch, a doctorate on the subject of >>> commercial open source software development, supports the team with = his >>> knowledge. Simon H=FCrlimann has left the company three years ago, = but is >>> still active in the OSS environment (although not for ALOIS). = Current >>> employee Daniel Lutz (a Debian Developer) has also contributed to = the >>> project. >>>=20 >>> =3D=3D Alignment =3D=3D >>>=20 >>> Besides that we strongly believe in the =84Apache way=93, we think = that >>> although that Apache hosts the Logging Services and different = security >>> projects, there is a gap when it comes to a superordinate security = view. >>> We therefore think it a good idea to add our SIEM project to the = Apache >>> repository. On the other side, Apache would become an even more = complete >>> software repository. >>>=20 >>>=20 >>> =3D Known Risks =3D >>>=20 >>> =3D=3D Orphaned products =3D=3D >>>=20 >>> Since the software is only maintained by employers of one company, = there >>> is a severe risk of being orphaned. But, on the one hand, the = company >>> has a sustained interest in keeping the project alive, because there = are >>> plans to offer services on top of ALOIS, and IMSEC uses the software = for >>> SIEM on their own systems. For this reason there exists a budget for = the >>> development and support of ALOIS. On the other hand, we believe that >>> ALOIS is of great interest for other people and companies tied to IT >>> security. Therefore, our step to the Apache incubator is also a step = to >>> a bigger community. >>>=20 >>> =3D=3D Inexperience with Open Source =3D=3D >>>=20 >>> While ALOIS has always been licenced under the GPL, access to the = source >>> code, bug tracker and version control system has been restricted to >>> internal users for most of the time. But the company has a strong >>> believe in the open source movement and therefore engages its = employees >>> to take part in the community. Furthermore, it is also a strategic >>> decision to build services on top of linux. >>>=20 >>> We understand that the Apache Incubator is a great opportunity for = us to >>> get assistance, when it comes to specific questions on the open = source >>> development. Even more, the company has created a part time position = for >>> the open source community work. >>>=20 >>> =3D=3D Homogenous Developers =3D=3D >>>=20 >>> Although ALOIS has been developed by employees of only one company, >>> there is a thorough openness. The company is designed to stay small = and >>> therefore works with several independent partners. Furthermore, its >>> employees work in geographically different parts of the country. >>> Therefore, it is no new experience for the developers to work in a >>> distributed environment and argue rather than to command. Already = today >>> the employees are enforced to document all face-to-face = communication in >>> the internal wiki. Sketches are photographed and stored in the = project's >>> digital folder. >>>=20 >>> =3D=3D Reliance on Salaried Developers =3D=3D >>>=20 >>> Until today all the development of ALOIS has been made in a paid >>> emplyoment. Therefore we know that this brings a significant danger. >>> Since it is our stated aim to encourage participation and recruit >>> commiters, we hope to eliminate this risk as soon as possible. >>> Furthermore, the employees of IMSEC are all open source enthusiasts = and >>> are in one way or another active in the community. Although we have = no >>> certainty, there is good indication that the current commiters would >>> continue their work on ALOIS, even if they wouldn't be paid for it. >>>=20 >>> =3D=3D Relationships with Other Apache Products =3D=3D >>>=20 >>> The Apache Logging Service would be a perfect home for ALOIS as a >>> centralized logging collection and analyzing tool. Furthermore, we = think >>> that we could share part of the code with the Chainsaw subproject, = since >>> both need similar functionality in the web frontend. Since it is our >>> statet aim to replace our own code with proofen open source = libraries, >>> we are open for any collaboration with other projects. For example, = the >>> replacement of the MySQL with a NoSQL database might be useful for >>> performance reasons; therefore HBase is a good candidate. >>>=20 >>> =3D=3D An Excessive Fascination with the Apache Brand =3D=3D >>>=20 >>> The Apache brand is in fact for its own a very good reason to join = the >>> Incubator. But much more our desire to become part of the Apache >>> Incubator is our strong believe in open source software in general = and >>> in the =84Apache way=93 in particular. We would love to learn from = the >>> experience and knowledge of the foundation's members and = participants, >>> which is an important part of the brand as well. The foundation has >>> shown many times, that it has the processes and people to succeed in >>> launching a project. We would be very proud to be part of this = success >>> story. >>>=20 >>>=20 >>> =3D Documentation =3D >>>=20 >>> The documentation is rather weak and scattered. It has mainly been >>> maintained on a wiki and is open to improvement. Since we are = totally >>> aware that this is a killer for a successfull open source project, = we >>> have already started an internal project with its own budget to = improve >>> this shortcomming. Once the project has been launched, writing a = blog or >>> open a forum are other possibilities we already thought of. >>>=20 >>> Furthermore, as the employees are used to work in a geographycally >>> distributed environment, a lot of the internal communication happens = in >>> a chat. Thus, opening a new chat channel for the community is = scheduled. >>> (To document the discussions for all those who were off-line, we = would >>> send the logs daily to the mailing list.) >>>=20 >>>=20 >>> =3D Initial Source =3D >>>=20 >>> Although the initial source comes from a project for a customer. it = has >>> an open source licence since the beginning. Therefore it doesn't = have >>> any propriatary code in it. A thorough revision before releasing it = to a >>> public repository is recommend and is also in planning. >>>=20 >>> The initial source will be a snapshot of the version control system, >>> accompanied by a related debian package. >>>=20 >>>=20 >>> =3D Source and Intellectual Property Submission Plan =3D >>>=20 >>> ALOIS is currently under a GPL licence. Since there are only two >>> contributors so far, both from the same company, there is no problem = to >>> re-licence the code and contribute it to Apache. The commitment of = the >>> company's owner has been granted. >>>=20 >>>=20 >>> =3D External Dependencies =3D >>>=20 >>> So far, no external dependencies are known. As mentioned before, a >>> thorough revision of the codebase is in planning. There it can be >>> controlled, that no other licence is affected by the code. >>>=20 >>>=20 >>> =3D Cryptography =3D >>>=20 >>> ALOIS does not involve cryptographic code. >>>=20 >>>=20 >>> =3D Required Resources =3D >>>=20 >>> =3D=3D Mailing lists =3D=3D >>>=20 >>> The following mailing lists will be required: >>>=20 >>> * alois-private >>> * alois-dev >>> * alois-commits >>> * alois-users >>>=20 >>> =3D=3D Subversion Directory =3D=3D >>>=20 >>> https://svn.apache.org/repos/asf/incubator/alois >>>=20 >>> =3D=3D Issue Tracking =3D=3D >>>=20 >>> JIRA ALOIS (ALOIS) >>>=20 >>> =3D=3D Other Resources =3D=3D >>>=20 >>> We would like to open a chat channel. If this isn't possible within = the >>> infrastructure of Apache, we would love to do this in our own = already >>> existing infrastructure. >>>=20 >>>=20 >>> =3D Initial Commiters =3D >>>=20 >>> * NAME EMAIL AFFILIATION = CLA >>> * Flavio Pellanda flavio.pellanda at logintas dot ch IMSEC = no >>> * Urs Lerch mail at ulerch dot net IMSEC = no >>> * Daniel Lutz daniel.lutz at logintas dot ch IMSEC = no >>> * Marcus Holthaus marcus.holthaus at imsec dot ch IMSEC = no >>>=20 >>>=20 >>> =3D Sponsors =3D >>>=20 >>> =3D=3D Champion =3D=3D >>>=20 >>> * Scott Deboy >>>=20 >>> =3D=3D Nominated Mentors =3D=3D >>>=20 >>> * Scott Deboy >>>=20 >>> =3D=3D Sponsoring Entity =3D=3D >>>=20 >>> The Incubator PMC (requested) b >>>=20 >>=20 >> Craig L Russell >> Architect, Oracle >> http://db.apache.org/jdo >> 408 276-5638 mailto:Craig.Russell@oracle.com >> P.S. A good JDO? O, Gasp! >>=20 >>=20 >>=20 >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org >> For additional commands, e-mail: general-help@incubator.apache.org >>=20 >>=20 --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org For additional commands, e-mail: general-help@incubator.apache.org