The Apache Ecosystem for Enterprise Applications

Apache Accumulo is based on the design of Google’s BigTable and is powered by Apache Hadoop, Apache Zookeeper, and Apache Thrift.

Ambari enables System Administrators to provision, Manage, and monitor a Hadoop Cluster. Ambari provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs.

Arrow enables execution engines to take advantage of the latest SIMD (Single Input Multiple Data) operations included in modern processors, for native vectorized optimization of analytical data processing. Columnar layout of data also allows for a better use of CPU caches by placing all data relevant to a column operation in as compact of a format as possible.

AsterixDB supports various storage and indexing options, including managed datasets, external datasets (e.g., data on HDFS – Hadoop File System), and secondary indexes.

Aurora runs applications and services across a shared pool of machines, and is responsible for keeping them running, forever. When machines experience failure, Aurora intelligently reschedules those jobs onto healthy machines.

Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation can be done for optimization purpose, but only worth implementing for statically typed languages.

Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.

When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present.

When Avro is used in RPC, the client and server exchange schemas in the connection handshake. This can be optimized so that, for most calls, no schemas are actually transmitted. Since both client and server both have the other’s full schema, correspondence between same named fields, missing fields, extra fields, etc. can all be easily resolved.

It contains many of the pieces that comprise a typical database management system, but omits some key functions: storage of data, algorithms to process data, and a repository for storing metadata.

Calcite intentionally stays out of the business of storing and processing data. This makes an excellent candidate for mediating between applications and one (or more) data storage locations and data processing engines.

Cassandra’s data model offers the convenience of column indexes with the performance of log-structured updates, strong support for denormalization and materialized views, and powerful built-in caching.

Cayenne is designed to be easy to use, without sacrificing flexibility or design. To that end, Cayenne supports database reverse engineering and generation, as well as a Velocity-based class generation engine.

All of these functions can be controlled directly through a GUI tool called CayenneModeler. No cryptic XML or annotation based configuration is required! An entire database schema can be mapped directly to Java objects within minutes, all from the comfort of the GUI-based CayenneModeler.

Cayenne supports numerous other features, including caching, a complete object query syntax, relationship pre-fetching, on-demand object and relationship faulting, object inheritance, database auto-detection, and generic persisted objects. Most importantly, Cayenne can scale up or down to virtually any project size. With a mature, 100% open source framework, an energetic user community, and a track record of solid performance in high-volume environments, Cayenne is an exceptional choice for persistence services.

CloudStack is a turnkey solution that includes the entire “stack” of features most organizations want with an IaaS cloud: compute orchestration, Network-as-a-Service, user and account management, a full and open native API, resource accounting, and a first-class User Interface (UI).

CloudStack currently supports the most popular hypervisors: VMware, KVM, Citrix XenServer, Xen Cloud Platform (XCP), Oracle VM server, and Microsoft Hyper-V.

Users can manage their cloud with an easy to use Web interface, command line tools, and/or a full-featured RESTful API. In addition, CloudStack provides an API that’s compatible with AWS EC2 and S3 for organizations that wish to deploy hybrid clouds.

Running on top of Hadoop MapReduce and Apache Spark, the Apache Crunch library is a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce. The APIs are especially useful when processing data that does not fit naturally into relational model, such as time series, serialized object formats like protocol buffers or Avro records, and HBase rows and columns.

DdlUtils is a small, easy-to-use component for working with Database Definition (DDL) files. These are XML files that contain the definition of a database schema, e.g. tables and columns.

Apache Derby™ is an relational database implemented entirely in Java and available. Some key advantages include: small footprint – about 2.6 megabytes for the base engine and embedded JDBC driver, is based on the Java, JDBC, and SQL standards, provides an embedded JDBC driver that lets you embed Derby in any Java-based solution. Derby is derived from Informix Cloudscape.

Apache JDO™ is a standard way to access persistent data in databases, using plain old Java objects (POJO) to represent persistent data.

Apache Torque is an object-relational mapper (ORM) for Java. It lets you access and manipulate data in a relational database using Java objects, without using reflection to access user-provided classes, but by generating the necessary classes (including the Data Objects) from an XML schema describing the database layout.

DeltaSpike consists of a core module and a number of optional modules for providing additional enterprise functionality to your applications. The modules include features for enhanced security with type-safe control over method invocations, integration with schedulers, injection of CDI objects into validators, a transactional context and scope, and much more.

Drill allows to get faster insights without the overhead (data loading, schema creation and maintenance, transformations, etc.)
Flexibility, analyze the multi-structured and nested data in non-relational datastores directly without transforming or restricting the data; and leverage your existing SQL skillsets and BI tools including Tableau, Qlikview, MicroStrategy, Spotfire, Excel, and more.

It is composed of AutoConf Resource Processor, Bundle Repository, Configuration Admin, Connect, Coordinator, Dependency Manager 4, Deployment Admin, Event Admin, File Install, Framework Security, Framework, Gogo Command, Gogo Runtime, Gogo Shell, HTTP Service API, HTTP Service Base, HTTP Service Bridge, HTTP Service Bundle, HTTP Service Cometd, HTTP Service Jetty, HTTP Service Proxy, HTTP Service SSL filter, HTTP Service Whiteboard, HTTP Servlet 2.6 + 3.0 API, Inventory Printer, iPOJO (Core), iPOJO Annotations, iPOJO API, iPOJO Architecture Command for Gogo, iPOJO Architecture Command for the Felix Shell, iPOJO BND Plugin, iPOJO Composite, iPOJO Event Admin handler, iPOJO Extender pattern handler, iPOJO JMX handler, iPOJO Manipulator Project (containing all iPOJO manipulator modules), iPOJO Manipulator, iPOJO Runtime Project (containing all iPOJO runtime modules), iPOJO Temporal Dependency handler, iPOJO WebConsole Plugin, iPOJO Whiteboard pattern handler, Jaas Support Boot Classpath Jar, Jaas Support Bundle, junit4osgi – shell command, junit4osgi – swing gui, junit4osgi, Lightweight HTTP Service Complete, Lightweight HTTP Service Core, Log, Main, Metatype, OSGi OBR service API, Preferences, Remote Shell, Resolver, SCR (Declarative Services), SCR Annotations, SCR Bnd Plugin, SCR Compat (Declarative Services), SCR DS Annotations, SCR Ext Anno, SCR Generator, Shell Text UI, Shell, UPnP Base Driver (JDK 1.3), UPnP Base Driver (JDK 1.4+), UPnP Extra, UPnP Tester, User Admin, User Admin File-based store, User Admin MongoDB-based store, Utils, Web Console, Web Console (all-in-one bundle), Web Console DS Plugin, Web Console Event Admin Plugin, Web Console Memory Usage Plugin, Web Console OBR Plugin, Web Console PackageAdmin Plugin, Web Console Script Console Plugin, Web Console ServiceDiagnostics Plugin, Web Console Subsystems Plugin, Web Console UPNP Plugin, and Web Console User Admin Plugin.

Each module has it’s own version and might be more recent than the core module.

Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.

Streamlined Architecture – Kudu provides a combination of fast inserts/updates and efficient columnar scans to enable multiple real-time analytic workloads across a single storage layer. As a new complement to HDFS and Apache HBase, Kudu gives architects the flexibility to address a wider variety of use cases without exotic workarounds.

Faster Analytics – Kudu is specifically designed for use cases that require fast analytics on fast (rapidly changing) data. Engineered to take advantage of next-generation hardware and in-memory processing, Kudu lowers query latency significantly for Apache Impala (incubating) and Apache Spark (initially, with other execution engines to come).

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments.

It is defined as “Apache Myriad enables the co-existence of Apache Hadoop and Apache Mesos on the same physical infrastructure. By running Hadoop YARN as a Mesos framework, YARN applications and Mesos frameworks can run side-by-side, dynamically sharing cluster resources.”

NiFi was built to automate the flow of data between systems. This kind of problem has been around since enterprises had more than one system, where some of the systems created data and some of the systems consumed data. NiFi aims at simplifying Enterprise Integration Patterns.

NiFi includes a web-based user interface, it is highly configurable, tracks data flows from beginning to end (aka Data Provenance). It is designed for extension.

Security is built in, with support for SSL, SSH, HTTPS, encrypted content, and pluggable role-based authentication/authorization.

OLE2 files include most Microsoft Office files such as XLS (Excel), DOC (Word), and PPT (PowerPoint) as well as MFC serialization API based file formats. The project provides APIs for the OLE2 Filesystem (POIFS) and OLE2 Document Properties (HPSF).

Office OpenXML Format is the new standards based XML file format found in Microsoft Office 2007 and later. This includes XLSX, DOCX and PPTX. POI provides a low level API to support the Open Packaging Conventions.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services can lead to management complexity when the applications are deployed.