Pivotal GemFire

Take Advantage of Opportunities Now

Real time means taking advantage of an opportunity at the appropriate time and in the right context. Acting in the right moment can give your business the greatest sustainable competitive advantage. Failing to act on data in real time can lead to lost sales, inventory oversupply and missed micro windows of opportunity. Today your enterprise takes in a huge amount of data from varied sources, but how do you process it immediately to help you make critical business decisions—especially when the relevant lifetime of that data is shrinking? Pivotal GemFire® is a real-time distributed data store that solves the hard problems of distributed systems for you. It provides a familiar Java interface for interacting with data of arbitrary complexity and shape, and accommodates ever-growing data sets and users with linear scalability, continuous uptime and predictable performance. With GemFire, you can have data consistency in an elastic environment built for the cloud. You can access information orders of magnitude faster. You can correlate data streams with enterprise data stores—creating competitive advantage where it counts for your organization.

Pivotal customers using GemFire have:

  • Reduced risk analysis time from 6 hours to 20 minutes, allowing for record profits in the flash crash of 2008 that other firms were not able to monetize.
  • Improved end-user response time from 3 seconds to 50 ms, worth 8 figures a year in new revenue from a project delivered in fewer than 6 months.
  • Tracked assets in real time to coordinate all the right persons and machinery into the right place at the right time to take advantage of immediate high-value opportunities.
  • Created end-user reservation systems that handle over a billion requests daily with no downtime.

PIVOTAL GemFire Features

Solve the Data Challenges Presented by Modern, Distributed Web-Oriented Applications

Pivotal GemFire is the premier in-memory data platform delivering speed and dynamic scalability, together with the reliability and data management capabilities of a traditional database. Fast and flexible, GemFire enables you to process massive quantities of data and scale within the cloud to suit the needs of users, while providing tools to simplify monitoring and management. The GemFire solution features:

  • Data Consistency with Cloud Scalability – GemFire solves the problem of moving data into the cloud by implementing a shared-nothing architecture. The only shared resource in the cluster is the network. Adding nodes to the cluster adds capacity for more users and data. The application tier does not need to be aware of how data is being made consistent, how big the cluster is, how many copies of data are available, or what the current cluster membership is. Typically an issue of complex maintenance, scripts and custom code, this logic is all built into GemFire. The solution can be configured for a high level of data availability by using redundancy settings. The logic used to update data ensures that all copies of the data in the cluster are consistent at any given point. And data can be sent to remote clusters via the WAN Gateway in case of catastrophic failures. GemFire can also be used as a reliable Operational Data Store, leveraging archival systems for OLAP and offloading of data when it is no longer relevant.
  • Extreme Performance and Continuous Uptime with Predictable Performance – GemFire provides for continuous uptime with built in high availability and disaster recovery. Multiple failure detection models detect and react to failures quickly, ensuring that the cluster is always available, and that the data set is always complete. During hardware refreshes and major GemFire upgrades, GemFire’s cloud elasticity allows you to add nodes to the cluster without impacting the clients. During data model changes, GemFire uses PDX Serialization to enable different nodes and clients in a cluster to have different data model versions running at the same time. This means that new data models can be deployed with zero impact to client applications. During server-logic code changes, GemFire provides hot redeployment of any user logic classes via our management tool, GemFire SHell (gfsh).
  • Data Aware Parallel Function Execution – By associating a function with a data set, GemFire will route function calls to the relevant nodes on the cluster without interference from the caller. For large data sets, this means dramatically reducing the total time to run complex calculations over traditional approaches. As the GemFire cluster expands and contracts, the calling application does not need to be aware of the cluster membership or where the data resides. Application developers can use a familiar HashMap interface to touch the data directly, or an easy-to-use function API to interact with the data set. If a failure occurs in the cluster during function execution, the function can be configured to be aware of this and restart the execution.
  • Data Stream with Enterprise Data Store Correlation – GemFire can receive data from any application that is able to call a C++, C#, Java, or REST interface. Conversely, GemFire can call into anything that is available via a Java API, such as JDBC or web services. Along with a powerful query interface, data is readily accessible. This flexible API, along with the ability to ingest data from external sources and send out event notifications to other systems, gives you the ability to correlate data from multiple sources into real time information that is important for your enterprise.

PIVOTAL GemFire Technology

What is GemFire?

GemFire is a distributed in-memory data grid for OLTP applications. It is backed by a robust, shared-nothing persistence architecture that guarantees the performance of in-memory and the reliability of disk. By combining data redundancy, GemFire provides the best-of-breed, high availability for data. It is a transformational piece of technology that was purpose-built to support large volumes of data with a very high degree of data consistency. Data is stored as key-value pairs or as JSON documents in a partitioned/replicated data grid. Every feature of the product is designed to minimize latency, allowing the development of highly responsive applications that never have to be redesigned for scaling.

Familiar Java HashMap API

Data sets in GemFire are internally held in a data structure called a region, which implements the java.util.HashMap interface. Developers familiar with Java will quickly understand GemFire.

Linearly Scalable

GemFire makes optimal use of system resources like CPU, memory, network and disk to ensure that if these resources are doubled, the overall system throughput can also double, without any visible increase in latency for end-user applications. It does this by intelligently managing the placement of data while reducing network round trips. Data gets replicated only to those nodes that need the data, and requests for access are routed intelligently using the most direct path available.

Elastic at Cloud Scale

GemFire’s resource usage is designed to be elastic. If you add more resources, it increases capacity and balances the use of resources across the network. If you take away resources, it responds to the decrease in resources without impacting end-user applications, as long as the remaining elements of the cluster can handle the load. The ability to add and remove nodes in response to changing load conditions makes GemFire cloud ready for private and public cloud deployments.

Highly Available with No Maintenance Windows Required, Even for Upgrade

GemFire runs as a cluster of processes on commodity hardware running on regular TCP/IP networks. In this environment, it takes extraordinary measures to protect data in the face of process failures, machine failures, rack failures and even data center outages. GemFire’s cluster management and resource utilization algorithms use a combination of retries, disk persistence, redundancy zones and network partitioning management techniques to ensure data consistency in a highly available system. GemFire supports the notion of rolling upgrades, which ensures your GemFire administrators do not have to plan for scheduled maintenance windows to upgrade the product or any application code that runs with GemFire.

PIVOTAL GemFire Technology

What Makes GemFire Easy to Use for Developers?

The following capabilities make GemFire an easy-to-use solution for developers:

Deep and Rich Spring Data Integration

As a fully supported Spring project within Pivotal, Spring Data GemFire allows developers to architect applications so that data access and business logic are separated from configuration and operation code. In addition, GemFire APIs and related code samples help make developers productive quickly. Spring Data support in GemFire is enhanced and updated with each product release which allows Spring users to bootstrap applications using very familiar programming techniques to exercise the powerful set of capabilities exposed by GemFire.

Support for Memcached Clients

GemFire servers can be started up as memcached servers with full support for the memcached API. Gemcached extends the reach of GemFire to any memcached client allowing end users an easy way to get started with GemFire.

L2 Hibernate Support and Scalable Session State Management Support

Applications store increasing amounts of data in user sessions and more importantly, user session expiration may be in the order of days, if not months. The ability to store session-state information in a data grid, support multiple topologies for scale (thousands of app servers that can access any session from anywhere), transmit deltas when user sessions are updated, and support for popular application servers such as tcServer, JBoss, WebLogic and WebSphere are common use cases supported by GemFire.

PIVOTAL GemFire Technology

What Legacy Integration Capabilities Does GemFire Have?

GemFire includes the following legacy integration capabilities:

Support for Multiple Deployment Topologies

GemFire supports multiple deployment topologies to provide highly customized application deployments based on their latency, throughput, scale and redundancy requirements. GemFire applications can be deployed as a peer-to-peer cluster (supporting hundreds of nodes), in a client/server cluster (supporting tens of thousands of clients) or in multiple clusters daisy-chained using GemFire’s WAN replication capabilities.

Reliable Publish-Subscribe Framework Allowing Integration with the Rest of the Enterprise

GemFire provides an “eventing” framework that allows end-user applications to receive notifications when data-change events happen in the system. This includes the ability for GemFire clients to configure very customized subscriptions for data change notifications, which is supported by a highly reliable publish-subscribe notification mechanism. These subscriptions can be set up as regular expressions on keys, key sets, or continuous queries using an SQL-like querying language. Event callbacks can be programmed to integrate data coming from GemFire with the rest of the enterprise (including backend databases and exchanges). Subscription management and event distribution is highly optimized to avoid wasting CPU and network cycles and to ensure that events are delivered to the right set of consumers.

Support for Native Clients in Java, C++ and .NET

GemFire is one of the few data grid providers that supports a rich set of APIs and native implementations for those APIs in Java, C++ and .NET. The clients are 64-bit enabled and support full client-side caching with eventual consistency for the cached data which allows you to build applications that are powerful, respond in real time, and can manage large volumes of data without ever going to disk.

Backward Compatibility Support

GemFire clients have been deployed in kiosks—on vendor sites where the customer has no control over the upgrade schedule. As a result, GemFire allows older clients to continue operating with newer servers. Support for rolling upgrades in the cluster ensures that peers from two versions can support data access during an upgrade process.

Support for Reliable Asynchronous Event Queues

GemFire supports callbacks for read-through and write-through operations allowing applications to integrate with legacy data sources. However, when GemFire is being used as the operational data store, it makes sense to use the Async Event Queueing mechanism to update traditional databases using batch operations. As with other aspects of GemFire, high availability is built into the Async Event Queues ensuring data consistency between GemFire and the traditional database being updated.

PIVOTAL GemFire Technology

Which Enterprise-Readiness Capabilities Does GemFire Support?

Support for Global WAN Replication

GemFire provides a globally consistent view of data using highly fault-tolerant asynchronous WAN replication capabilities that are designed to work on high-latency congested networks on the open internet. Using techniques like batch replication, tuned sliding windows, support for conflict detection and failover for both sender and receiver, GemFire WAN replication helps you synchronize data across the globe, ensuring that your users can be served even if a data center goes down for any reason.

Support for Backward- and Forward-Compatible Serialization Protocol

Applications evolve their data models at different rates. When a new application introduces additional attributes to an existing data entity, GemFire’s backward- and forward-compatible serialization protocol ensures that older and newer applications can continue to exchange information in native binary format without having knowledge of the newer attributes to the entity while preserving them in the system.

Scalable Management and Monitoring Framework

GemFire collects statistics within each GemFire server through a highly optimized sampling algorithm, which provides insight into what is happening within each server. GemFire aggregates these statistics within a managing member and makes them available to management and monitoring tools via JMX MBeans. The public statistics API allows your users to plug in their own statistics within application callbacks to see what is happening across the system.

Scalable Behavior Execution Support

GemFire allows your users to write functions in Java that get executed in proximity to the data that the function will use. This is akin to writing distributed stored procedures in Java that get executed in parallel. The function-execution framework in GemFire is generic enough and supports multiple execution models including MapReduce-like executions across the grid.

Support for Transactions, Querying and Indexing

GemFire stores data in tables that are referred to as regions, and there can be any number of regions in the system that store data either as partitioned or replicated. Regions can be co-located, allowing for optimized scalable transactions across co-located data in the cluster. GemFire supports Object Querying Language (OQL), which offers support for indexing and SQL-like querying capabilities over complex object-graph collections that are stored in regions. Some of the capabilities provided to JSON documents include the ability to query within and across JSON documents, federated queries with support for joins across JSON, and key-value regions/tables and transactional support.

Online Monitoring Dashboards and Command Line Utilities

GemFire includes a web-based monitoring dashboard, called Pulse, that provides a birds-eye view into the cluster with relevant summary statistics. The operator can use it to drill down into individual members, look at individual data sets, or dive into the performance characteristics of individual aspects of product behavior that are depicted via Pulse.

Powerful Offline Analysis Capabilities

GemFire supports an offline analysis tool called Visual Statistics Display (VSD). VSD opens stat files created by GemFire to provide powerful correlation capabilities across events and across members. VSD is useful in pinpointing performance or resource bottlenecks in the distributed system that otherwise would be hard to diagnose.

Sophisticated Proactive Resource Management Capabilities

GemFire actively manages CPU, disk, memory and network resources. Using auto-tuning techniques, GemFire ensures that applications have the access to the right quantities of resources when executing both read and write workloads. The distributed resource manager detects critical memory conditions in the server and moves to prevent operations that can destabilize the container while working to remedy the excessive use of memory in the server.

RELATED RESOURCES

News and events, blog posts, videos, case studies, whitepapers, and other related resources.

SEE ALL RESOURCES

In 2010, Southwest.com’s e-commerce engine was responsible for over $8.5 billi...


Blog Post | Jul 16, 2014

Achieving significant ROI with improved performance and scalability


Case Study |

The biggest annual movement of humans on the planet happens around the Chinese New Year


Blog Post | Jan 23, 2014

At GE Power & Water’s Thermal Remote Monitoring and Diagnostics (RM&D) center


Blog Post | Jan 22, 2014

Scaling online sales for the largest railway in the world


Case Study |

Enabling real-time transactions and analysis


Case Study |

Major Financial Institution strengthens competitiveness with speed and scalability


Case Study |

Contact Pivotal
Pivotal Support