Showing posts with label Storm. Show all posts
Showing posts with label Storm. Show all posts

Wednesday, June 26, 2019

Flink vs Storm

Flink supports batch and streaming analytics, in one system. 

Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

Flink has Out-of-the box connector to HDFS, S3 and kinesis

Wednesday, April 10, 2019

Apache Storm main concepts

Apache Storm main concepts

Topology
Stream
Spout
Bolt
Stream groupings
Reliability
Tasks
Workers

Topology :
A topology is a graph of spouts and bolts that are connected with stream groupings.  A Storm topology is analogous to a MapReduce job. One key difference is that a MapReduce job eventually finishes, whereas a topology runs forever (or until you kill it, of course).

Stream :
A stream is an unbounded sequence of tuples that is processed and created in parallel in a distributed fashion. The stream is the core abstraction in Storm. Streams are defined with a schema that names the fields in the stream's tuples.

Spout :
A spout is a source of streams in a topology. Generally spouts will read tuples from an external source (e.g. Kafka) and emit them into the topology.
The main method on spouts is nextTuple. nextTuple either emits a new tuple into the topology or simply returns if there are no new tuples to emit.
IRichSpout: this is the interface that spouts must implement.

Bolt :
All processing in topologies is done in bolt. Bolt can do anything from filtering, functions, aggregations, joins, talking to databases, and more. Bolt can do simple stream transformations. Doing complex stream transformations often requires multiple steps and thus multiple bolts.
The main method in bolt is the execute method which takes in as input a new tuple. Bolt emit new tuples using the OutputCollector object. Its perfectly fine to launch new threads in bolts that do processing asynchronously. OutputCollector is thread-safe and can be called at any time.

Ref : http://storm.apache.org/releases/2.0.0-SNAPSHOT/Concepts.html

Sunday, February 26, 2017

Apache Storm Overview

Apache Storm is a free and open source distributed realtime computation systemStorm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more

Architecture of Apache Storm



Storm Components 

Topology :

In simple words, Topology is a network of spouts and bolts as in above figure. It is analogous to a MR Job in Hadoop. It is a graph of computation consisting of spouts and bolts. Spouts as data stream source tasks and Bolts as actual processing tasks.

Spout :
Spout is the entry point in a storm topology. It is the source of streams in the topology. A spout connects to the actual data source such as a message queue as Kafka , gets continuous data , converts the actual data into stream of tuples, emits them to bolts for actual processing. 

Bolt :Bolt contains the actual processing logic. It works only on streams and can emit streams too for further processing downstream by other bolts or can export/save data for persistent storage. It receives stream from either one or more spouts or some other bolts. Bolts can do anything from run functions, filter tuples, do streaming aggregations, do streaming joins, talk to databases, and more.