Modern Data Needs

Communication and data is critical to most modern systems. The world is built around the movement of data and being able to do things with that data. More and more components are being added. More and more internal and external systems are being integrated. And more and more data is moving between those components and systems.

As you keep adding systems, processes, and technologies, the task of trying to get them all to talk to each other becomes quite difficult and the teams responsible for making this all work are getting overwhelmed.

Many organizations have two common challenges that they struggle with: communicating between services and moving data within an organization.

As an organization grows in requirements, use cases, and the amount of data it's handling, communication can break down. For example, as more data flows through the system, a component may start to struggle with a spike in traffic or unplanned down time. Likewise, an increased number of components trying to communicate and interoperate with each other can also create challenges. It can become difficult to manage the data and difficult to evolve the system.

Message queuing helps you deal with these issues because it facilitates interactions between systems by exchanging messages asynchronously .

In addition to facilitating the day-to-day communication between services, moving data from one place in your organization to another is critical. On a small scale, moving data around via files might work well, but as the amount of data increases, the systems break down. Companies start to worry about security such as encryption, compliance, and so on.

Data streaming is a solution to this problem because it can efficiently move large amounts of data in and throughout a system.

Next steps

• Learn about Using Pulsar for Messaging.
• Learn about Using Pulsar for Streaming.