In this tutorial, we will guide you through integrating Apache Kafka with PHP without using Docker on an Ubuntu server. Kafka is a powerful distributed streaming platform that enables real-time data streaming and processing. We will implement a Kafka producer and consumer using the latest PHP version, showcasing how to send and receive messages between different parts of an application.
This tutorial will cover installing Kafka and PHP on an Ubuntu server and writing a Kafka producer and consumer in PHP using the modern syntax of PHP 8.3
Prerequisites
To follow this tutorial, ensure you have:
- Ubuntu 20.04+
- PHP 8.3
- Apache Kafka
- librdkafka (Kafka’s client library for PHP)
- php-rdkafka (PHP extension for Kafka)
Step 1: Installing Apache Kafka on Ubuntu
First, we need to install Kafka and its dependencies on the Ubuntu server. This includes setting up Java, Zookeeper, and Kafka.
1.1 Install Java
Kafka requires Java to run. Install the default Java package by running:
sudo apt update
sudo apt install default-jdk -y
Verify the Java installation:
java -version
1.2 Download and Install Kafka
Next, download Kafka from the official Apache website. First, navigate to the /opt directory, where we will install Kafka:
cd /opt
sudo wget https://downloads.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
Extract the downloaded Kafka tar file:
sudo tar -xvzf kafka_2.13-3.0.0.tgz
sudo mv kafka_2.13-3.0.0 kafka
1.3 Start Zookeeper and Kafka
Kafka relies on Zookeeper to manage its brokers. Start the Zookeeper service:
cd /opt/kafka
sudo bin/zookeeper-server-start.sh config/zookeeper.properties
Open a new terminal window or SSH session to start the Kafka broker:
cd /opt/kafka
sudo bin/kafka-server-start.sh config/server.properties
Kafka should now be running on your Ubuntu server.
Step 2: Installing PHP and Kafka Extension on Ubuntu
2.1 Install PHP 8.3
If PHP 8.3 is not installed on your Ubuntu server, add the repository and install it:
sudo apt update
sudo apt install software-properties-common -y
sudo add-apt-repository ppa:ondrej/php -y
sudo apt update
sudo apt install php8.3 php8.3-cli php8.3-fpm php8.3-mbstring php8.3-xml -y
Verify PHP installation:
php -v
2.2 Install librdkafka and php-rdkafka
To integrate Kafka with PHP, install librdkafka (the Kafka client library) and the php-rdkafka extension:
sudo apt install librdkafka-dev -y
sudo pecl install rdkafka
Enable the Kafka extension by adding the following line to your php.ini:
extension=rdkafka.so
Restart your PHP service to apply the changes:
sudo systemctl restart php8.3-fpm
Step 3: Implementing Kafka Producer in PHP
Now that we have Kafka and PHP set up, we can create a Kafka producer to send messages to a Kafka topic.
Kafka Producer Class
Here’s a PHP class for a Kafka producer that uses the modern syntax of PHP 8.3+.
<?php
declare(strict_types=1);
class KafkaProducer
{
private \RdKafka\Producer $producer;
private string $topic;
public function __construct(string $broker, string $topic)
{
$this->producer = new \RdKafka\Producer();
$this->producer->addBrokers($broker);
$this->topic = $topic;
}
public function produce(string $message): void
{
$topic = $this->producer->newTopic($this->topic);
$topic->produce(RD_KAFKA_PARTITION_UA, 0, $message);
$this->producer->poll(0);
// Ensure the message is delivered
while ($this->producer->getOutQLen() > 0) {
$this->producer->poll(50);
}
echo "Message produced: $message\n";
}
}
// Usage Example
$producer = new KafkaProducer('localhost:9092', 'my_topic');
$producer->produce('Hello from PHP Kafka Producer!');
Breakdown of Code:
- Producer Initialization: The constructor initializes the Kafka producer and connects to the Kafka broker (
localhost:9092). - Produce Method: The
produce()method pushes a message to the specified Kafka topic. It also ensures the message is properly delivered by polling the producer queue.
Step 4: Implementing Kafka Consumer in PHP
A Kafka consumer reads messages from Kafka topics and processes them.
Kafka Consumer Class
Here’s how you can implement a Kafka consumer class in PHP:
<?php
declare(strict_types=1);
class KafkaConsumer
{
private \RdKafka\KafkaConsumer $consumer;
public function __construct(string $broker, string $groupId, string $topic)
{
$conf = new \RdKafka\Conf();
$conf->set('group.id', $groupId);
$conf->set('metadata.broker.list', $broker);
// Disable auto-commit to manually handle offset commits
$conf->set('enable.auto.commit', 'false');
$this->consumer = new \RdKafka\KafkaConsumer($conf);
$this->consumer->subscribe([$topic]);
}
public function consume(int $timeoutMs): ?string
{
$message = $this->consumer->consume($timeoutMs);
switch ($message->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
// Commit the message manually
$this->consumer->commit();
echo "Received message: {$message->payload}\n";
return $message->payload;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
echo "Consumer timed out\n";
return null;
default:
echo "Error: {$message->errstr()}\n";
return null;
}
}
}
// Usage Example
$consumer = new KafkaConsumer('localhost:9092', 'my_group', 'my_topic');
while (true) {
$consumer->consume(1000); // Wait up to 1 second for a message
}
Breakdown of Code:
- Consumer Initialization: We initialize the consumer with the broker address and group ID. The consumer subscribes to the specified Kafka topic.
- Manual Offset Commit: The
consume()method waits for messages, processes them, and manually commits the offset after successful consumption.
Step 5: Producing and Consuming Messages
Now that we’ve implemented both the producer and consumer, let’s produce a message and consume it.
-
Run the Producer:
$producer = new KafkaProducer('localhost:9092', 'my_topic'); $producer->produce('Test message from PHP!'); - Run the Consumer:
$consumer = new KafkaConsumer('localhost:9092', 'my_group', 'my_topic'); while (true) { $consumer->consume(1000); }
The consumer should successfully receive the message sent by the producer. The output will display the message payload in the terminal.
Step 6: Handling Errors and Retry Logic
In a production environment, you will need to handle various failure scenarios, such as network issues, message delivery failures, or Kafka broker downtime. Here’s an example of how to handle exceptions in the producer:
public function produce(string $message): void
{
try {
$topic = $this->producer->newTopic($this->topic);
$topic->produce(RD_KAFKA_PARTITION_UA, 0, $message);
$this->producer->poll(0);
while ($this->producer->getOutQLen() > 0) {
$this->producer->poll(50);
}
echo "Message produced: $message\n";
} catch (\Exception $e) {
echo "Failed to produce message: {$e->getMessage()}\n";
}
}
Conclusion
In this tutorial, we demonstrated how to integrate Kafka with PHP on an Ubuntu server without using Docker. By setting up Kafka, installing the necessary libraries, and implementing a Kafka producer and consumer, you can now build powerful, real-time applications with PHP.
Summary:
- Installed Kafka on Ubuntu: Set up Java, Zookeeper, and Kafka.
- Installed PHP and the Kafka Extension: Used
librdkafkaandphp-rdkafkafor PHP-Kafka integration. - Implemented a Kafka Producer: Sent messages to a Kafka topic using PHP.
- Implemented a Kafka Consumer: Consumed messages from Kafka topics.
With this setup, you can further extend your application, implement additional Kafka topics, handle structured data like JSON, and improve message handling and error recovery for production environments.