> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.

> The File source connector pulls messages from files in directories and persists the messages to Pulsar topics.

# File source

The File source connector pulls messages from files in directories and persists the messages to Pulsar topics.

# Configuration

The configuration of the File source connector has the following properties.

## Property

| Name | Type | Required | Default | Description |
| - | - | - | - | - |
| `inputDirectory` | String | true | No default value | The input directory to pull files. |
| `recurse` | Boolean | false | true | Whether to pull files from subdirectory or not. |
| `keepFile` | Boolean | false | false | If set to true, the file is not deleted after it is processed, which means the file can be picked up continually. |
| `fileFilter` | String | false | \[^\\.].\* | The file whose name matches the given regular expression is picked up. |
| `pathFilter` | String | false | NULL | If `recurse` is set to true, the subdirectory whose path matches the given regular expression is scanned. |
| `minimumFileAge` | Integer | false | 0 | The minimum age that a file can be processed. <br /><br />Any file younger than `minimumFileAge` (according to the last modification date) is ignored. |
| `maximumFileAge` | Long | false | Long.MAX\_VALUE | The maximum age that a file can be processed. <br /><br />Any file older than `maximumFileAge` (according to last modification date) is ignored. |
| `minimumSize` | Integer | false | 1 | The minimum size (in bytes) that a file can be processed. |
| `maximumSize` | Double | false | Double.MAX\_VALUE | The maximum size (in bytes) that a file can be processed. |
| `ignoreHiddenFiles` | Boolean | false | true | Whether the hidden files should be ignored or not. |
| `pollingInterval` | Long | false | 10000L | Indicates how long to wait before performing a directory listing. |
| `numWorkers` | Integer | false | 1 | The number of worker threads that process files.<br /><br /> This allows you to process a larger number of files concurrently. <br /><br />However, setting this to a value greater than 1 makes the data from multiple files mixed in the target topic. |

## Example

Before using the File source connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "inputDirectory": "/Users/david",
      "recurse": true,
      "keepFile": true,
      "fileFilter": "[^\\.].*",
      "pathFilter": "*",
      "minimumFileAge": 0,
      "maximumFileAge": 9999999999,
      "minimumSize": 1,
      "maximumSize": 5000000,
      "ignoreHiddenFiles": true,
      "pollingInterval": 5000,
      "numWorkers": 1
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      inputDirectory: "/Users/david"
      recurse: true
      keepFile: true
      fileFilter: "[^\\.].*"
      pathFilter: "*"
      minimumFileAge: 0
      maximumFileAge: 9999999999
      minimumSize: 1
      maximumSize: 5000000
      ignoreHiddenFiles: true
      pollingInterval: 5000
      numWorkers: 1
  ```

# Usage

Here is an example of using the File source connecter.

1. Pull a Pulsar image.

   ```bash theme={null}
   $ docker pull apachepulsar/pulsar:{version}
   ```

2. Start Pulsar standalone.

   ```bash theme={null}
   $ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-standalone apachepulsar/pulsar:{version} bin/pulsar standalone
   ```

3. Create a configuration file *file-connector.yaml*.

   ```yaml theme={null}
   configs:
       inputDirectory: "/opt"
   ```

4. Copy the configuration file *file-connector.yaml* to the container.

   ```bash theme={null}
   $ docker cp connectors/file-connector.yaml pulsar-standalone:/pulsar/
   ```

5. Download the File source connector.

   ```bash theme={null}
   $ curl -O https://mirrors.tuna.tsinghua.edu.cn/apache/pulsar/pulsar-{version}/connectors/pulsar-io-file-{version}.nar
   ```

6. Start the File source connector.

   ```bash theme={null}
   $ docker exec -it pulsar-standalone /bin/bash

   $ ./bin/pulsar-admin sources localrun \
   --archive /pulsar/pulsar-io-file-{version}.nar \
   --name file-test \
   --destination-topic-name  pulsar-file-test \
   --source-config-file /pulsar/file-connector.yaml
   ```

7. Start a consumer.

   ```bash theme={null}
   ./bin/pulsar-client consume -s file-test -n 0 pulsar-file-test
   ```

8. Write the message to the file *test.txt*.

   ```bash theme={null}
   echo "hello world!" > /opt/test.txt
   ```

   The following information appears on the consumer terminal window.

   ```bash theme={null}
   ----- got message -----
   hello world!
   ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.