Configure Gradle/Maven for Compressed Parquet Support

Introduction

For supporting compressed Parquet files in your project, you need to make some updates to your Gradle or Maven files. This guide will walk you through the necessary changes.

Steps to Update Your Gradle File

1. Add Dependencies

Include the following lines in the dependencies section of your `build.gradle` file:

implementation 'com.hadoop.gplcompression:hadoop-lzo:0.4.20'
implementation 'com.github.rdblue:brotli-codec:0.1.1'
    

These libraries provide support for LZO and Brotli compression algorithms, which are essential for handling compressed Parquet files.

2. Add Maven Repositories

Include the following Maven repositories in your Gradle file:

maven {
    url = 'https://maven.twttr.com/'
}
maven {
    url = 'https://jitpack.io'
}
    

These repositories host the necessary libraries for our project.

For full example check build.gradle.

Steps to Update Your Maven File

1. Add Dependencies

Include the following dependencies in your Maven project's `pom.xml`:

<dependencies>
    <dependency>
        <groupId>com.hadoop.gplcompression</groupId>
        <artifactId>hadoop-lzo</artifactId>
        <version>0.4.20</version>
    </dependency>
    <dependency>
        <groupId>com.github.rdblue</groupId>
        <artifactId>brotli-codec</artifactId>
        <version>0.1.1</version>
    </dependency>
</dependencies>
    

These dependencies provide the necessary support for LZO and Brotli compression algorithms.

2. Configure Maven Repositories

Add the following repositories to your `pom.xml`:

<repositories>
    <repository>
        <id>twttr</id>
        <url>https://maven.twttr.com/</url>
    </repository>
    <repository>
        <id>jitpack</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>
    

These repositories host the required libraries for your Maven project.