Installation

How to run

Note: If you are using Windows, please install Spark and set environment variables as mentioned here before going through the installation below. Specifically, you'll need to have the environment variables set, as given here.

• Clone the repository (with all the submodules):

  $ git clone --recurse-submodules https://github.com/RADAR-base/radar-pipeline.git

• Change the directory to radar-pipeline:

  $ cd radar-pipeline

• Checkout the development branch:

  $ git checkout dev

• Create a virtual environment and activate it.

  • Install the virtualenv package:

    $ python -m pip install --user virtualenv

  • Create a python virtual environment:

    $ python -m virtualenv env

  • Activate the virtual environment:

    On Windows, run:

    $ .\env\Scripts\activate

    On Linux or MacOS, run:

    $ source ./env/bin/activate

• Install the dependencies:

  $ pip install -r requirements.txt

• Run the following command in the project root directory to run the pipeline:

  $ python .

• The pipeline would do a mock run and ingest the data in the mock-data directory.

A note on submodules

The project uses git submodules to fetch additional resources such as the mock data. While cloning the repo for the first time, they are downloaded with the repo, if the --recurse-submodules flag is provided to the git clone command.

If the submodule needs to be updated again, run the following command in the project root directory:

$ git submodule update --init --recursive

For contributors

This project uses pre-commit to run isort, flake8, and black on the codebase before each commit.

To initialize the pre-commit hooks, run the following command in the project root directory:

$ pre-commit install