Sentinel 1 and 2 ingestion plan

Introduction

These are the high level steps for the ingestion workflows of Sentinel1 and 2. The document has been drafted during the evo-odas code sprint of the first week of May 2017.

The word workflow is replaced with the acronym DAG which stands for Direct Acyclic Graph. DAG is the term used in the context of the Airflow engine, which is the workflow engine chosen for this project. In the Airflow terminology a single step of a workflow is called Operator and a running istance of an operator is a Task.

DAG Sentinel1

ROOT_DIR=BASE/S1/orig
PROC_DIR=BASE/S1/proc
ROOT_QUOTA=100GB
PROC_QUOTA=100GB

1. read configuration
2. download
3. move to archive (to ROOT_DIR/YEAR/MONTH/DAY/ID.zip)
4. check constraints (e.g. quota, collection exists, ...)
5. processing data (in /WORK)
   • generate EO/OM (via gdal/ogr for EOXserver)
   • generate THUMBS (via ssat1_meatadata.py write to DB)
   • generate granules tif (warp,addo,tiling,compress,alpha)
   • generate wkt footprints (from SAFE xml or via gdal)
6. move to store (to PROC_DIR/YEAR/MONTH/DAY/ID_(GRD|SLC|..).(.tif|_eop.xml|_QL.jpeg|.wkt))
7. register data (GS only | EOX poll)
8. test access
9. notification

A diagram of the steps can be found here

DAG Sentinel2

ROOT_DIR=BASE/S2/orig
PROC_DIR=BASE/S2/proc
ROOT_QUOTA=100GB
PROC_QUOTA=100GB

1. read configuration
2. download
3. move to archive (to ROOT_DIR/YEAR/MONTH/DAY/ID.zip)
4. check constraints (e.g. quota, collection exists, ...)
5. processing data (in /WORK)
6. generate EO/OM (via s2reader for EOXserver)
   • generate THUMBS (via ssat2_meatadata.py write to DB)
   • generate wkt footprints (from SAFE xml or via gdal)
   • EOXSERVER: generate granules tif (translate,addo) keeping the UTM coordinates
   • GEOSERVER: extract original granules jp2 (to store directly)
7. move to store (to PROC_DIR/YEAR/MONTH/DAY/ID_(10|20|60|..).(.tif|_eop.xml|.jp2|_QL.jpeg|.wkt))
8. register data (GS only | EOX poll)
9. test access
10. notification