Skip to content
Snippets Groups Projects
user avatar
Marian Neagul authored
83e44730

SciHub S2 Download Function

This function is meant to be triggered using an CloudEvent (of type io.y22.eo.s2.scihub.download_product).

The function expects to receive a CloudEvent containing as data the description of an Sentinel-2 product, as generated by the discover-s2-data function.

Requirements

This function requires the following environment variables to be set:

  • SCIHUB_USERNAME
  • SCIHUB_PASSWORD
  • BACKOFF_MAX_TIME: Backoff maxtime in case of dhus issues
  • BACKOFF_MAX_TRIES: Maximum number of retries in case of dhus issues
  • DESTINATION_PATH: Destination path inside the S3 bucket. All data will be stored beneth this directory
  • KEEP_ORIGINAL_PRODUCT_FILE: Store in the bucket the original file. Values of yes, True or 1 are interpreted as True, everything else as False
  • UPLOAD_EXTRACTED_PRODUCT: Should the extracted product be stored inside the bucket ?. Values of yes, True or 1 are interpreted as True, everything else as False
  • (optional) S3_ENDPOINT_URL: S3 Endpoint to be used.
  • S3_DESTINATION_BUCKET: The destination bucket name
  • S3_KEY: S3 authentication key
  • S3_SECRET: S3 secret

Return

This function return an object of type io.y22.eo.s2.scihub.product_downloaded containing modified product metadata pointing to the location of downloaded data instead of the original data.

Deployment

Using kn

kn service create -n functions --force download-s2-data -p 8080 --image registry.terrasigna.com/internal/infrastructure/functions/download-s2-data:latest --scale-min 0 --scale-max 20 --pull-policy Always -e SCIHUB_USERNAME=${SCIHUB_USERNAME} -e SCIHUB_PASSWORD=${SCIHUB_PASSWORD} -e DESTINATION_PATH=${DESTINATION_PATH} -e S3_DESTINATION_BUCKET=${S3_DESTINATION_BUCKET} -e S3_ENDPOINT_URL=${S3_ENDPOINT_URL} -e S3_KEY=${S3_KEY} -e S3_SECRET=${S3_SECRET}

Using offline configuration

Equivalent service definition:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  creationTimestamp: null
  name: download-s2-data
  namespace: functions
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/max-scale: "20"
        autoscaling.knative.dev/min-scale: "0"
        client.knative.dev/updateTimestamp: "2022-05-16T20:48:58Z"
        client.knative.dev/user-image: registry.terrasigna.com/internal/infrastructure/functions/download-s2-data:latest
      creationTimestamp: null
    spec:
      containers:
      - env:
        - name: SCIHUB_USERNAME
          value: ${SCIHUB_USERNAME} # To be replaced
        - name: SCIHUB_PASSWORD
          value: ${SCIHUB_PASSWORD} # To be replaced
        - name: DESTINATION_PATH
          value: ${DESTINATION_PATH} # To be replaced
        - name: S3_DESTINATION_BUCKET
          value: ${S3_DESTINATION_BUCKET} # To be replaced
        - name: S3_ENDPOINT_URL
          value: ${S3_ENDPOINT_URL} # To be replaced
        - name: S3_KEY
          value: ${S3_KEY} # To be replaced
        - name: S3_SECRET
          value: ${S3_SECRET} # To be replaced
        image: registry.terrasigna.com/internal/infrastructure/functions/download-s2-data:latest
        imagePullPolicy: Always
        name: ""
        ports:
        - containerPort: 8080
        resources: {}