Astronomer kubernetes8/26/2023 For example:īy default, the KubernetesPodOperator expects to pull a Docker image that's hosted publicly on Docker Hub. To do so, define a 1ResourceRequirements object and provide that to the container_resources argument of the KubernetesPodOperator. Because of this, Astronomer recommends specifying compute resource requests and limits for each task. Resources used by the KubernetesPodOperator are not technically limited, which means that the operator could theoretically use any CPU and memory that's available in your Deployment to complete a task. Configure task-level Pod resources Īstro automatically allocates resources to Pods created by the KubernetesPodOperator. To further customize the way your tasks are run, see the topics below. This is the minimum configuration required to run tasks with the KubernetesPodOperator on Astro. To pull an image from a private registry, see Pull images from a Private Registry. The value you specify is assumed to be an image tag that's publicly available on Docker Hub. image: This is the Docker image that the operator will use to run its defined task, commands, and arguments.Information about this namespace can be programmatically imported as long as you set this variable. namespace = conf.get("kubernetes", "NAMESPACE"): Every Deployment runs on its own Kubernetes namespace within a cluster.To use the KubernetesPodOperator in a DAG, add the following import statements and instantiation to your DAG file:įor each instantiation of the KubernetesPodOperator, you must specify the following values: If you need assistance launching KubernetesPodOperator tasks in external Kubernetes clusters, contact Astronomer support. To run a task on a different node instance type, you must launch it in an external Kubernetes cluster. For example, a KubernetesPodOperator task that is triggered by an m5.4xlarge worker on AWS will also be run on an m5.4xlarge node. (Hybrid only) You cannot run a KubernetesPodOperator task in a worker queue or node pool that is different than the worker queue of its parent worker.PersistentVolumes (PVs) are not supported on Pods launched in an Astro cluster.To allow access to external data sources, you can provide credentials and secrets to tasks. Cross-account service accounts are not supported on Pods launched in an Astro cluster.On Astro, the Kubernetes infrastructure required to run the KubernetesPodOperator is built into every cluster and is managed by Astronomer. Set Kubernetes-native configurations in a YAML file, including volumes, secrets, and affinities.Scale task growth horizontally in a way that is cost-effective, dynamic, and minimally dependent on worker resources.This gives you flexibility and can enable new use cases across teams. Write task logic in a language other than Python.Specify CPU and memory as task-level limits or minimums to optimize performance and reduce costs.This can be helpful when you need to run individual tasks on infrastructure that isn't currently supported by Astro, such as GPU nodes or other third-party services. Run tasks in a Kubernetes cluster outside of Astro.This includes Docker images in a private registry or repository. Execute a custom Docker image per task with Python packages and dependencies that would otherwise conflict with the rest of your Deployment's dependencies.You can use the KubernetesPodOperator to: To configure defaults for all KubernetesPodOperator Pods, see Configure Kubernetes Pod resources. This document describes how to configure individual Pods for different use cases. This results in an isolated, containerized execution environment for each task that is separate from tasks otherwise being executed by Celery workers. Similar to the Kubernetes executor, this operator dynamically launches a Pod in Kubernetes for each task and terminates each Pod once the task is complete. The KubernetesPodOperator is one of the most powerful Apache Airflow operators.
0 Comments
Leave a Reply.AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |