One Article Review
| Source |  ProofPoint |
|---|---|
| Identifiant | 8548846 |
| Date de publication | 2024-08-01 10:25:29 (vue: 2024-08-01 13:18:15) |
| Titre | Création de déploiements de kubernetes évolutifs avec des mesures personnalisées Creating Scalable Kubernetes Deployments with Custom Metrics |
| Texte | Engineering Insights is an ongoing blog series that gives a behind-the-scenes look into the technical challenges, lessons and advances that help our customers protect people and defend data every day. Each post is a firsthand account by one of our engineers about the process that led up to a Proofpoint innovation. Proofpoint Enterprise Archive offers a central, searchable repository that allows ingestion and meets Financial Industry Regulatory Authority (FINRA) standards for compliance. This blog explores how we achieved scale using Kubernetes as our deployment platform. Kubernetes and Horizontal Pod Autoscaler (HPA) Kubernetes is an open-source platform that provides straightforward container orchestration, scalability, resilience and fault tolerance for microservices. All of these factors made it a clear choice as a deployment platform for our application. HPA in Kubernetes is a built-in feature that automatically adjusts the number of replicas (pods) of a deployment based on observed metrics. In a regular environment, HPA queries the Kubernetes metric server, calculates the intended number of replicas and updates a deployment with a desired replica count. The deployment scales the pod count to the desired value. Horizontal Pod Autoscaler (HPA) in Kubernetes. Scaling a deployment based on custom metrics Autoscaling solutions are typically based on runtime metrics like CPU load. However, built-in system metrics are not always adequate for making autoscaling decisions. You need custom metrics to make useful autoscaling decisions. The Proofpoint Enterprise Archive includes a task scheduling system that takes tasks from the queue and executes them one by one. CPU-based autoscaling might not be optimal here because: Tasks can be IO-bound, not CPU-bound Tasks can have priorities, schedules and deadlines For prolonged periods, NO tasks might be scheduled to run at all; in this case, the scaling solution ideally would downscale to zero pods Another applicable scenario would be to scale proactively based on an artificial intelligence (AI) system that predicts load based on past usage patterns. For our use case, the tasks queue length can be a better metric to make scaling decisions, but it requires a custom queue length metric. Although you can set up Kubernetes HPA for this type of scaling, it can be challenging to implement custom metrics for scaling. Furthermore, HPA does not support scaling down to zero pods, which is essential to manage costs. Kubernetes Event Driven Autoscaling (KEDA) is a complementary autoscaling technology you can integrate into HPA. It offers a wide variety of scalers that can fetch metrics from various sources, including Prometheus monitors. KEDA uses these metrics for its integration with Kubernetes HPA. KEDA using its Prometheus Scaler to query metrics from Prometheus. The diagram above shows KEDA using its Prometheus Scaler to query metrics from Prometheus. In our system, the external metrics are mostly exposed by the application itself. KEDA uses these metrics to manage the HPA thereby indirectly controlling the pods\' replica count. KEDA also allows scaling the pods to zero by simply deleting the HPA object. When metrics demand the creation of pods, KEDA recreates the HPA object and starts to manage it. Sample definition and scaling behavior of a processor As a scaling example, we will show a background task processor (BTP) that waits for customer-driven events, picks them up and processes them. It is idle when there are no events to process. Our application exposes metrics that signal when and how many processors are required to handle customer events at any given time. Prometheus scrapes these metrics and makes them available to KEDA for processing. KEDA scale definition The above KEDA query definition includes the following parameters: btp_scaling_metrics is the Prometheus metric for scaling& |
| Notes | ★★★ |
| Envoyé | Oui |
| Condensat | 1600m 1600m/5 2*5 4400m 4400m/5 8/5 =10 about above abstracting acceptable account achieved activationthreshold active activity add additional adequate adjusts advances after against all all; allowed allows also although always among analyze another any applicable application applications architecture archive archiving are argorollout artificial authority author automatically autoscaler autoscaling available background backgrounds based because because: been behavior behind being best better blend blog bound bound brands btp build built business but calculates can capabilities capacity captured career careers case central challenges challenging choice clear cloud collaborate committed compatibility complementary complexity compliance concepts configured configures connection constantly container continuation controlling cost costs count cpu creates creating creation custom customer customers data day deadlines decisions deep defend definition definition deleting demand demonstrates depicts deployment deployments desired destroyed detailed detected detection detection developer diagram direct discussed distributed diversity does down down downscale driven driving during each earlier effective effectively efficient eight engineer engineering engineers enhance enjoys enough enterprise environment equal equates essential event events every evolving example executes experiences explores exposed exposes extending external factors failure fallback family fault feature fetch fifth final financial finra firsthand five flexible focus following force found four free from functionality furthermore generation getting given gives goal handle handling has have help helping here highly hire his horizontal how however hpa ideally idle implement improving includes including indicates indirectly industry infrastructure ingestion innovation insight insights integrate integration intelligence intelligence intended interested invest its itself join keda kids kubernetes kumar largest last learning led length less lessons like lived load look made make makes making manage management manages many meets metric metrics metrics microservices might monitors more mostly must need needed needing new next not note now number object observed offer offers one ongoing only open opportunities optimal orchestration other overloading page parameter parameters: party passion past patterns people per periods pg10 picks place platform platform playing pod pod/ pods pods post powerful practices predicts previous priorities proactively problems process processes processing processor processor processors prolonged prometheus proofpoint protect protecting proven provide provides providing queries query queue reached received recreates regular regulatory replica replicas repository required requires resilience resource resulting risks rules run running runtime safest sample scalability scalable scale scaler scalers scales scaletargetref scaling scaling scenario scenes scheduled schedules scheduling scrapes searchable seconds security series server set setting settings should show shown shows signal signals significantly simple simplifying simply since snapshot solution solutions solve source sources specification specify speed stabilizationwindowseconds staff standards starting starts statefulset straightforward success sufficient sumit support system systems s advanced takes target task tasks team team technical technologies technology terminating than them thereby these third thoughts threats threats and compliance three threshold through time to: today tolerance tools total toughest cybersecurity challenges trouble turn two type typically up updates usage use used useful users uses using value value values variety various waits what when which who wide will without works would you zero “desiredreplicas” “target” visit |
| Tags | Tool Cloud Technical |
| Stories | |
| Move | 
|
L'article ne semble pas avoir été repris aprés sa publication.
L'article ne semble pas avoir été repris sur un précédent.