Job Description
Operate and tune HDFS, YARN, Hive, and Spark across both clusters, including a mixed-version estate one cluster runs an older HDP-era stack with Spark 2.x and a legacy Python driver environment the other has recently been upgraded to a modern Hive release. Managing that version skew and the compatibility traps it creates for job authors is part of the role.
Own gateway host capacity. Monitor and set limits on concurrent worker pools, multiprocessing fan-outs, and Spark drivers so a single workload cannot starve the box. Push back on resource requests that the available evidence doesn t actually support.
Run the Hive Metastore and HiveServer2, including JVM sizing and OOM exposure. Diagnose transient failures such as partition-add serialization errors and thrift session rejections.
Administer the clusters through Ambari service configs, alerts, rolling restarts, and version upgrades.
Handle capacity planning, HDFS quotas, and data retention. Every dataset we generate needs a documented retention and cleanup story, and you ll help enforce that on the storage side.
Partner with the data engineering team, which orchestrates pipelines with Apache Airflow on AWS MWAA. Most remote work reaches the clusters over SSH from the gateways, so you ll be the escalation point when a pipeline task fails for infrastructure reasons rather than logic reasons.
Manage access control, SSH key hygiene, and service accounts across gateways and cluster nodes.
Build the monitoring and alerting we re missing host load, per-tenant resource consumption, metastore health, and early warning before a gateway becomes unresponsive.
