Scientific Computing Linux and Cluster Engineer

Lawrence Berkeley National Lab’s (LBNL) Information Technology Division has an opening for a Scientific Computing Linux and Cluster Engineer to join the team.

In this exciting role, you will provide computing support to the Berkeley Lab research community and work in Berkeley Lab’s IT Division Scientific Computing Group. This position will provide wide-ranging expertise to support the Lab’s High Performance Computing infrastructure and provide state of the art Linux solutions in support of the science at Berkeley Lab. We help to enable some of the most advanced fundamental research in the world by providing the computing tools, networks, and expertise to enable pioneering science.

What You Will Do:

• Under the supervision of the Group Lead or senior team members, the successful candidate will participate in building, integrating and supporting Linux-based resources and end-users to meet the computing needs for various scientific disciplines. In addition, they will revamp and automate complex sysadmin processes to make them more robust. This person may also support large high performance computing cluster systems depending on the individual's experience, aptitude and skill set. Applicants should exhibit a passion for learning; the ability to integrate new computing technologies; an ability to comprehensively re-engineer sysadmin processes; and a deep desire to support scientific research.
• Within defined policies, procedures and practices provide Linux systems administration and user support for LBNL scientific research groups. This includes:
  • Linux system and HPC cluster maintenance and installations, operating system upgrades, system security hardening and intrusion detection, storage and file system management, system hardware, customization of user group working environment, troubleshooting, network monitoring, and crash recovery.
  • Design and maintenance of customer-facing container orchestration platforms focusing on both microservices and scientific workflows, including AAA, auditing, security policy design and performance tuning.
  • Design and implement automated build, deployment, and configuration management systems; Build and test automation tools for infrastructure provisioning; Handle code deployments; Monitor metrics, logging and monitoring while developing ways to improve; Build and manage CI and CD tools.
  • Configure, administer, and troubleshoot server and storage infrastructures as well as racking, installing, and maintaining systems in a datacenter.
  • Plan, organize, prioritize and complete assigned tasks and projects in a timely manner.
  • Frequently and clearly communicate task or project status to customers to either set or negotiate expectations.
  • Market IT Division services to the scientific community by providing excellent customer service coupled with competent technical support skills.
  • Participate in developing system administration, security, and network policies, documentation, and tools oriented towards efficient systems management.
  • Provide cluster support to LBNL and UC researchers. This includes: travel to remote sites if necessary, initial installation, integration and the on-going maintenance of Linux High Performance Computing cluster systems.
  • Lead technical efforts in one or more areas of HPC technologies such as job schedulers, high performance interconnects, parallel file systems, cybersecurity, cluster management, container orchestration, VM infrastructure, networking, performance tuning, or data center planning.
  • Lead group projects, of small to medium size and complexity, to implement and deploy new computing technologies and associated services to the research community.
  • Works on complex issues where analysis of situations or data requires an in-depth evaluation of variable factors. Exercises judgment in selecting methods, techniques and evaluation criteria for obtaining results. Networks with key contacts outside their own area of expertise.

What is Required:

• Typically requires a minimum of 8 years of related experience with a Bachelor’s degree; or 6 years and a Master’s degree; or equivalent experience.Typically requires a minimum of 8 years of related experience with a Bachelor’s degree; or 6 years and a Master’s degree; or equivalent experience.
• Wide-ranging expertise in the areas of Linux system administration experience in a large distributed computing environment. Experience providing systems and end-user support for multiple scientific or computational research groups.
• Expert-level experience with Red Hat Enterprise Linux (including derivatives such as CentOS and Scientific Linux), Debian, Ubuntu and use of large scale system administration tools and configuration management tools such as Kickstart, Ansible, Puppet, Chef, or in-house developed systems management tools, such as Warewulf. Support of common services such as NFS, LDAP, SMB, MySQL, Apache/Nginx HTTPD.
• Experience with implementing solutions based on Virtual Machines (VM) technologies such as KVM, VMWare, OpenStack etc. as well as container technologies such as Docker, Singularity and Kubernetes Platforms (Rancher, Rafay, etc).
• In-depth expertise in two or more areas of HPC technologies such as Linux operating systems, job schedulers, high performance interconnects, parallel file systems, cybersecurity, container orchestration, cluster management, VM infrastructure, networking, performance tuning, support of scientific applications, or data center planning.
• Having wide-ranging experience, uses professional concepts and company objectives to resolve complex issues in creative and effective ways.
• Must be able to demonstrate programming proficiency in Python and Bash. Must understand how to build, optimize and debug scientific codes that are written in C, C++, Fortran and Java. Must have experience with popular compilers (e.g. GCC, Intel), program debugging tools, use of Makefiles, use of version-control systems such as git and Subversion.
• Ability to plan, organize and successfully implement group projects for deploying new technologies and services.
• Ability to climb stairs, ladders, scaffolds; work at heights on above rack cabling; work in confined spaces, under florescent lights; ability to bend, stoop, kneel, crawl; manual dexterity in both hands; able to lift 60 lbs. to chest height; distinguish colors.

Desired Qualifications:

• Experience supporting HPC systems and end-users. HPC Linux clustering technology expertise (Job schedulers, MPI, Infiniband, parallel file systems, parallel programming).
• Experience supporting user-facing container orchestration platforms.
• Software engineering or development experience.
• Previous experience supporting research at a National Lab or academic institution.

For full consideration, please apply by April 5, 2023.

Notes:

• This is a full-time career appointment, exempt (monthly paid) from overtime pay.
• This position may be subject to a background check. Any convictions will be evaluated to determine if they directly relate to the responsibilities and requirements of the position. Having a conviction history will not automatically disqualify an applicant from being considered for employment.
• Work will be primarily performed at: Lawrence Berkeley National Lab, 1 Cyclotron Road, Berkeley, CA.

Salary:

The Scientific Computing Linux and Cluster Engineer position is expected to pay $139,140.00 - $196,958.00 annually, which fits within the full salary range of $123,684.00-$208,716.00 for the C70.3 - Computer Systems Engineer 3 position. Salary for this position will commensurate with the final candidate’s qualification and experience, including skills, knowledge, relevant education, certifications, plus also aligned with the internal peer group.

Based on University of California Policy - SARS-CoV-2 (COVID-19) Vaccination Program and U.S Federal Government requirements, Berkeley Lab requires that all members of our community obtain the COVID-19 vaccine as soon as they are eligible. As a condition of employment at Berkeley Lab, all Covered Individuals must Participate in the COVID-19 Vaccination Program by providing proof that vaccination requirements have been met or submitting a request for Exception or Deferral. Visit covid.lbl.gov for more information.

Berkeley Lab is committed to Inclusion, Diversity, Equity and Accountability (IDEA) and strives to continue building community with these shared values and commitments. Berkeley Lab is an Equal Opportunity and Affirmative Action Employer. We heartily welcome applications from women, minorities, veterans, and all who would contribute to the Lab's mission of leading scientific discovery, inclusion, and professionalism. In support of our diverse global community, all qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, or protected veteran status.

Equal Opportunity and IDEA Information Links: Know your rights, click here for the supplement: Equal Employment Opportunity is the Law and the Pay Transparency Nondiscrimination Provision under 41 CFR 60-1.4.  

Based on University of California Policy - SARS-CoV-2 (COVID-19) Vaccination Program, Berkeley Lab requires that all members of our community obtain the COVID-19 vaccine as soon as they are eligible. As a condition of Physical Presence at a Berkeley Lab Location, all Covered Individuals must Participate in the COVID-19 Vaccination Program by providing proof that vaccination requirements have been met or submitting a request for Exception or Deferral.

Berkeley Lab is committed to Inclusion, Diversity, Equity and Accountability (IDEA) and strives to continue building community with these shared values and commitments. Berkeley Lab is an Equal Opportunity and Affirmative Action Employer. We heartily welcome applications from women, minorities, veterans, and all who would contribute to the Lab's mission of leading scientific discovery, inclusion, and professionalism. In support of our diverse global community, all qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, or protected veteran status.

Equal Opportunity and IDEA Information Links: Know your rights, click here for the supplement: "Equal Employment Opportunity is the Law."  and the Pay Transparency Nondiscrimination Provision under 41 CFR 60-1.4.