Data Engineer
1 giorno fa
Modena, Emilia-Romagna, Italia
BairesDev
Da remoto
Tempo pieno
Gratuito con email o Google
Salva questo lavoro e mantieni la tua ricerca organizzata
Crea un account gratuito per salvare lavori, creare avvisi e tornare a questa inserzione dalla tua dashboard.
Gratuito con email o Google
Continuando accetti i nostri Termini & Informativa sulla privacy.
At BairesDev®, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley.
Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works remotely on roles that drive significant impact worldwide.
When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success.
Data Engineer (Spark) at BairesDev
As a Data Engineer specializing in Spark, you will design and build large-scale data pipelines to process complex batch and streaming workloads. You will serve as a key link between raw data sources and analytical insights, ensuring the efficiency and scalability of our data infrastructure.
What You'll Do
- Design and develop robust data pipelines using Python and PySpark to process large-scale datasets.
- Optimize Spark jobs by deep-diving into internals to improve performance and manage costs.
- Implement scalable storage solutions using Delta Lake or Iceberg for reliable data management.
- Orchestrate complex data workflows using Airflow to ensure timely and accurate data delivery.
- Build and maintain data processing infrastructure across cloud environments like AWS, Azure, or GCP.
- Write and optimize complex SQL queries for data transformation and validation within the pipeline. What we are looking for:
- 4+ years of experience in Data Engineering or Software Engineering.
- Proven expertise in building large-scale batch and streaming pipelines using Apache Spark and Python.
- Hands-on experience with Spark internals, PySpark, and Delta Lake or Apache Iceberg.
- Proficiency in SQL and workflow orchestration using tools like Airflow.
- Practical experience with cloud data platforms such as AWS, Azure, or GCP.
- Advanced proficiency in English. How we do make your work (and your life) easier:
- 100% remote work (from anywhere).
- Excellent compensation in USD or your local currency if preferred
- Hardware and software setup for you to work from home.
- Flexible hours: create your own schedule.
- Paid parental leaves, vacations, and holidays.
- Innovative and multicultural work environment: collaborate and learn from the global Top 1% of talent.
- Supportive environment with mentorship, promotions, skill development, and diverse growth opportunities. Apply now and become part of a global team where your unique talents can truly thrive
What You'll Do
- Design and develop robust data pipelines using Python and PySpark to process large-scale datasets.
- Optimize Spark jobs by deep-diving into internals to improve performance and manage costs.
- Implement scalable storage solutions using Delta Lake or Iceberg for reliable data management.
- Orchestrate complex data workflows using Airflow to ensure timely and accurate data delivery.
- Build and maintain data processing infrastructure across cloud environments like AWS, Azure, or GCP.
- Write and optimize complex SQL queries for data transformation and validation within the pipeline. What we are looking for:
- 4+ years of experience in Data Engineering or Software Engineering.
- Proven expertise in building large-scale batch and streaming pipelines using Apache Spark and Python.
- Hands-on experience with Spark internals, PySpark, and Delta Lake or Apache Iceberg.
- Proficiency in SQL and workflow orchestration using tools like Airflow.
- Practical experience with cloud data platforms such as AWS, Azure, or GCP.
- Advanced proficiency in English. How we do make your work (and your life) easier:
- 100% remote work (from anywhere).
- Excellent compensation in USD or your local currency if preferred
- Hardware and software setup for you to work from home.
- Flexible hours: create your own schedule.
- Paid parental leaves, vacations, and holidays.
- Innovative and multicultural work environment: collaborate and learn from the global Top 1% of talent.
- Supportive environment with mentorship, promotions, skill development, and diverse growth opportunities. Apply now and become part of a global team where your unique talents can truly thrive