Remote, but you must be in the following location
**Position: Data Engineer (Platform Modernization & Migration) ****Location: Fully Remote (EMEA & APAC Preferred) ****Contract Type: Full-time, Permanent****Time Zone Alignment: Requires overlap until at least 12:00 PM Pacific Time (PT)**🧭 About Inallmedia.com Inallmedia.com is a global technology and design firm focused on building impactful digital solutions through remote, distributed teams. We partner with international clients across industries, providing long-term technical expertise, product innovation, and team augmentation.🚀 Project Overview You will join a core Data Engineering organization, working in close partnership with the Data Platform team to drive a major data platform modernization initiative. The primary focus of this project is executing pre-designed migration blueprints for approximately 1,000 datasets to Snowflake with Airflow 3 orchestration. You will play a crucial role in accelerating time-sensitive infrastructure goals by evolving legacy Lakehouse formats to Apache Iceberg, modernizing streaming infrastructure across cloud regions, and adapting existing pipelines to uphold strict data quality and performance standards.🔍 Key Responsibilities* **Pipeline Migration & Upgrades:** Execute the end-to-end migration of approximately 1,000 datasets to Snowflake, including upgrading orchestration workflows from Apache Airflow 2 to Airflow 3. * **Storage & Data Lake Modernization:** Migrate legacy Hive data structures and datasets to Apache Iceberg format using the Snowpark Connect framework/connectors. * **Streaming Infrastructure & Region Migration:** Reconfigure, update, and migrate Confluent/Apache Kafka streaming connections across cloud regions (e.g., migrating from US East 1 to US East 2). * **Pipeline Adaptation & Performance Tuning:** Adapt existing data pipelines to target Snowflake environments while optimizing SQL queries (utilizing CTEs, window functions, and complex joins) to ensure high performance and data reliability. * **Workflow Scheduling & Monitoring:** Manage, test, and debug DAGs (Directed Acyclic Graphs) and workflow scheduling within Apache Airflow. * **Testing & Quality Assurance:** Implement testing strategies and conduct hands-on code debugging to guarantee data quality and consistency throughout the platform modernization.💡 Must-Have Skills* **Industry Experience:** Minimum 2+ years of hands-on data engineering experience working with large-scale datasets, Apache Airflow, and Apache Spark. * **Core Languages & SQL:** Strong software engineering fundamentals with hands-on Python development experience and advanced SQL skills (complex joins, CTEs, window functions, and query tuning). * **Data Warehousing & Modeling:** Proven experience with Snowflake data modeling and schema architecture. * **Data Processing & Orchestration:** Direct experience building and testing pipelines using Apache Airflow, Apache Kafka, and Apache Spark / PySpark. * **Data Formats & Cloud Storage:** Expertise working with Apache Iceberg and Apache Parquet data formats, alongside AWS S3 (data lakes, bucket partitioning, and lifecycle policies). * **Testing & Debugging:** Ability to define testing strategies and independently debug complex distributed data pipelines. * **Communication & Autonomy:** Strong communication skills with a proven track record of working independently in a fast-paced environment. * **AWS WorkSpace Requirement:** This role requires the use of a client-provisioned AWS WorkSpace. Candidates **must** demonstrate good connectivity to an AWS WorkSpace region by completing the connectivity test at . _Please note: You will be required to submit a screenshot of these results as supporting evidence with your application._✨ Nice-to-Have Skills* **Snowflake Migration:** Experience with enterprise Snowflake migrations (warehouse modernization, cost and performance tuning). * **Airflow Migration:** Direct experience with Apache Airflow platform migrations (version upgrades, executor changes, or platform moves). * **AI Tooling:** Hands-on experience using AI-assisted coding/development tools to accelerate workflow velocity.🌐 Time Zone & Collaboration The role is fully remote, with a strong preference for candidates located in the EMEA (Europe, Middle East, Africa) or APAC (Asia-Pacific) regions. **Time zone coverage is required until at least 12:00 PM PT (Pacific Time)** to ensure seamless overlap with the core engineering team.💬 Language All interviews, documentation, and daily communication will be in English.#LI
