Senior Data Engineer

Job Title: Senior Data Engineer
Location: Tampa, FL (Hybrid)
Job type: Full-time
 
Job Summary:

• Design, develop, and optimize scalable ETL/ELT pipelines using Snowflake / Databricks for batch and streaming data processing.
• Develop and maintain data models that support analytical and operational workloads.
• Implement best practices for performance tuning, cost optimization, and query efficiency in Snowflake or Databricks.
• Optimize storage, compute, and data-sharing strategies for cost-effective and performant solutions.
• Leverage platform-specific features (e.g., Snowflake's Time Travel, Zero-Copy Cloning, Streams & Tasks or Databricks' Delta Lake, Apache Iceberg, Delta sharing, and Unity Catalog).
• Ensure data security, access control, and governance frameworks are implemented and adhered to.
• Monitor data quality and performance, identifying areas for optimization and improvement.
• Ensure code quality and implement test cases for data pipeline code.
• Stay updated with industry trends in cloud data platforms and evaluate their application to improve our data ecosystem.
• Focus on building and maintaining the Data platform
• Partner with and advise multiple teams by guiding the creation and consumption of our data pipelines
• Lead and manage a team of junior data engineers to design, build, and scale the cloud data platform, ensuring it meets business needs and technical requirements
• Design and implement data processing services (stream and batch) with a focus on high scalability, performance, and cost-efficiency
• Apply best API design principles to build modular, reusable, and well-documented services
• Implement best practices in CI/CD pipelines, testing, and monitoring to maintain high-quality releases
• Work closely with product, application teams and other cross-functional teams to ensure seamless delivery of features
• Collaborate with data architects, data engineers, application owners, and product managers to deliver data solutions that align with data best practices and governance
• Drives cross functional design alignment of highly used enterprise data
• Initiate a culture of continuous improvement, promoting technical excellence and modern cloud development practices
• Monitor and optimize Azure resource usage to ensure a cost-effective platform
• Work in a collaborative environment to ensure a shared understanding of complex topics
• Contribute to and lead process improvement initiatives across the entire department
• Work with product & application leads to understand data needs and vet desired solutions
• Validates feasibility of the data design to ensure successful implementation of a cohesive data design
• Drive cross functional design alignment of highly used enterprise data
• Define and manage data integration standards (APIs, EDI, etc.) across our external partnerships, customers, and SaaS solutions
• Leads the planning phases (proposal, concept, feasibility) for new capabilities to identify high-level technology needs and roadmaps, assess feasibility, and collate customer expectations from the product or capability
• Lead the development and implementation of data solutions, using appropriate technologies and tools, providing technical guidance and leadership
• Lead requirements for data management, identifying batch and streaming use cases
• Define and build consensus on the roadmap, strategy, and vision of our platform
• Manage product backlog & stories
• Actively participate in the product development lifecycle from inception to support

• Provide technical leadership and guidance to development teams, driving the design and implementation
• Design, develop, and optimize streaming data pipelines and applications using technologies such as Apache Kafka, Apache Flink, or similar
• Identify and address performance bottlenecks in streaming applications, ensuring low-latency and high-throughput data processing
• Coach and mentor junior and mid-level engineers, fostering a culture of continuous learning and improvement
• Update knowledge repositories on a team level to help new resources enter the team and organize knowledge transfer sessions
• Attending various training sessions organized in the organization on Business/ Technical/ Agile practices topics
• Lead code reviews, enforce coding standards, and ensure code quality through constructive feedback and best practices
• Tackle complex technical challenges, troubleshoot issues, and drive resolution in a fast-paced environment
• Stay updated on industry trends, emerging technologies, and best practices. Explore and propose innovative solutions to improve our technology stack
• Communicating effectively with stakeholders at all levels of the organization to provide updates on the system and project progress
• Gather requirements and perform business Product and feasibility study of the project
• Build proof of concept for the new business proposals and get feedback from the business stakeholders
• Track project execution and identify any possible risks and provide resolution
• Plan for post-release production activities and support.
   

• Providing production support - monitoring day-to-day processes, timely fixing of production issues, performing data fixes, executing Month/Quarter/Year End batch processes & troubleshooting issues for business users and end users
• Regular interaction with the VP of Engineering, Data Engineering team, Product Owner, Customer Relationship Manager, Delivery Manager, Account manager, Client managers, and Support team – all belonging to different departments and divisions of client and organization’s client base, to ensure the smooth running and maintenance of applications and products.

• Bachelor's or Master's degree in Computer Science, Engineering or related field
• 6+ years of experience in data engineering with a focus on large-scale cloud-based data processing.
• Expertise in Snowflake / Databricks, including performance tuning, cost optimization, and best practices.
• Strong experience in batch and streaming data processing.
• Strong experience in Python/PySpark for data processing and automation.
• Strong experience in data modeling for both Operational and Analytical data.
• Hands-on experience with data security, access control, and governance frameworks in Snowflake or Databricks.
• Familiarity with Agile and/or DevOps methodologies
• Deep understanding of APIs and Experience building data processing pipelines with high availability, scalability, and performance
• Familiarity with event-driven architectures and data streaming technologies; kafka
• Experience with Incident management processes and handling tickets
• Solid understanding of distributed systems, microservices architecture, and containerization (e.g., Docker, Kubernetes)
• Experience with cloud platforms (e.g., Azure, AWS) is a plus
• Experience with CI/CD practices, automated testing, and monitoring for cloud-native applications
• Excellent problem-solving and analytical skills with a focus on delivering scalable, maintainable, and high-performing solutions
• Good communication and leadership skills, with experience leading and mentoring engineering teams
• Strong analytical and problem-solving abilities
• Good to have : Experience with Apache Druid/ MSSQL/ MySQL/ MongoDB/ Oracle/ PostgreSQL .
• Good to have: Experience with Graph Databases to manage complex relationships within data.
• Good to have: Experience in working with the Open-Source community.