Apache Spark Designer

Responsibilities Translate business needs to technical specifications Architecting, Design, build and deploy data processing solutions Hands on tools to process, stage, store data Conduct recon of data Collaborate with teams to integrate systems Develop and execute Spark Queries, database queries and conduct analyses Create dataset, RDD, Data Frame for requested projects Develop and update technical documentation Sound knowledge about effort estimation methods for data processing solutions requirements Proven experience as Apache Spark Designer. Hands-on coding experience for Apache Spark designing Background in data processing design (e.g. large scale data processing with API based status) and data curation In-depth understanding of database management systems, Spark clustering, Spark executor, Driver, real time data processing platform, Java 1.8, and ETL (Extract, transform, load) framework Knowledge of SQL queries, Spark Engine and Spark Services, Experience of Spark query tuning and performance optimization