Spark & Scala Developer

Exp- 5-8 yrs

Shift- Malaysian

Full time Contract Opportunity

Fully Remote

please share resume at ******@weradioactive.com

Responsibilities-

Create Scala/Spark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

Skills-

Scala (with a focus on the functional programming paradigm)

Scalatest, JUnit, Mockito {{ , Embedded Cassandra }}

Apache Spark 2.x

{{ Apache Spark RDD API }}

{{ Apache Spark SQL DataFrame API }}

{{ Apache Spark MLlib API }}

{{ Apache Spark GraphX API }}

{{ Apache Spark Streaming API }}

Spark query tuning and performance optimization

Data Modelling, Data Warehousing

SQL database integration {{ Microsoft, Oracle, Postgres, and/or MySQL }}

Experience working with {{ HDFS, S3, Cassandra, and/or DynamoDB }}

Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)