Scala

Just Enough For Developing Spark Applications

@jaceklaskowski / StackOverflow / GitHub
Books: Mastering Apache Spark / Mastering Spark SQL / Spark Structured Streaming

curly brackets


object Main extends App {
  println("Hello, Friends!")
}

Hello World in Action


scala> :paste
// Entering paste mode (ctrl-D to finish)

object Main extends App {
  println("Hello, Friends!")
}

// Exiting paste mode, now interpreting.

defined object Main

scala> Main.main(null)
Hello, Friends!

Hello World in Action


scala> :load yourScalaCode.txt
defined object Main
Hello, Friends!
# spark-shell ### Learning Scala (and Spark) Interactively
## Enjoy Scala (Exercises) 1. Get familiar with type literals * Strings, Ints, Longs, Doubles, Chars, Symbols 1. **Println** your name to the console 1. Define your name as a **val** and use to println * Use string interpolation #protip 1. Save the code to a file and **:load** it
# Scala Types
# Expressions ### ... And Statements
# Scaladoc ## Scala Documentation
## Scala Standard Library * https://www.scala-lang.org/api/2.11.12 * Spend as much time with the docs as possible * That will pay off quickly
# Methods in Scala ## Named executable blocks of code
## Method Definition * **def** Keyword * Optional input parameters (in round brackets) * **name: Type** * comma as separator for multiple input params * Optional return type ``` scala> def greet(name: String): String = s"Hello, $name" greet: (name: String)String scala> greet("Jacek") res8: String = Hello, Jacek // What's res8? ```
## Methods * Methods belong to classes * Class as a container of methods (and state) * Implicit **this** parameter for the owning object ``` case class Person(name: String) { def introduceYourself: String = s"I'm $name" } // Use :paste or :load scala> val jacek = new Person("Jacek") jacek: Person = Person(Jacek) scala> jacek.introduceYourself res9: String = I'm Jacek ```
## Methods in spark-shell * spark-shell auto-generates an implicit object * Another reason to use spark-shell early and often * Makes for a better learning experience ``` scala> def greet(name: String) = s"Hello, $name" greet: (name: String)String scala> greet("Jacek") res8: String = Hello, Jacek ```
## Named Parameters * Use parameter names at call site * Any order of parameters possible ``` scala> def greet(name: String) = s"Hello, $name" greet: (name: String)String scala> greet(name = "Jacek") res8: String = Hello, Jacek ```
# Functions in Scala ## (Other) Named executable blocks
## Function Definition * Functions are values of a function type * Part of objects or classes ``` scala> val greet: String => String = { name => s"Hello, $name" } greet: String => String = scala> val greet = { name: String => s"Hello, $name" } greet: String => String = scala> greet("Jacek") res10: String = Hello, Jacek ```
## _ Placeholder * **_** (underscore) to reference one input parameter * \# underscores = \# input parameters * Access ordered from left to right * Very useful notation with higher-order functions * Higher-order functions defined on the following slide ``` // What's the more proper name of the function? scala> val f: (Int, Int) => Int = _ + _ f: (Int, Int) => Int = ```
# for-comprehension ## Functional for Loop
## Demo * Generating Coordinates (as Pairs of Numbers)
# IntelliJ IDEA ## Integrated development environment ## for Scala and Spark

IntelliJ IDEA

  1. Integrated development environment
    • The best development tool for Scala
  2. Support for Scala using separate Scala plugin
  3. Scala Worksheet for Learning and Prototyping
  4. "IntelliSense" Feature
  5. Toggle Distraction Free mode
  6. Lots of Key Shortcuts
    • You're going to use them quite often!

IntelliJ IDEA — Demo

  1. Creating a Scala sbt-managed project
    • Enable Use auto-import checkbox
    • Disable Create directories for empty content roots automatically checkbox
    • Wait till IntelliJ IDEA (and sbt) resolves the project's dependencies that may take some longer time
  2. Running the Hello World application
    • Right-click the object name and select Run
    • Click double-window icon on the stripe on the left

IntelliJ IDEA — Exercise

  1. Create a Scala standalone application using IntelliJ IDEA
    • Use a Scala sbt-managed project
  2. Run the Scala application
  3. Pass command-line arguments
    • Run > Edit Configurations menu
    • Program arguments field
    • Run the application once to have an configuration entry

ScalaTest

ScalaTest

  1. Scala Testing Framework
  2. Matchers - Assertions
  3. sbt supports ScalaTest
    • sbt test (and don't forget about ~ - a tilde)
  4. IntelliJ IDEA supports ScalaTest, too.
  5. http://www.scalatest.org
  6. Demo: Installation and Execution

Exercise: Writing a Test

  1. Follow http://www.scalatest.org
  2. Define libraryDependencies in build.sbt
  3. Run tests using sbt test