web UI

Apache Spark 2.4

@jaceklaskowski / StackOverflow / GitHub
Books: Mastering Apache Spark / Mastering Spark SQL / Spark Structured Streaming

## Agenda 1. [Spark Properties](#/spark-properties) 1. [web UI](#/webui) 1. [web UI / Spark Properties](#/webui-spark-properties) 1. [web UI Tabs](#/webui-tabs) 1. [Demo: web UI Walkthrough](#/demo)

## Spark Properties 1. **spark-submit**'s Command-Line Options 1. Spark Properties 1. Environment Variables <small>(considered obsolete)</small> 1. Switch to [Mastering Apache Spark](http://bit.ly/mastering-apache-spark) * [Spark Properties and spark-defaults.conf Properties File](https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-properties.html) * [SparkSubmitOptionParser — spark-submit's Command-Line Parser](https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-submit-SparkSubmitOptionParser.html)

## web UI 1. **web UI** is the web interface of a singe Spark application 1. Monitoring and inspecting Spark jobs and tasks 1. Opens at http://localhost:4040 (by default) 1. One big **SparkListener** 1. Composed of tabs and pages <small>(on the following slides)</small>

## web UI / Spark Properties 1. Enabled by default unless **spark.ui.enabled** is false 1. Use **spark.ui.port** to set the port number 1. Actual availability and URL depends on the deployment environment (e.g. YARN, Mesos, Standalone, local, Kubernetes)

## web UI Tabs <small>(1 of 2)</small> 1. **Jobs** shows status of Spark jobs 1. **Stages** shows status of stages of all Spark jobs 1. **Storage** shows RDD size and memory use 1. **Environment** shows the environment 1. **Executors** shows status of executors 1. **SQL** shows status of batch and streaming queries

## web UI Tabs <small>(2 of 2)</small> 1. Module-specific tabs for **Streaming** and **Thrift Server** 1. Switch to [Mastering Apache Spark](http://bit.ly/mastering-apache-spark) * [Web UI — Spark Application’s Web Console](https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-webui.html)

Demo

web UI Walkthrough

## Recap 1. [Spark Properties](#/spark-properties) 1. [web UI](#/webui) 1. [web UI / Spark Properties](#/webui-spark-properties) 1. [web UI Tabs](#/webui-tabs) 1. [Demo: web UI Walkthrough](#/demo)

# Questions? * Read [Mastering Apache Spark](http://bit.ly/mastering-apache-spark) * Read [The Internals of Spark SQL](https://bit.ly/mastering-spark-sql) * Read [The Internals of Spark Structured Streaming](http://bit.ly/spark-structured-streaming) * Follow [@jaceklaskowski](https://twitter.com/jaceklaskowski) on twitter * Upvote [my questions and answers on StackOverflow](http://stackoverflow.com/users/1305344/jacek-laskowski)