Troubleshooting, Monitoring, and Optimization

It installs Python libraries into the notebook session (notebook-scoped libraries).

Narrow transformations (filter, map) need no shuffle — partitions map 1:1; wide transformations (groupBy, join, distinct) require a shuffle across partitions.

Call df.cache() (or persist()) to keep it in memory/disk across actions, avoiding recomputation when reused multiple times.

df.collect() returns all rows to the driver as a list; collecting a large DataFrame can cause driver OutOfMemory errors — prefer take/limit/show for previews.

Read the cell's output area for the exception type and message, then follow the stack trace to the failing line or function.

They display live progress of the Spark jobs triggered by a cell — number of jobs, stages, and completed/total tasks — helping spot stalls or skew.