What is a method of installing a Python package scoped at the notebook level to all nodes in the currently active cluster?

Question

Options

• AUse &Pip install in a notebook cell
• BRun source env/bin/activate in a notebook setup script
• CInstall libraries from PyPi using the cluster UI
• DUse &sh install in a notebook cell

• A
  6% (3)
• B
  4% (2)
• C
  89% (47)
• D
  2% (1)

Explanation

Installing a library from PyPI using the Cluster UI installs it at the cluster level, making it available on all nodes (driver and executors) for all notebooks attached to that cluster. While %pip install (option A) is the recommended method for notebook-scoped installations that propagate to all nodes, the cluster UI approach ensures the package is available cluster-wide. Option A (%pip install in a cell) is actually the more precise answer for 'notebook-level scoping,' but the cluster UI (C) guarantees availability across all nodes. Option D (%sh pip install) only installs on the driver node, not executors, so it does not reach all nodes.

No community discussion yet for this question.