Cloudera CDH#

Cloudera provides Apache Hadoop-based software, support, and services, as well as training to business customers. Their open-source Apache Hadoop distribution, CDH (Cloudera Distribution Including Apache Hadoop), targets enterprise-class deployments of that technology.

You can use Anaconda on an existing cluster with Cloudera CDH, Cloudera’s distribution including Apache Hadoop.

See the blog post Self-service Open Data Science: Custom Anaconda parcels for Cloudera, or follow the steps below.

Install the Anaconda parcel#

The following procedure describes how to install the Anaconda parcel on a CDH cluster using Cloudera Manager. The Anaconda parcel provides a static installation of Anaconda, based on Python 2.7, that can be used with Python and PySpark jobs on the cluster.

  1. In the Cloudera Manager Admin Console, in the top navigation bar, click the Parcels icon.

  2. At the top right of the parcels page, click the Edit Settings button.

  3. In the Remote Parcel Repository URLs section, click the plus symbol, and then add the following repository URL for the Anaconda parcel:

    https://repo.anaconda.com/pkgs/misc/parcels/

  4. At the top of the page, click the Save Changes button.

  5. In the top navigation bar, click the Parcels icon to return to the list of available parcels, where you should see the latest version of the Anaconda parcel that is available.

  6. To the right of the Anaconda parcel listing, click the Download button.

  7. After the parcel is downloaded, click the Distribute button to distribute the parcel to all of the cluster nodes.

  8. After the parcel is distributed, click the Activate button to activate the parcel on all of the cluster nodes.

  9. When prompted, confirm the activation.

After the parcel is activated, Anaconda is available on all of the cluster nodes.

You can submit Spark jobs along with the PYSPARK_PYTHON environment variable that refers to the location of Anaconda. For example, enter the following command all on one line:

PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/ python spark-submit pyspark_script.py

Note

The repository URL shown above installs the most recent version of the Anaconda parcel. To install an older version of the Anaconda parcel, add https://repo.anaconda.com/pkgs/misc/parcels/archive/ to the Remote Parcel Repository URLs in Cloudera manager, and then follow the above steps with your desired version of the Anaconda parcel.

Parcel updates#

Anaconda builds new Cloudera parcels at least once a year each spring and also offers custom parcel creation for our enterprise customers. The Anaconda parcel provided at the repository URL shown above is based on Python 2.7. To use the Anaconda parcel with other versions of Python or with additional packages, contact sales@anaconda.com for more information about custom Anaconda parcel builds or other enterprise solutions for using Anaconda with cluster computing.

Anaconda Workgroup and Enterprise tier subscribers can also use the professional repository to create and distribute their own custom Anaconda parcels for Cloudera Manager.

For more information about managing Cloudera parcels, see the Cloudera documentation.