Setting up computing resources

In this lesson you will set up a Python computing environment for scientific computing on your own computer and also learn a bit about Google Colab, a cloud service for running Jupyter notebooks.

It is advantageous to learn how to set up a Python distribution and manage packages on your own machine, as each person can have different needs. That said, Google Colab is a nice, free resource to run Jupyter Notebooks on Google’s computers without any local installations necessary.

Setting up Google Colab

In order to use Google Colab, you must have a Google account. Caltech students and employees have an account through Caltech’s G Suite. Many of you may have a personal Google account, usually set up for things like GMail, YouTube, etc. For your work in this class, use your Caltech account. This will facilitate collaboration with your teammates in the course, as well as with course staff.

Many of you probably use your personal Google account on your machine, so it can get annoying to log in and out of it. A trick that I find useful is to use one browser, e.g., Safari or Microsoft Edge, for your personal use, web browsing, etc., and a different browser for your scientific work, including the work in this class. Google Colab are most tested for Chrome, Firefox, and Safari (in fact JupyterLab, which you will use on your own machine, only supports these three browsers).

Once you have either logged out of all of your personal accounts or have a different browser open, you can launch a Colab notebook by simply navigating to https://colab.research.google.com/. Alternatively, you can click the “Launch in Colab” badge at the top right of this page, and you will launch this notebook in Colab. That badge will appear in the top right of all pages in the course content generated from notebooks.

Watchouts when using Colab

If you do run a notebook in Colab, you are doing your computing on one of Google’s computers via a virtual machine. You get two CPU cores and limited (about 12 GB, but it varies) RAM. You can also get GPUs and TPUs (Google’s tensor processing units), but we will not use those in this course. The computing resources should be enough for all of our calculations this term (though you will need more computing power in the sequel of this course). However, there are some limitations you should be aware of.

  • If your notebook is idle for too long, you will get disconnected from your notebook. “Idle” means that cells are not being edited or executed. The idle timeout varies depending on the load on Google’s computers; I find that I almost always get disconnected if idle for an hour.

  • Your virtual machine will disconnect if it is being used for too long. It typically will only available for 12 hours before disconnecting, though times can vary, again based on load.

These limitations are in place so that Google can offer Colab for free. If you want more cores, longer timeouts, etc., you might want to check out Colab Pro. However, the free tier should work well for you in the course. You can of course always run on your own machine, and in fact are encouraged to do so.

There are additional software-specific watchouts when using Colab.

  • Colab does not allow for full functionality of Bokeh apps.

  • Colab instances have specific software installed, so you will need to install anything else you need in your notebook. This is not a major burden, and is discussed in the next section.

I recommend reading the Colab FAQs for more information about Colab.

Software in Colab

When you launch a Google Colab notebook, much of the software we will use in class is already installed. It is not always the latest version of the software, however. In fact, as of July 2024, Colab is running Python 3.10, whereas you will run Python 3.12 on your machine through your Anaconda installation. Nonetheless, most (but not all) of the analyses we do for this class will work just fine in Colab. We will make every effort to let you know when Colab will not be able to handle activities in class, the most important example being some dashboarding applications.

Because the notebooks in Colab have software preinstalled, and no more, you will often need to install software before you can run the rest of the code in a notebook. To enable this, when necessary, in the first code cell of each notebook in this class, we will have the following code (or a variant thereof depending on what is needed or if the default installations of Colab change). Running this code will not affect running your notebook on your local machine; the same notebook will work on your local machine or on Colab.

# Colab setup ------------------
import os, sys, subprocess
if "google.colab" in sys.modules:
    cmd = "pip install --upgrade iqplot bebi103 watermark"
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    data_path = "https://s3.amazonaws.com/bebi103.caltech.edu/data/"
else:
    data_path = "../data/"
# ------------------------------

In addition to installing the necessary software on a Colab instance, this also sets the relative path to data sets we will use in the course. When running in Colab, the data set is fetched from cloud storage on AWS. When running on your local machine for homeworks, the path to the data is one directory up from where you are working.

In most notebooks, the Colab and data path setup code cells are hidden in the HTML rendering to avoid clutter, but will be present when you download the notebooks.

Collaborating with Colab

If you want to collaborate with another student or with the course staff on a notebook, you can click “Share” on the top right corner of the Colab window and choose with whom and how (the defaults are fine) you want to share.

When we talk about Git in a future lesson, we will discuss Colab’s GitHub support, which will be necessary for version control and submitting and sharing your homework.

Installation on your own machine

We now proceed to discuss installation of the necessary software on your own machine.

Downloading and installing Miniconda

If you already have Anaconda or Miniconda installed on your machine, you can skip this step and proceed to install node.js.

To download and install Miniconda, do the following.

Windows

  1. Go to the Miniconda page and go to the “Quick command line install” section.

  2. Click on the “Windows PowerShell” tab.

  3. Copy all of the contents in the gray box (starting the curl).

  4. Go to the Start menu and search for “PowerShell.” Click to open a PowerShell window. Alternatively, you can hit Windows + R and type PowerShell in the text box.

  5. Paste the copied text into the PowerShell window and hit enter.

macOS

  1. Go to the Miniconda page and go to the “Quick command line install” section.

  2. Click on the “macOS” tab.

  3. Copy all of the contents in the gray box (starting the mkdir).

  4. Open a Terminal window. You can do this by hitting Command-space bar, typing Terminal, and hitting enter. Alternatively, the Terminal application is located in the /System/Applications/Utilities/ folder, which you can navigate to using Finder.

  5. Paste the copied text into the Terminal window and hit enter.

Linux

  1. Go to the Miniconda page and go to the “Quick command line install” section.

  2. Click on the “Linux” tab.

  3. Copy all of the contents in the gray box (starting the mkdir).

  4. Open a terminal window. I assume you know how to do this if you are using Linux.

  5. Paste the copied text into the terminal window and hit enter.

Install node.js

node.js is a platform that enables you to run JavaScript outside of the browser. We will not use it directly, but it needs to be installed for some of the more sophisticated JupyterLab functionality. Install node.js by downloading the appropriate installer for your machine here.

Setting up a conda environment

I have created a conda environment for use in this class. You can download the YML specification for the environment via the link below (you may need to right-click and then download).

You can set up and activate the environment on the command line. (By “command line,” I mean a prompt in a PowerShell or terminal window.) Navigate to the directory where you saved the bebi103.yml file. (For example, if it is in a directory called Downloads in your home directory, you would do cd ~/Downloads on the command line to navigate there.) Then, on the command line, enter

conda env create -f bebi103.yml

This should build the environment for you (it may take several minutes). To then activate the environment, enter

conda activate bebi103

on the command line.

Launching JupyterLab

You can launch JupyterLab via your operating system’s terminal program (Terminal on macOS and PowerShell on Windows). If you are on a Mac, open the Terminal program. You can do this hitting Command + space bar and searching for “terminal.” Using Windows, you should launch PowerShell. You can do this by hitting Windows + R and typing “powershell” in the text box.

Once you have a terminal or PowerShell window open, you will have a prompt. At the prompt, type

conda activate bebi103

This will ensure you are using the bebi103 environment you just created.

You need to make sure you are using the bebi103 environment whenever you launch JupyterLab, so you should do conda activate bebi103 each time you open a terminal.

Now that you have activated the bebi103 environment, you can launch JupyterLab by typing

jupyter lab

on the command line. You will have an instance of JupyterLab running in your default browser. If you want to specify the browser, you can, for example, type

jupyter lab --browser=firefox

on the command line.

Checking your distribution

We’ll now run a quick test to make sure things are working properly. We will make a quick plot that requires some of the scientific libraries we will use.

Use the JupyterLab launcher (you can get a new launcher by clicking on the + icon on the left pane of your JupyterLab window) to launch a notebook. In the first cell (the box next to the [ ]: prompt), paste the code below. To run the code, press Shift+Enter while the cursor is active inside the cell. You should see a plot that looks like the one below. If you do, you have a functioning Python environment for scientific computing!

You can also test this in Colab (and it should work with no problems).

[2]:

import numpy as np
import bokeh.plotting
import bokeh.io

bokeh.io.output_notebook()

# Generate plotting values
t = np.linspace(0, 2*np.pi, 200)
x = 16 * np.sin(t)**3
y = 13 * np.cos(t) - 5 * np.cos(2*t) - 2 * np.cos(3*t) - np.cos(4*t)

p = bokeh.plotting.figure(height=250, width=275)
p.line(x, y, color='red', line_width=3)
text = bokeh.models.Label(x=0, y=0, text='BE/Bi 103 a', text_align='center')
p.add_layout(text)

bokeh.io.show(p)
Loading BokehJS ...

Computing environment

[3]:

%load_ext watermark
%watermark -v -p numpy,bokeh,jupyterlab

Python implementation: CPython
Python version       : 3.12.5
IPython version      : 8.27.0

numpy     : 1.26.4
bokeh     : 3.4.1
jupyterlab: 4.2.5