Setting up computing resources
In this lesson you will set up a Python computing environment for scientific computing on your own computer and also learn a bit about Google Colab, a cloud service for running Jupyter notebooks.
It is advantageous to learn how to set up a Python distribution and manage packages on your own machine, as each person can have different needs. That said, Google Colab is a nice, free resource to run Jupyter Notebooks on Google’s computers without any local installations necessary.
Setting up Google Colab
In order to use Google Colab, you must have a Google account. Caltech students and employees have an account through Caltech’s G Suite. Many of you may have a personal Google account, usually set up for things like GMail, YouTube, etc. For your work in this class, use your Caltech account. This will facilitate collaboration with your teammates in the course, as well as with course staff.
Many of you probably use your personal Google account on your machine, so it can get annoying to log in and out of it. A trick that I find useful is to use one browser, e.g., Safari or Microsoft Edge, for your personal use, web browsing, etc., and a different browser for your scientific work, including the work in this class. Google Colab are most tested for Chrome, Firefox, and Safari (in fact JupyterLab, which you will use on your own machine, only supports these three browsers).
Once you have either logged out of all of your personal accounts or have a different browser open, you can launch a Colab notebook by simply navigating to https://colab.research.google.com/. Alternatively, you can click the “Launch in Colab” badge at the top right of this page, and you will launch this notebook in Colab. That badge will appear in the top right of all pages in the course content generated from notebooks.
Watchouts when using Colab
If you do run a notebook in Colab, you are doing your computing on one of Google’s computers via a virtual machine. You get two CPU cores and limited (about 12 GB, but it varies) RAM. You can also get GPUs and TPUs (Google’s tensor processing units), but we will not use those in this course. The computing resources should be enough for all of our calculations this term (though you will need more computing power in the sequel of this course). However, there are some limitations you should be aware of.
If your notebook is idle for too long, you will get disconnected from your notebook. “Idle” means that cells are not being edited or executed. The idle timeout varies depending on the load on Google’s computers; I find that I almost always get disconnected if idle for an hour.
Your virtual machine will disconnect if it is being used for too long. It typically will only available for 12 hours before disconnecting, though times can vary, again based on load.
These limitations are in place so that Google can offer Colab for free. If you want more cores, longer timeouts, etc., you might want to check out Colab Pro. However, the free tier should work well for you in the course. You can of course always run on your own machine, and in fact are encouraged to do so.
There are additional software-specific watchouts when using Colab.
Colab does not allow for full functionality of Bokeh apps that we will use when we do dashboarding.
Colab instances have specific software installed, so you will need to install anything else you need in your notebook. This is not a major burden, and is discussed in the next section.
I recommend reading the Colab FAQs for more information about Colab.
Software in Colab
When you launch a Google Colab notebook, much of the software we will use in class is already installed. It is not always the latest version of the software, however. In fact, as of mid-September 2023, Colab is running Python 3.10, whereas you will run Python 3.11 on your machine through your Anaconda installation. Nonetheless, most (but not all) of the analyses we do for this class will work just fine in Colab. We will make every effort to let you know when Colab will not be able to handle activities in class, the most important example being some dashboarding applications.
Because the notebooks in Colab have software preinstalled, and no more, you will often need to install software before you can run the rest of the code in a notebook. To enable this, when necessary, in the first code cell of each notebook in this class, we will have the following code (or a variant thereof depending on what is needed or if the default installations of Colab change). Running this code will not affect running your notebook on your local machine; the same notebook will work on your local machine or on Colab.
# Colab setup ------------------
import os, sys, subprocess
if "google.colab" in sys.modules:
cmd = "pip install --upgrade iqplot bebi103 watermark"
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
data_path = "https://s3.amazonaws.com/bebi103.caltech.edu/data/"
else:
data_path = "../data/"
# ------------------------------
In addition to installing the necessary software on a Colab instance, this also sets the relative path to data sets we will use in the course. When running in Colab, the data set is fetched from cloud storage on AWS. When running on your local machine for homeworks, the path to the data is one directory up from where you are working.
In most notebooks, the Colab and data path setup code cells are hidden in the HTML rendering to avoid clutter, but will be present when you download the notebooks.
Collaborating with Colab
If you want to collaborate with another student or with the course staff on a notebook, you can click “Share” on the top right corner of the Colab window and choose with whom and how (the defaults are fine) you want to share.
When we talk about Git in a future lesson, we will discuss Colab’s GitHub support, which will be necessary for version control and submitting and sharing your homework.
Installation on your own machine
We now proceed to discuss installation of the necessary software on your own machine. Before we get into that, there are some preliminaries for Windows users.
Windows users: Install Git and Chrome or Firefox
We will be using JupyterLab in this course. It is browser-based, and Chrome, Firefox, and Safari are supported. Microsoft Edge is not. Therefore, if you are a Windows user, you need to be sure you have either Chrome of Firefox installed.
Git is natively installed on Macs. For Windows users, you need to install Git by itself. You can do this by following the instructions here.
Downloading and installing Anaconda
If you already have Anaconda installed on your machine, you can skip this step.
Downloading and installing Anaconda is simple.
Go to the Anaconda’s download page and click the Download button.
Follow the on-screen instructions for installation. While doing so, be sure that Anaconda is installed in your home directory (which is the default), not in root.
That’s it! After you do that, you will have a functioning Python distribution.
Install node.js
node.js is a platform that enables you to run JavaScript outside of the browser. We will not use it directly, but it needs to be installed for some of the more sophisticated JupyterLab functionality. Install node.js by downloading the appropriate installer for your machine here.
Setting up a conda environment
I have created a conda environment for use in this class. You can download the YML specification for the environment:
You can set up and activate the environment on the command line or by using the Anaconda Navigator, which should be installed with Anaconda. You can do either of the two options, (a) or (b), below.
a) Activating from the command line
To set up your conda environment from the command line, navigate to the directory where you saved the bebi103.yml file. Then, on the command line, enter
conda env create -f bebi103.yml
This should build the environment for you (it may take several minutes). To then activate the environment, enter
conda activate bebi103
on the command line.
Launching JupyterLab
You can alternatively launch JupyterLab via the Anaconda Navigator or via your operating system’s terminal program (Terminal on macOS and PowerShell on Windows). If you wish to launch using the latter (which I prefer), skip to the next section.
In the Anaconda Navigator, click Home on the left pane. To the right, you will have a pane from which you can launch JupyterLab. On the top of the right pane, you will see two pulldown menus separated by the word “on.” Be sure you select bebi103 on the right pulldown menu. This ensures that you are using the environment you just set up.
You need to make sure you are using the pol_stats environment whenever you launch JupyterLab during the workshop.
You should see a card for JupyterLab. Do not confuse this with Notebook; you want to launch JupyterLab. Click Launch on the JupyterLab card. This will launch JupyterLab in your default browser.
Launching JupyterLab from the command line
While launching JupyterLab from the Anaconda Navigator is fine, I generally prefer to launch it from the command line on my own machine. If you are on a Mac, open the Terminal program. You can do this hitting Command + space bar and searching for “terminal.” Using Windows, you should launch PowerShell. You can do this by hitting Windows + R and typing “powershell” in the text box.
Once you have a terminal or PowerShell window open, you will have a prompt. At the prompt, type
conda activate bebi103
This will ensure you are using the bebi103 environment you just created.
You need to make sure you are using the bebi103 environment whenever you launch JupyterLab, so you should do conda activate bebi103 each time you open a terminal.
Now that you have activated the bebi103 environment, you can launch JupyterLab by typing
jupyter lab
on the command line. You will have an instance of JupyterLab running in your default browser. If you want to specify the browser, you can, for example, type
jupyter lab --browser=firefox
on the command line.
It is up to you if you want to launch JupyterLab from the Anaconda Navigator or command line.
Checking your distribution
We’ll now run a quick test to make sure things are working properly. We will make a quick plot that requires some of the scientific libraries we will use.
Use the JupyterLab launcher (you can get a new launcher by clicking on the + icon on the left pane of your JupyterLab window) to launch a notebook. In the first cell (the box next to the [ ]: prompt), paste the code below. To run the code, press Shift+Enter while the cursor is active inside the cell. You should see a plot that looks like the one below. If you do, you have a functioning Python environment for scientific computing!
You can also test this in Colab (and it should work with no problems).
[2]:
import numpy as np
import bokeh.plotting
import bokeh.io
bokeh.io.output_notebook()
# Generate plotting values
t = np.linspace(0, 2*np.pi, 200)
x = 16 * np.sin(t)**3
y = 13 * np.cos(t) - 5 * np.cos(2*t) - 2 * np.cos(3*t) - np.cos(4*t)
p = bokeh.plotting.figure(height=250, width=275)
p.line(x, y, color='red', line_width=3)
text = bokeh.models.Label(x=0, y=0, text='BE/Bi 103 a', text_align='center')
p.add_layout(text)
bokeh.io.show(p)
Computing environment
[3]:
%load_ext watermark
%watermark -v -p numpy,bokeh,jupyterlab
Python implementation: CPython
Python version : 3.11.5
IPython version : 8.15.0
numpy : 1.24.3
bokeh : 3.2.1
jupyterlab: 4.0.6