1 Understanding Your Computer and Operating System#
When you use a computer to analyze experimental data, generate a graph, run a scientific simulation, or collect measurements from a remote sensor, several different components work together. Your computer has physical hardware, including a processor, memory, and storage devices. It also has software that allows you to interact with that hardware and perform useful tasks.
Most of us interact with computers through graphical interfaces, opening applications, clicking icons, and navigating folders. However, scientific computing often requires us to work more directly with the computer through a terminal, where we can enter commands, execute programs, and automate repetitive tasks.
Before installing Python and the software we will use in this course, it is helpful to understand how computers organize files, execute programs, and manage their resources. We will also explore the differences among Windows, macOS, and Linux, and why many scientific computing tools were developed for Unix-like operating systems.
Although students in this course may use different operating systems, our goal is to establish a common computing environment that allows everyone to work with the same Python packages, notebooks, and scientific datasets.
Learning Objectives#
After completing this notebook, you should be able to:
Describe the functions of an operating system.
Identify important similarities and differences among Windows, macOS, and Linux.
Explain how files and directories are organized and how file paths identify their locations.
Distinguish between persistent storage and random-access memory (RAM).
Explain the difference between a program and a running process.
Distinguish between a graphical user interface, a terminal, and a shell.
Explain why Unix-like computing environments are commonly used in scientific research.
1. What Is an Operating System?#
An operating system (OS) is the software that manages a computer’s hardware and provides the environment in which other programs run. Whether you are using a Windows laptop, an Apple Mac, or a computer running Linux, an operating system is responsible for coordinating the activities of the computer.
A computer consists of physical components, collectively called hardware, and the instructions that tell those components what to do, called software. Hardware includes the central processing unit (CPU), random-access memory (RAM), storage devices, and peripherals such as keyboards, displays, and scientific instruments. Software includes the operating system as well as applications such as web browsers, spreadsheets, and scientific programming tools.
The operating system serves as an intermediary between applications and the computer’s hardware. For example, when a Python program reads a file containing experimental measurements, it typically does not communicate directly with the storage device. Instead, it requests the data through services provided by the operating system.
1.1 Major Functions of an Operating System#
Although operating systems perform many tasks, their principal responsibilities include:
Process management: The OS manages running programs, allocating CPU time so that multiple applications can operate concurrently. For example, you might run a Python program while simultaneously using a web browser and editing a document.
Memory management: The OS allocates RAM to running programs and manages how memory is shared and protected. When a Python program loads a large scientific dataset, the operating system helps provide the memory needed to work with that data.
File management: The OS organizes information into files and directories (folders), allowing programs to create, read, modify, and delete data. Scientific measurements saved in a CSV file, for example, can later be retrieved and analyzed using Python.
Device management: The OS coordinates communication with hardware devices, including keyboards, displays, printers, storage devices, and connected scientific instruments. Device drivers allow the operating system to communicate with particular types of hardware.
Networking: The OS provides services that allow applications to communicate with other computers over local networks or the internet. A Python program might use these services to retrieve data from a scientific database or receive measurements from a remote sensor.
Security and access control: The OS manages user accounts, permissions, and access to files and system resources. These protections help prevent unauthorized access and keep programs from interfering with one another.
1.2 Operating Systems and Scientific Computing#
Consider a Python program that retrieves environmental measurements from an online database, calculates average temperatures, and generates a graph.
Although we may think of this as a single Python program, the operating system supports several activities behind the scenes:
It provides network services for retrieving the measurements.
It allocates memory for storing and processing the data.
It schedules CPU time for the calculations.
It provides access to the display for viewing the graph.
It manages the file system if the results are saved for later use.
These activities illustrate an important principle: Python performs the scientific calculations and data processing, while the operating system manages the computing resources that make those operations possible.
Understanding this relationship will help us install and configure Python, manage scientific data, and troubleshoot problems when programs do not behave as expected.
2. Windows, macOS, and Linux#
The three major operating systems used on personal computers are Microsoft Windows, Apple macOS, and Linux. Although they differ in their design, appearance, and underlying architecture, they perform essentially the same functions: managing hardware, running applications, organizing files, and providing interfaces through which users interact with their computers.
2.1 Microsoft Windows#
Windows is widely used on personal computers in homes, schools, businesses, and research laboratories. It supports a large variety of commercial and scientific software and provides a familiar graphical user interface (GUI) for interacting with files and applications.
Windows also includes command-line tools, such as PowerShell, that allow users to interact with the operating system by entering text commands. In this course, students using Windows will install the Windows Subsystem for Linux (WSL 2), which allows Linux to run alongside Windows. They will then install Ubuntu, a popular Linux distribution, within WSL 2. This provides a Linux computing environment without replacing Windows or requiring a separate computer.
2.2 Apple macOS#
macOS is the operating system used on Apple Macintosh computers. Like Windows, it provides a graphical interface for managing files, launching applications, and interacting with the computer.
An important distinction is that macOS is built on a Unix-based foundation. This means it already includes many of the command-line tools commonly used in scientific computing. Mac users can access these tools through the Terminal application without installing a separate Linux operating system.
2.3 Linux#
Linux is a family of free and open-source operating systems widely used in scientific research, web servers, cloud computing, and high-performance computing (HPC). Unlike Windows and macOS, which are developed primarily by individual companies, Linux is developed through contributions from an international community of programmers and organizations.
Linux comes in different distributions, including Ubuntu, Debian, and Fedora. These distributions share the Linux kernel, which is the core of the operating system, but differ in the software and tools they provide.
Linux is particularly important in scientific computing because many research computing systems, including supercomputers and remote servers, run Linux.
2.4 Comparing Operating Systems#
Despite their differences, all three operating systems can run Python and common scientific software packages, including NumPy, pandas, Matplotlib, and JupyterLab.
Feature |
Windows |
macOS |
Linux |
|---|---|---|---|
Developer |
Microsoft |
Apple |
Open-source community |
Common command-line shell |
PowerShell |
Zsh |
Bash |
Typical home directory |
|
|
|
Unix-like environment |
Available through WSL 2 |
Built in |
Native |
Python support |
Yes |
Yes |
Yes |
One important difference among these systems is how they organize files and interpret commands. For example, Windows traditionally uses backslashes (\) to separate directories in file paths, whereas macOS and Linux use forward slashes (/).
In the next notebook, we will establish a common Unix-like computing environment. Windows users will install Ubuntu through WSL 2, while macOS users will work with their existing Terminal application. This will allow us to use many of the same commands and tools throughout the course, regardless of the computer students own.
3. Files, Directories, and File Paths#
Computers organize information using a file system, which manages how data is stored, named, and retrieved. Whether you are working with experimental measurements, Python programs, or scientific images, the information is typically stored in files organized into directories.
A file is a named collection of data, such as a text document, a table of measurements, an image, or a Python program. A directory (often called a folder) is a container that can hold files and other directories. Directories provide a hierarchical structure that allows us to organize related information.
Windows, macOS, and Linux all use hierarchical file systems, but they differ in how file locations are represented. Because we will use Ubuntu through WSL 2 on Windows and a Unix-based environment on macOS, we will focus on Unix-style file paths throughout this course.
3.1 The Directory Hierarchy#
In Unix-like operating systems, the file system is organized as a tree of directories beginning at the root directory, represented by a forward slash (/).
For example, a student’s files might be organized as follows:
/
└── home/
└── student/
└── py4sci/
├── data/
│ └── temperatures.csv
├── notebooks/
│ └── analysis.ipynb
└── scripts/
└── analyze.py
In this example:
/represents the root of the file system./home/studentis the student’s home directory on a typical Ubuntu installation.py4sciis a directory containing files for our Python for Science course.data,notebooks, andscriptsare subdirectories used to organize different types of files.temperatures.csvis a file containing data,analysis.ipynbis a Jupyter notebook, andanalyze.pyis a Python script.
On macOS, a student’s home directory typically begins with /Users/ rather than /home/, but the same hierarchical organization and path conventions apply.
3.2 Absolute and Relative Paths#
A file path identifies the location of a file or directory within the file system. There are two principal ways to specify a path.
An absolute path gives the complete location starting from the root directory. For example:
/home/student/py4sci/data/temperatures.csv
A relative path specifies a location relative to the current working directory.
If our current working directory is /home/student/py4sci, we can refer to the same file using:
data/temperatures.csv
Relative paths are particularly useful in scientific programming because they allow projects to be moved between computers without requiring changes to every file reference.
Two special directory references are frequently used:
.represents the current directory...represents the parent directory, one level above the current directory.
For example, ../data/temperatures.csv refers to a file in a directory named data located one level above the current directory.
3.3 The Home Directory and Working Directory#
Every user generally has a home directory, which serves as the default location for personal files, documents, and projects. In Unix-like environments, the symbol ~ is commonly used as shorthand for the current user’s home directory.
For example:
~/py4sci
refers to the py4sci directory inside the user’s home directory.
The current working directory is the directory from which a program or command is operating. It determines how relative file paths are interpreted.
This distinction is especially important when working with Python and Jupyter notebooks. A program may fail to locate a data file if its working directory is different from the directory you expected.
3.4 Why File Paths Matter in Scientific Computing#
Suppose you have collected temperature measurements and saved them in a CSV file. A Python program might load those measurements using:
import pandas as pd
data = pd.read_csv("data/temperatures.csv")
This example assumes that the current working directory contains a subdirectory named data, which contains the file temperatures.csv.
Python uses the path you provide to locate the data file. If the file is stored elsewhere, or the program is running from a different working directory, the file may not be found.
In the next notebook, we will learn to navigate the file system using terminal commands, create directories for our projects, and locate files using absolute and relative paths.
4. Memory and Storage#
Computers need to store both the instructions that tell them what to do and the data they process. However, not all information is stored in the same way. Two important components are random-access memory (RAM) and persistent storage, such as solid-state drives (SSDs) and hard disk drives (HDDs).
The fundamental difference between them is whether information is retained when the computer is turned off. This distinction is described using the terms volatile and nonvolatile memory.
4.1 Volatile Memory: RAM#
Random-access memory (RAM) is the computer’s primary working memory. It temporarily holds the instructions and data that running programs need to access.
RAM is considered volatile because its contents are lost when the computer is powered off. It is designed for rapid access, allowing the CPU to retrieve and modify information much faster than it typically could from a storage drive.
For example, suppose you are using Python to analyze a CSV file containing thousands of temperature measurements. The file is initially stored on your computer’s SSD. When Python reads the file, the data is loaded into RAM, where it can be processed, analyzed, and used to generate graphs.
RAM is a limited resource. If you attempt to load a dataset that is too large for the available memory, your program may slow down considerably or fail to execute.
4.2 Nonvolatile Memory: Persistent Storage#
Nonvolatile memory retains information even when electrical power is removed. Computers use persistent storage devices to preserve operating systems, applications, documents, and scientific datasets.
Two common storage technologies are:
Solid-state drives (SSDs): Store information electronically using flash memory. They have no moving mechanical parts and are generally faster than traditional hard drives.
Hard disk drives (HDDs): Store information magnetically on rotating disks. They are generally slower than SSDs but can provide large storage capacities at relatively low cost.
Unlike RAM, the contents of persistent storage remain available after the computer is turned off or restarted.
For scientific computing, this means that experimental measurements, Python scripts, and Jupyter notebooks can be saved and retrieved during future computing sessions.
4.3 How Memory and Storage Work Together#
When you run a Python program, the operating system loads the instructions and necessary data from storage into RAM. The CPU executes the program’s instructions, accessing and modifying data in memory as needed.
Consider a simple scientific workflow:
A dataset containing environmental measurements is saved on an SSD.
A Python program reads the dataset from storage into RAM.
The CPU executes instructions to calculate averages, identify trends, or generate graphs.
The results can be saved back to the SSD for future use.
This illustrates an important distinction: data being processed in RAM is not necessarily saved to persistent storage.
For example, if you modify a dataset in Python but do not save the changes to a file, those modifications may be lost when the Python program ends or the computer is restarted.
This distinction is particularly important when working in Jupyter notebooks. Saving a notebook preserves its code, markdown, and saved outputs, but it does not automatically preserve every variable or dataset currently held in the Python kernel’s memory.
5. Programs and Processes#
When you open an application, such as a web browser or JupyterLab, your computer begins executing instructions that tell it what to do. In computing, it is important to distinguish between a program and a process.
A program is a collection of instructions that can be executed by a computer. Programs are typically stored as files on persistent storage, such as an SSD. Examples include web browsers, Python interpreters, and applications used to analyze scientific data.
A process is a running instance of a program. When a program is executed, the operating system creates a process and allocates resources, including memory and CPU time, so that its instructions can be carried out.
For example, a Python script saved as analyze.py is a file containing program instructions. When you execute that script using Python, the operating system runs a Python interpreter process that reads and executes those instructions.
5.1 Running Multiple Processes#
Modern operating systems support multitasking, allowing multiple processes to execute concurrently. For example, you might have a web browser open while Python analyzes a dataset and another application downloads scientific measurements.
Each process has its own execution state and generally operates within a protected memory space. The operating system manages these processes, allocating CPU time and other resources as needed.
On a computer with multiple CPU cores, different processes may execute instructions simultaneously. Even on a single-core processor, the operating system can rapidly switch between processes, creating the appearance that several programs are running at once.
5.2 Processes in Scientific Computing#
Consider working with a Jupyter notebook to analyze experimental measurements. Several components may be operating simultaneously:
JupyterLab server: A running process that provides the JupyterLab interface and manages notebook sessions.
Python kernel: A separate process that executes Python code and maintains variables and other data in memory.
Web browser: An application that displays the JupyterLab interface and allows you to interact with notebooks.
When you execute a code cell in a Jupyter notebook, the browser sends the request through the JupyterLab infrastructure to the Python kernel, which executes the code and returns the results.
An important consequence of this arrangement is that the notebook interface and the Python kernel are separate components. Closing a browser tab does not necessarily stop the Python kernel, and restarting the kernel clears its Python execution state without deleting the notebook file.
We will explore this relationship in greater detail when we begin using JupyterLab.
6. Graphical Interfaces, Terminals, and Shells#
Operating systems provide different ways for users to interact with computers. Most people are familiar with graphical interfaces, where programs are launched by clicking icons and files are managed through windows and menus. However, computers can also be controlled by entering text commands.
Both approaches allow users to accomplish many of the same tasks, but they differ in how instructions are communicated to the operating system.
6.1 Graphical User Interfaces (GUIs)#
A graphical user interface (GUI) allows users to interact with a computer through visual elements such as windows, icons, menus, and buttons. Examples include the Windows desktop, macOS Finder, and file managers found in Linux distributions.
GUIs make many computing tasks intuitive. For example, you can create a directory, rename a file, or launch an application by clicking the appropriate controls.
Scientific software also frequently provides graphical interfaces. Applications such as spreadsheet programs, molecular visualization tools, and integrated development environments (IDEs) allow users to analyze data, create visualizations, and develop programs through interactive interfaces.
JupyterLab, which we will use extensively in this course, provides a graphical interface that runs in a web browser. It allows us to create and edit notebooks, execute Python code, view results, and manage files within a common workspace.
6.2 Terminals and Shells#
A terminal is an application that provides a text-based interface through which users can enter commands and view their results. Rather than clicking through menus, users type instructions to perform tasks such as navigating directories, managing files, and running programs.
A shell is a program that interprets those commands and launches the requested operations.
Although the terms terminal and shell are sometimes used interchangeably, they refer to different components:
Terminal: The application or interface in which commands are entered and their output is displayed.
Shell: The program that interprets the commands and initiates the requested actions.
Common shells include:
Bash (Bourne Again Shell): Widely used on Linux systems, including Ubuntu.
Zsh (Z Shell): The default interactive shell on modern macOS systems.
PowerShell: A command-line shell and scripting environment commonly used on Windows.
For example, entering the following command in a Bash or Zsh shell displays the current working directory:
pwd
The shell interprets the command and runs the appropriate utility, which obtains the current directory information and displays the result in the terminal.
In the next notebook, we will learn to use terminal commands to navigate directories, create files, and prepare our computing environment.
6.3 Why Use a Command-Line Interface?#
A command-line interface (CLI) provides a way to interact with a computer through text commands rather than graphical controls.
Although graphical interfaces are convenient for many everyday tasks, command-line tools offer several advantages in scientific computing.
Automation: A sequence of commands can be saved in a script and executed repeatedly, reducing the need for manual interaction.
Reproducibility: Commands can be documented and shared, making it easier for others to repeat a computational procedure.
Remote computing: Many scientific servers and high-performance computing systems are accessed remotely through command-line interfaces rather than graphical desktops.
Software management: Many scientific programming tools, including Python package managers and development utilities, provide command-line interfaces for installing, configuring, and running software.
Graphical interfaces and command-line tools are not competing approaches. Scientists frequently use both, choosing whichever is best suited to the task.
For example, we will use terminal commands to set up Python environments and launch JupyterLab, while using JupyterLab’s graphical interface to write Python code, analyze scientific datasets, and generate visualizations.
7. Why Scientists Use Unix-Like Environments#
Python runs on Windows, macOS, and Linux, and many scientific applications work equally well on all three operating systems. So why are we using Unix-like computing environments in this course?
One reason is that Linux is widely used in scientific research, particularly on servers, cloud computing platforms, and high-performance computing (HPC) systems. Many scientific software packages and computational workflows are developed and tested in Linux environments.
Unix-like operating systems also provide a collection of command-line tools that make it convenient to manage files, execute programs, automate repetitive tasks, and work with large datasets. These tools can be combined into scripts that perform sequences of operations, making computational workflows easier to reproduce and share.
Another important advantage is consistency. Although Windows and macOS differ in how they manage software and interact with their underlying operating systems, we can use many of the same Unix commands and development tools on both platforms.
In this course:
Windows users will install Ubuntu Linux within Windows Subsystem for Linux (WSL 2).
macOS users will use the Unix-based environment already provided by their operating system.
Linux users can work directly within their existing Linux environment.
These environments are not identical, but they share many common conventions and tools. This will allow us to follow similar instructions for managing files, installing Python packages, and running scientific programs.
Learning to work in a Unix-like environment also provides skills that transfer beyond this course. Whether you are analyzing data on your laptop, accessing a remote scientific database, or running calculations on a university computing cluster, many of the same commands and concepts will apply.
8. Introducing Python and Its Computing Environments#
Python is a free, open-source programming language widely used in scientific research, engineering, data analysis, and software development. Scientists use Python to analyze experimental measurements, generate visualizations, perform simulations, access scientific databases, and communicate with instruments and sensors.
One of Python’s greatest strengths is its extensive collection of scientific packages, which provide tools for tasks ranging from numerical calculations to machine learning and molecular modeling.
Before we begin working with Python, however, it is important to distinguish between the Python programming language, the software that executes Python code, and the computing environments in which that software runs.
8.1 What Is Python?#
A programming language is a formal system for writing instructions that a computer can execute. Python provides a relatively readable syntax that makes it accessible to beginners while remaining powerful enough for complex scientific applications.
For example, the following Python code calculates the average of five temperature measurements:
temperatures = [21.3, 22.1, 20.8, 23.0, 21.7]
average = sum(temperatures) / len(temperatures)
print(average)
The result is:
21.78
This example illustrates how Python can store data, perform calculations, and display results using only a few lines of code.
However, the code itself does not execute automatically. It must be processed by software called a Python interpreter.
The Python interpreter is a program that executes Python instructions. When you run a Python script or execute a cell in a Jupyter notebook, a Python interpreter carries out the requested operations.
This introduces an important distinction:
Python the language defines the syntax and behavior of Python programs.
A Python interpreter is the software that executes those programs.
A computer can have more than one Python interpreter installed, and different interpreters may use different versions of Python or have access to different scientific packages.
8.2 Python Packages#
Python includes a collection of built-in functions and modules, often called its standard library, that support common programming tasks.
However, many scientific applications require additional capabilities. These are provided by packages developed by the Python community.
Some widely used scientific packages include:
Package |
Scientific Applications |
|---|---|
NumPy |
Numerical calculations and multidimensional arrays |
pandas |
Organizing and analyzing tabular data |
Matplotlib |
Creating graphs and scientific visualizations |
SciPy |
Scientific algorithms, statistics, and numerical methods |
scikit-learn |
Machine learning and predictive modeling |
RDKit |
Cheminformatics and molecular structure analysis |
Packages allow scientists to build upon software developed by others rather than writing every algorithm from scratch.
For example, pandas can read measurements stored in a CSV file and organize them into a table for analysis.
Different projects may require different packages or versions of those packages. Managing these dependencies is one reason we use separate Python computing environments.
8.3 Python Installations and Environments#
One potentially confusing aspect of scientific programming is that a single computer may contain several separate Python installations.
For example, after setting up Ubuntu and Miniconda, you may encounter three general categories:
1. System Python
Linux operating systems such as Ubuntu often include a Python interpreter installed as part of the operating system. It may be used by system utilities and other software.
On Ubuntu, this interpreter is commonly located at:
/usr/bin/python3
Because system tools may depend on this installation, we generally avoid modifying it or installing scientific packages directly into its system-managed directories.
2. Conda Base Python
When we install Miniconda, it creates a default environment called base. This environment includes a Python interpreter and the Conda software used to manage environments and packages.
For example, if Miniconda is installed in a user’s home directory, its Python interpreter may be located at:
~/miniconda3/bin/python
The base environment is primarily used for managing Conda and creating other environments. We will generally avoid installing our scientific project packages directly into base.
3. Project-Specific Python Environments
Conda allows us to create separate environments, each containing its own Python interpreter and collection of packages.
For example, we might create an environment named py4sci containing Python, NumPy, pandas, Matplotlib, and other packages needed for this course.
Another project might require different packages or even a different version of Python.
These environments are isolated from one another, reducing the likelihood that installing or updating software for one project will interfere with another.
Importantly, these are not three different Python programming languages. They are different installations or environments that use the same Python language, potentially with different versions and packages.
8.4 Conda and Environment Management#
Conda is a software tool used to manage packages and computing environments. It allows us to create separate environments, install software into those environments, and control which versions of Python and its packages are available.
We will use Miniconda, a minimal distribution that provides Conda, Python, and the basic tools needed to manage environments.
For example, a scientific computing project might require Python and packages such as NumPy and pandas. Rather than installing everything into the operating system’s Python installation, we can create a dedicated environment containing those dependencies.
This approach provides several advantages:
Isolation: Packages installed for one project do not normally interfere with packages used by another.
Compatibility: Different environments can use different Python and package versions.
Reproducibility: Environment specifications can be shared so that others can recreate similar computing environments.
Maintenance: Environments can be updated, replaced, or removed without modifying the operating system’s Python installation.
In later notebooks, we will learn how to install Miniconda, create environments, activate them, and install scientific packages.
8.5 JupyterLab, Notebooks, and Kernels#
Although Python programs can be written as text files and executed from a terminal, we will do much of our work in JupyterLab, an interactive development environment that runs in a web browser.
JupyterLab allows us to work with Jupyter notebooks, which combine executable code, explanatory text, mathematical equations, tables, and visualizations in a single document.
This makes notebooks particularly useful for scientific computing because they allow us to document not only the results of an analysis but also the procedures used to obtain them.
A Jupyter notebook is typically saved as a file with the extension .ipynb.
However, JupyterLab itself does not execute Python code. When a code cell is executed, the instructions are sent to a separate process called a Python kernel.
The kernel runs the code using a Python interpreter and maintains the variables and data created during the computing session.
An important feature of Jupyter is that the kernel can use a Python interpreter from an environment different from the environment running JupyterLab.
For example, we may install JupyterLab in an environment named jbenv, while using a kernel associated with our py4sci environment to execute scientific Python code.
This separation allows us to maintain JupyterLab independently from the packages used in individual projects.
In subsequent notebooks, we will configure these environments and learn how to launch JupyterLab, select a Python kernel, and begin writing scientific programs.