Python-based Websites Using WSGI

The Web Server Gateway Interface (WSGI) is a standard interface between web server software and web applications written in Python. By using this standard, it will be easy to migrate your site to a different hosting platform should that become necessary.

This document describes one way to set up a Python web application running under Phusion Passenger using WSGI on the CS infrastructure. This mechanism will work with Python web frameworks such as Django, Flask, Bottle, or Pyramid.

All command line instructions below are to be done on a "cycles" machine.

Contents

Security Considerations

File Permissions

The webserver will run your Python code with your UID/NetID (i.e., "as you") and with a GID/group (to which you belong) of your choosing. This has the benefit that you can include secrets/credentials in files that are not world-readable. A consideration to remember is that because the webserver will run "as you", your running code will have access to any project space files (in any project) that you have access.

Note that before running "as you," the webserver is running as a user/group that can only read certain configuration/control files if they are world-readable. These files are noted in the documentation below.

Environment Variables

If your shell is bash, all the environment variables you set up in your .bashrc and .bash_profile will be available to your running Python code. (See: the default "on" configuration of PassengerLoadShellEnvvars.) This can obviously be handy; however, there are some downsides. First, from a security perspective, environment variables containing secrets can be exposed to other running processes on the shared webserver. Second, from a configuration management perspective, having some of the configuration for your site be in your personal .bashrc file in your home directory and the rest of the configuration in the project space for the site may lead to issues when you are ready to hand the reins over to another person.

Before the webserver runs your .bashrc and .bash_profile, it will set the environment variable IN_PASSENGER. If you want the webserver to effectively skip your normal Bash configuration, you can put this at the top of your .bashrc and .bash_profile files:

if [ -n "$IN_PASSENGER" ]; then
    umask 027  # no group-write or world-anything; adjust to taste
    return 0
fi

As an alternative, you can pass environment variables and secrets to the webserver application, you can use the [environment] and [secret_files] sections in the config.toml file as described below.

Requesting Resources / Initial Set-up

The step-by-step instructions in this section assume you are starting from scratch and running commands from a "cycles" machine. If you already have some of the resources that you wish to use, adapt these instructions as appropriate. As always, you can reach out to csstaff@cs.princeton.edu for guidance.

Note: In the CS infrastructure, you don't have direct access to web server logs for debugging. Therefore, it's important to be methodical and test along the way. The wsgi-shim package can catch common configuration mistakes and give feedback without access to the web server logs. Also, in a pinch, you can reach out to CS Staff and ask about errors in the log for your site if you get stuck. Be sure to give a precise timestamp to help us find your messages within the common web server log.

1. Request "project disk space"

Request disk space to put the website files and a new POSIX group for the files.

In the "Request Forms" block on the CS Guide, click on "Project Disk Space: new space request". The key fields are:

Additional people: Create new POSIX group with me as member

Requested Partition Name: myproject

Note that the "Additional people" field is free form where you can request a POSIX group. In general, the name of the new POSIX group will match the name of the partition.

2. Verify/set permissions on the project space

After receiving notification that the space has been created, verify that it exists with the correct group and permissions.

Note that project space is "auto-mounted" and will exist on a given machine only after it is accessed. Using ls does not count as accessing. Using cd will access the space.

Verify project space permissions:

cd /n/fs/myproject  # to ensure the space is mounted
ls -ld /n/fs/myproject

The output should look like something like this:

drwxr-sr-x. 4 mynetid mygroup 59 Sep 20 13:24 /n/fs/myproject

If the owner is not mynetid (i.e., your NetID), contact CS Staff.

The group should not be your default group (i.e., the one displayed when you run the id -gn command). If it is your default group, contact CS Staff.

If the permissions are not drwxr-sr-x (note the lowercase "s"), change them with chmod 2755 /n/fs/myproject. The lowercase "s" in this position is the SETGID bit and ensures that files and subdirectories created under /n/fs/myproject will belong to the mygroup group by default.

3. Build Python

Build a local version of Python.

These are abbreviated instructions from the CS Guide: Building a Local Version of Python. For example, to build Python 3.12.0 and install it in your project space at /n/fs/myproject/python-3.12.0, use these commands (adjusting PROJECT_DIR and PYVER as appropriate):

PROJECT_DIR="/n/fs/myproject"
PYVER="3.12.0"
cd $PROJECT_DIR
mkdir temp
cd temp
wget https://www.python.org/ftp/python/$PYVER/Python-$PYVER.tar.xz
tar xJf Python-$PYVER.tar.xz
cd Python-$PYVER
./configure --prefix=$PROJECT_DIR/python-$PYVER
make
make install
cd $PROJECT_DIR
rm -rf temp

Test the Python installation:

/n/fs/myproject/python-3.12.0/bin/python3 --version

Output should be:

Python 3.12.0
4. Create Website Directory and Build Python Virtual Environment

Create the directory to hold the files associated with your website. This directory must be world-readable but not world-writeable:

mkdir /n/fs/myproject/mywebsitefiles
chmod o=rx /n/fs/myproject/mywebsitefiles

Build the Python virtual environment for your website in, for example, venv.

cd /n/fs/myproject/mywebsitefiles
/n/fs/myproject/python-3.12.0/bin/python3 -m venv venv

Note: This is the Python virtual environment that will be used for the running website. This is, for example, where you would "pip install" packages such as Flask or Django.

5. Install wsgi-shim and Create Initial Configuration

Activate your virtual environment, install the wsgi-shim package, and populate the initial PassengerAppRoot directory:

cd /n/fs/myproject/mywebsitefiles
source venv/bin/activate
pip install --upgrade pip
pip install wsgi-shim
wsgi-shim install --puppet /n/fs/myproject/mywebsitefiles
deactivate

The --puppet option will print configuration information that you will need when you request "project web space" in the next step. (The option is called "puppet" because the format is suitable for including in a Puppet definition of the web space and Puppet is used by CS Staff to automate the configuration of the CS infrastructure.) The output you will need will look like this:

Requesting these attributes:
passenger_enabled     => true,
passenger_user        => 'mynetid',
passenger_group       => 'mygroup',
passenger_app_root    => '/n/fs/myproject/mywebsitefiles/www-approot',
passenger_restart_dir => '/n/fs/myproject/mywebsitefiles/www-approot/tmp',
passenger_app_type    => 'wsgi',
passenger_python      => '/n/fs/myproject/mywebsitefiles/venv/bin/python';

Your project space directory structure will look like this:

/n/fs/myproject
+-- python-3.12.0/
|   +-- ...
+-- mywebsitefiles/
    +-- venv/
    |   +-- ...
    +-- www-approot/
    |   +-- config.toml
    |   +-- passenger_wsgi.py
    |   +-- tmp/
    |       +-- maint.txt
    |       +-- restart.txt
    +-- www-docroot/
6. Request "project web space"

With the directory structure created, it is time to request "project web space". This will configure the CS web server to recognize mywebsite.cs.princeton.edu as a valid website and route its requests to your Python code via the provided passenger_wsgi.py file.

You will need the key/value pairs in the [passenger] section of the /n/fs/myproject/mywebsitefiles/config.toml file created in the previous step to populate the "Additional Notes" section in the form below.

In the "Request Forms" block on the CS Guide, click on "Project Web Space". The key fields are:

Project Name: mywebsite

Location of Document Root: /n/fs/myproject/mywebsitefiles/www-docroot

Group Ownership: mygroup

Additional Notes:

Requesting these attributes:
passenger_enabled     => true,
passenger_user        => 'mynetid',
passenger_group       => 'mygroup',
passenger_app_root    => '/n/fs/myproject/mywebsitefiles/www-approot',
passenger_restart_dir => '/n/fs/myproject/mywebsitefiles/www-approot/tmp',
passenger_app_type    => 'wsgi',
passenger_python      => '/n/fs/myproject/mywebsitefiles/venv/bin/python';

Warning: The original Document Root directory must not be deleted without coordinating with CS Staff (even if you recreate it with the same name).

7. Test Initial Configuration

Tip: Do not continue with testing until you have received confirmation from CS Staff that this is ready. DNS resolvers sometimes perform "negative caching" where they remember that a host name is not found for a period of time before attempting another lookup. Premature testing may require that you wait until a timeout elapses.

When you get verification from CS Staff that the project web space is set up, verify that the DNS name for the website is in place:

host mywebsite.cs.princeton.edu

The output should look something like this:

mywebsite.cs.princeton.edu is an alias for wwwprx.cs.princeton.edu.
wwwprx.cs.princeton.edu has address 128.112.136.67

If you get a response indicating "not found," wait an hour and try again. If at that point you are still getting a "not found" indication, follow-up with CS Staff.

Open a browser and visit https://mywebsite.cs.princeton.edu

You should see a "Maintenance" page. Now, disable maintenance mode and force a restart of your site:

  1. Run: rm /n/fs/myproject/mywebsitefiles/www-approot/tmp/maint.txt
  2. Run: touch /n/fs/myproject/mywebsitefiles/www-approot/tmp/restart.txt
  3. Reload the site a few times (waiting several seconds between reloads) until it shows updated content.

You should see a "Hello, World" page. If so, congratulations! You're ready to start installing and developing code as described in the next section.

If you are not getting the "Hello, World" page, carefully review your steps and also refer to the Troubleshooting section, below.

Install / Develop Site-Specific Code

Directory Structure

Other than being readable by the user/group specified in the config.toml file, the webserver doesn't put specific restrictions on where in project space the code implementing your web application must go. One possibility is the following layout:

/n/fs/myproject
+-- python-3.12.0/
|   +-- ...
+-- mywebsitefiles/
    +-- log/
    |   +-- logfile
    +-- myapp/
    |   +-- file1.py
    |   +-- file2.py
    |   +-- ...
    +-- secrets/
    |   +-- credentials.env
    +-- venv/
    |   +-- ...
    +-- www-approot/
    |   +-- config.toml
    |   +-- passenger_wsgi.py
    |   +-- tmp/
    |       +-- restart.txt
    +-- www-docroot/

In this example, the secrets directory contains credentials and, as such, should not be world-readable; the myapp directory contains the Python code making up the web application. The myapp directory might reasonably be a checked-out Git repository of your code.

The next section describes how to edit your config.toml file to point to your application in myapp.

Configuration Settings

The config.toml file created by the wsgi-shim install contains the documentation on available settings. Modify it to meet your needs. Here is an example using the directory structure above with the application callable defined in file1.py:

[passenger]
web_doc_root = "/n/fs/myproject/mywebsitefiles/www-docroot"
user = "mynetid"
group = "projectgroup"
passenger_app_root = "/n/fs/myproject/mywebsitefiles/www-approot"
passenger_restart_dir = "/n/fs/myproject/mywebsitefiles/www-approot/tmp"
passenger_python = "/n/fs/myproject/mywebsitefiles/venv/bin/python"
[wsgi]
chdir = "/n/fs/myproject/mywebsitefiles/myapp"
app = "file1.application"
[secret_files]
SECRETS = "/n/fs/myproject/mywebsitefiles/secrets/credentials.env"
[environment]
LOG_FILENAME = "/n/fs/myproject/mywebsitefiles/log/logfile"
Logging

In the CS infrastructure, you won't have ready access to the webserver logs; therefore, it is important to set up logging as early as possible in your application. The Flask example in the next section demonstrates logging.

Flask Example

This is a simple "Hello, Flask World" Flask-based site that includes logging to a local file in your project space.

1. Create a directory to store your logs
mkdir /n/fs/myproject/mywebsitefiles/log
chmod go= /n/fs/myproject/mywebsitefiles/log
2. Create a directory to store your Python code
mkdir /n/fs/myproject/mywebsitefiles/myapp
chmod go= /n/fs/myproject/mywebsitefiles/myapp
3. Create your Flask app

Create a file in /n/fs/myproject/mywebsitefiles/myapp/file1.py with the following contents:

# /n/fs/myproject/mywebsitefiles/myapp/file1.py
import logging.config
import os

from flask import Flask

# Configure logging before instantiating Flask app
if LOG_FILENAME := os.environ.get('LOG_FILENAME'):
    logging.config.dictConfig({
        'version': 1,
        'formatters': {
            'default': {
                'format': '%(asctime)s %(levelname)s %(name)s '
                          '%(threadName)s : %(message)s',
            }
        },
        'handlers': {
            'file': {
                'class': 'logging.handlers.RotatingFileHandler',
                'formatter': 'default',
                'filename': LOG_FILENAME,
                'maxBytes': 40000,
                'backupCount': 2,
            }
        },
        'root': {
            'level': 'INFO',
            'handlers': ['file'],
        }
    })

app = Flask(__name__)


@app.route("/")
def hello_world():
    app.logger.info('Request: /')
    return "<p>Hello, Flask World!</p>"
4. Install Flask into your Python virtual environment
source /n/fs/myproject/mywebsitefiles/venv/bin/activate
pip install Flask
deactivate
5. Edit your configuration

Your config.toml file should be:

[passenger]
web_doc_root = "/n/fs/myproject/mywebsitefiles/www-docroot"
user = "mynetid"
group = "projectgroup"
passenger_app_root = "/n/fs/myproject/mywebsitefiles/www-approot"
passenger_restart_dir = "/n/fs/myproject/mywebsitefiles/www-approot/tmp"
passenger_python = "/n/fs/myproject/mywebsitefiles/venv/bin/python"
[wsgi]
chdir = "/n/fs/myproject/mywebsitefiles/myapp/file1"
app = "fil1.app"
[secret_files]
[environment]
LOG_FILENAME = "/n/fs/myproject/mywebsitefiles/log/logfile"
6. Restart and visit your site

Tell Passenger to restart your application and then visit your web page.

7. View your local log file

After visiting your Flask-based site, you can view your own logs:

cat /n/fs/myproject/mywebsitefiles/log/logfile

and you will see something like this:

2023-10-16 09:13:15,856 INFO file1 MainThread : Request: /
Django Example

A Django example is on its own page here: CS Guide: Setting up a Django Website in the Princeton CS Environment.

Operation

The webserver will start your application when it receives a request to a URL in https://mywebsite.cs.princeton.edu. This process will generally handle multiple requests. After a period of inactivity, the webserver will kill the process until there is another request.

Restarting Your Application

You can force a restart of your application by updating the timestamp of the restart.txt file:

touch /n/fs/myproject/mywebsitefiles/www-approot/tmp/restart.txt

Note that the restart is not immediate and the file is only checked when a URL is visited on your site but no more often than every 10 seconds (the value of PassengerStatThrottleRate in the webserver global configuration)

Using Maintenance Mode

The provided passenger_wsgi.py file (via uwsgi-shim install) implements a mechanism that, when enabled, will not load your website code but, instead, show a "maintenance" page where all requests to the site return a 503 Service Unavailable response. In this mode, your site-specific code (e.g., based on Django, Flask, etc.) will not be accessed.

To enable maintenance mode, create the sentinel file:

touch /n/fs/myproject/mywebsitefiles/www-approot/tmp/maint.txt

To disable maintenance mode, delete the sentinel file:

rm /n/fs/myproject/mywebsitefiles/www-approot/tmp/maint.txt

Note that the existence of this file is only checked when the WSGI server (e.g., Phusion Passenger) restarts the application. Therefore, anytime you enable/disable maintenance mode, you need to force a restart of the application. For example, to update the code for a website, one might follow these steps:

  1. Run: touch /n/fs/myproject/mywebsitefiles/www-approot/tmp/maint.txt
  2. Run: touch /n/fs/myproject/mywebsitefiles/www-approot/tmp/restart.txt
  3. Reload the site a few times until it shows the maintenance page. This can take up to 10 seconds.
  4. Update the site
  5. Run: rm /n/fs/myproject/mywebsitefiles/www-approot/tmp/maint.txt
  6. Run: touch /n/fs/myproject/mywebsitefiles/www-approot/tmp/restart.txt
  7. Reload the site a few times until it shows the updated content.

Note that the restart.txt file must be world-readable. If you delete this file and later recreate it, be sure it is world-readable.

Troubleshooting

Check your own logs

If your site has been working and/or you have successfully set up logging, the first place to check for trouble is your own logs. The Python logging system will generally capture exceptions and leave a traceback in your log.

Check your directory structure

Use the wsgi-shim check command to verify directory structure and permissions and then address any issues reported:

cd /n/fs/myproject/mywebsitefiles
source venv/bin/activate
wsgi-shim check /n/fs/myproject/mywebsitefiles/www-approot
deactivate

If your application writes to the file system (including logs), verify that you haven't run out of space.

Common HTML Responses
"The developer of this site has put it into maintenance mode."

In this case, the web server side of things are working.

See: Using Maintenance Mode

"This site has experienced a developer-side configuration error or exception."

In this case, the web server side of things are working. * Use the wsgi-shim checkcommand (above) to check the directory structure. * Address any issues detailed on the page.

"Hmm. We’re having trouble finding that site."

Check that your website host name is resolvable in DNS:

host mywebsite.cs.princeton.edu

If "not found" and you have recently requested the project web space, wait an hour and try again. Otherwise, contact CS Staff.

"Forbidden - you don't have permission to access this resource."

This can indicate an issue with your docroot directory. If you deleted (or deleted and then recreated) your docroot (i.e., /n/fs/myproject/mywebsitefiles/www), ask CS Staff to check the docroot mount point for your site and force-remount it if necessary.

"We're sorry, but something went wrong."

Verify that your .bashrc and .bash_profile are not exiting when invoked by the webserver (i.e., when IN_PASSENGER is set).

If all else fails, contact CS Staff

If you believe there is a key bit of information in the webserver logs, contact CS Staff and specify the URL that is not working along with a specific timestamp.

/node/6878 built from python-wsgi.md on 2024-01-30 09:56:06

Tags: