From time to time, you may need to get one or more files into or out of the department. This file transfer can be done in a few different ways, some secure, and some not. The recommended method of transferring a file is to either use an encrypted mechanism, or one that does not require you to enter a username and password. The preferred method is to use scp / sftp, which is part of the SSH suite of programs. SSH stands for Secure SHell, and can be obtained from a number of places, including the CS Software site, and www.ssh.com. Other alternatives include rsync or anonymous FTP. The rsync service requires that you contact csstaff@cs.princeton.edu to assist in setting up an rsync module, after which you can choose to either publish or receive data, or both. The FTP service also requires a directory be created by csstaff@cs.princeton.edu, at which point you can receive anonymous incoming data.
High Speed Transfers with Globus
For cases where a large volume of data needs to be moved into or out of the CS Department, CS Staff manages a Globus Data Transfer Node with 10Gbps networking that can be used to schedule and manage the transfer of datasets to or from other Globus DTNs.
All CS account holders have access to the CS Department DTN.
To get started, go to Globus.org, and click on the Log In link. You'll be prompted to login using "your existing organizational login". In the lookup box, locate Princeton University and click Continue. This should redirect you to Princeton/OIT CAS, where you should login using your main campus credentials.
Once authenticated, you'll be taken to the Transfer Files page. On this page, select the two endpoints of your transfer. These can be any Globus DTNs to which you have access, either at Princeton or at other institutions. To select an endpoint, click in the Endpoint section and, in the search box that pops up, start typing an identifier (try "Princeton"). The CS Department project space endpoint is listed as Princeton Computer Science POSIX Project Spaces.
Once you select an endpoint, you may be prompted to authenticate to that endpoint, as you will be for the CS Department DTN. Use your OIT credentials to authenticate to Globus, after which you should see a view of the files available for transfer. Click in the Path bar to provide a different filesystem location. Follow the above steps for both endpoints of the transfer, select your Transfer Settings at the bottom of the page, and you're ready to start the transfer.
The CS Department also has an endpoint listed as Princeton Computer Science AWS S3 Gateway. As its name implies, this endpoint can be used to setup transfers between your AWS S3 data and other Globus endpoints, typically CS Project Space. Upon selecting the S3 Gateway, you will be prompted to configure the gateway by adding S3 credentials with access to the buckets you wish to access.
For further information, please see the Globus Getting Started page.
If you have trouble using the CS DTN, or have any questions, please reach out to CS Staff for assistance.
SCP / SFTP
scp is the secure copy program. sftp is the secure file transfer program. They are similar to the BSD rcp and ftp programs, except that they send everything, including authentication credentials, over an encrypted channel, preventing eavesdroppers from capturing your username and password, and using them to break into your account.
Details on using scp or sftp under unix, linux, or MacOS X can be found on their man pages. In general, scp or sftp can be used to connect to any host on which you are able to login with ssh (e.g. portal.cs.princeton.edu).
rsync Service
rsync is an open source utility that provides fast incremental file transfer. As a service in this department, it is neither encrypted nor strongly authenticated. However, it can be used for both publishing and receiving data with some simple (insecure) password protection. The CS Department rsync service offers three main options for how to publish your module:
- Read-only Anonymous - this option allows you to publish data to the whole world with no access restriction.
- Read-only Password-protected - this option allows you to publish your data protected by a password. This is not a secure or encrypted password or connection, and should therefore not be used with any password that you use elsewhere.
- Read-write Password-protected - this option is similar to the previous, except you can allow others who have the password to write data as well as reading it. To reiterate, this is not a secure or encrypted password or connection.
To see which (public) modules are currently available on our server, try this command, which will produce a list of available modules:
> rsync rsync://mirror.cs.princeton.edu/ slackware Slackware Linux Mirror centos CentOS Linux Mirror fedora Fedora Linux Mirror fedora-epel Fedora EPEL Mirror redhat Fedora EPEL Mirror
To use or view any of these modules, simply drill down deeper:
> rsync rsync://mirror.cs.princeton.edu/centos/6/ drwxrwxr-x 191 2011/09/22 20:10:03 . lrwxrwxrwx 18 2011/09/22 14:36:49 centosplus lrwxrwxrwx 15 2011/09/22 14:36:49 contrib lrwxrwxrwx 10 2011/09/23 09:37:52 cr lrwxrwxrwx 14 2011/09/22 14:36:49 extras lrwxrwxrwx 17 2011/09/22 14:36:49 fasttrack lrwxrwxrwx 12 2011/09/22 14:36:49 isos lrwxrwxrwx 9 2011/09/22 14:36:49 os lrwxrwxrwx 15 2011/09/22 14:36:49 updates
Before making extensive use of rsync, we strongly recommend you play around with it in a safe area to become familiar with its eccentricities. Thorough documentation is available at the rsync web page and in the rsync(1) man page.
Anonymous FTP
The other method of getting files into or out of the CS department is to use anonymous FTP, so called because, when prompted for a user name, the word anonymous is used. Typically, the user's email address is used for the password, so that no private information (such as a real password) is revealed. There is a command line ftp program available for both unix and MS-Windows systems. Alternatively, you can use a web browser, and specify the ftp location as a URL, as in:
ftp://ftp.cs.princeton.edu/
FTP Into the CS Department
In order to use anonymous FTP to copy files into the CS department from the outside, you need to request that csstaff@cs.princeton.edu set up an incoming FTP directory for you. The directory will have the same name as your user ID. From any department unix machine, you can access the directory with the path /n/fs/ftp/incoming/$USER. To access the directory from a web browser, go to ftp://ftpupload.cs.princeton.edu/userID, where userID is your unix user name. To access the directory from a traditional (command line) FTP client, connect to ftpupload.cs.princeton.edu and 'cd' to userID.
FTP From the CS Department
To get files from the CS department using anonymous FTP from a machine outside of the department, you need to access the pub directory on our server: ftp.cs.princeton.edu. From a web browser, you would use ftp://ftp.cs.princeton.edu/pub/. From a traditional (command line) FTP program, you would connect to ftp.cs.princeton.edu and 'cd' to /pub.
Of course, in order for somebody to get a file from the /pub directory, it first needs to be put there by somebody within the CS department. You can do this on a department unix machine by copying your file(s) to the directory /n/fs/ftp/pub.