Transferring Data to Terra Dotta

In a standard integration, clients use SFTP or SCP to push text files with SIS/HR data to an SSH folder hosted by Terra Dotta. This SSH folder is specific to each client's account. If an institution has multiple accounts (for instance, an account for Study Abroad as well as ISSS), then Terra Dotta will create multiple SSH folders, one for each account.

Terra Dotta uses BitVise WinSSHD as our SSH server. The client may choose any compatible SSH client for their connection.

The process described here is the only way clients should send personally identifying information (PII) to Terra Dotta. Never send this information via email.


Security

The Terra Dotta server that hosts the SSH folders is secured by a firewall. Each client must provide us with a list of no more than four static public IP addresses. These IPs will be added to our whitelist to allow the client's machine through the firewall. Please note our policy prohibits whitelisting a range of IPs.

Past the firewall, each individual SSH folder is protected by a 4096-bit key. As part of the integration process, the client must generate a key pair for their SSH folder and provide Terra Dotta with the public key. The private key must be saved securely in the client's environment. The client must use the private key, instead of a password, when connecting to the SSH folder. Our environment does not allow a shell connection; files may only be transmitted via SFTP or SCP.

SFTP/SCP encrypts the data while it is in transit between the client's environment and the SSH folder.


Automation

Once the final version of the SIS/HR data file(s) have been approved via the integration case, clients must set up an automated job to SFTP/SCP the file to their SSH folder once per day. Terra Dotta creates a timed automated data loader job that picks up the file and loads it into a temporary database, called the SIS database, which is mapped to the client's production TDS site. The SIS database is completely overwritten each time a new data file is loaded. Once loaded in the SIS database, the data is available to create new user profiles or update existing active user profiles in TDS. After the data has been imported, the loader job deletes the SIS/HR data file from the SSH folder.

The SSH folder contains a log file with the result of each day's automated data loader process. If the SIS/HR data file is loaded successfully, the log records the number of data rows imported. If on a particular day there is no data file to import, then the loader job does not produce an error; it simply does not update the data in the database, and the log records the fact that no file was received. The loader job will then run again the next day at the regularly scheduled time.

Any changes to the filename, column header names, column/row delimiters, or overall structure of the data file prevents the loader job from importing the file or updating the data, and the log records the error.


Technical instructions

1. Follow the directions below to generate the required 4096-bit key pair. Provide Terra Dotta with the public key and store the private key securely in your environment.

2. Provide Terra Dotta with a list of up to four static public IP addresses for the firewall whitelist. These are the IPs of the machine that will be connecting to the SSH folder.

3. Once Terra Dotta has installed your key and whitelisted your IP(s), test the connection by pushing a test .txt file to your SSH folder, via SFTP or SCP. Be sure to use the private key when connecting. Notify us in the integration case and wait for confirmation that the file was received.

SSH folder details:

  • Use SSH-2
  • Bit length: 4096
  • Username: [communicated via the integration case]
  • SSH server hostname: sftp-us.terradotta.com (sftp-au.terradotta.com for the non-US data center)
  • Port: 22

4. When the connection is established, SFTP/SCP Terra Dotta your SIS/HR data file(s). Do not send these files via email.

5. When the SIS/HR data file format has been finalized and approved via the integration case, create an automated job to SFTP/SCP the data file to the SSH folder each night. Time the job to place the file on the folder before 3 am Eastern time (US).

6. (Deferred fetch integrations only) TDS creates a list of the UUUIDs of active applicants, called the pool file, which is placed on the SSH folder at approximately midnight Eastern time (US) each night. Create a timed job to pick up the pool file so it can be used to generate the custom data file. See more information about deferred fetch integrations here: SIS/HR Data (Study Abroad)