Managing Software and Code

Managing Software and Code#

Role: Researcher / Data Analyst and Data Engineer

The Secure Data Enclave (SDE) provides approved software, programming languages, development tools, and packages for research and analysis. SDE VMs do not have internet access. Software and package installation is controlled by Northwestern IT.

Licensed software, such as Stata or MATLAB, may be considered for installation when there is an approved research need. The Data Engineer has the permissions to install approved software, but no installation should occur without Northwestern IT approval and coordination.

Code and other files entering or leaving the SDE must follow the approved ingress and egress processes. SDE systems log ingress and data movement activity. The research group is accountable for unauthorized ingress associated with their environment.

Action

Responsibility

Use software and packages already available in the SDE

Researcher / Data Analyst and Data Engineer

Request licensed software such as Stata or MATLAB

Researcher / Data Analyst or Data Engineer

Approve and coordinate software installation

Northwestern IT

Install approved licensed software

Data Engineer, with IT approval and coordination

Ingress approved code or other files

Data Engineer through the approved ingress process

Develop and modify code in the SDE

Researcher / Data Analyst and Data Engineer

Export code from the SDE

Data Engineer through the approved egress process

Warning

Ingressing packages into the SDE is not an approved method for adding R or Python packages to the environment.

The following workflows apply throughout the SDE:

Activity

Approved workflow

Install licensed software

Request → IT review and approval → approved files enter through ingress → malware scan → Data Engineer installs with IT coordination

Bring code into the SDE

Approved ingress source → Data Engineer ingresses files → malware scan → files move to the appropriate SDE project

Export code from the SDE

Researcher or Data Engineer prepares code → approved egress process → Data Engineer completes transfer → code is downloaded after approval

Important

All ingress and data movement activity is logged. The research group is responsible for ensuring that only authorized files and data enter or leave its environment.

Use software already available#

Role: Researcher / Data Analyst and Data Engineer

SDE VMs include approved software, programming languages, R and Python packages, and development tools.

See Available Software in the SDE to find out what is already available.

If the software or package you need is already available, you can use it without submitting a request.

If an R or Python package is not available, do not attempt to bring it into the SDE. Request the package by submitting a request ticket to Northwestern IT.

Request licensed software#

Role: Researcher / Data Analyst and Data Engineer

Licensed software such as Stata or MATLAB may be considered for installation when it is needed for approved research.

Before submitting a request, confirm that the software:

  • Supports Ubuntu Linux Operating System

  • Can operate without direct internet access

  • Meets the CPU, memory, and storage requirements of the target VM

  • Has any required license or activation information available

Check the VM’s configuration by following these steps.

Submit a software request#

Role: Researcher / Data Analyst and Data Engineer

Submit a request through the resource request process.

Include:

  • Software name

  • Version, if applicable

  • Link to the vendor’s official page

  • Why the software is needed

  • Which SDE VM requires the software

  • License or activation requirements

  • Any known Ubuntu or system requirements

IT will review the request and determine whether the software can be approved for use in the SDE.

Important

Approval is required before the software is installed. The Data Engineer may have the technical permissions needed to perform the installation, but those permissions do not replace IT approval.

The same approval requirement applies when installing software on the Data Ops VM or a Workspace Project VM.

Provide installation files when requested#

Role: Researcher / Data Analyst and Data Engineer

If IT requests an installer or other installation file, follow the instructions in the request ticket.

Do not install the software yourself.

The Data Engineer will use the approved ingress process to bring the required files into the SDE.

After the files have completed the required ingress process and IT has approved the installation, the Data Engineer may install the software on the approved VM.

Warning

Do not proceed with an installation based only on a software request, a downloaded installer, or your access to the VM.

The Data Engineer’s ability to install software is a technical capability, not an authorization to install it. Wait for IT approval and coordination.

Ingress code into the SDE#

Role: Data Engineer

The Data Engineer can ingress approved code and other files into the SDE.

The SDE does not provide direct access to GitHub, public package repositories, or other external code repositories. Code from an external source must enter through the approved ingress process.

The Data Engineer can use the Data Ingress Project and the approved ingress infrastructure to bring files into the SDE. Approved code may then be moved to the appropriate project, such as the Data Ops Project or a Workspace Project.

Request a new ingress source#

Role: Researcher / Data Analyst and Data Engineer

If code needs to come from a source that is not already configured for ingress, submit an ingress request ticket.

The source must be approved and configured before the Data Engineer uses it to bring files into the SDE.

Danger

Do not download, clone, copy, or otherwise bring code or other files into the SDE outside the approved ingress process.

This rule applies even when the Data Engineer has access to the target VM or has the technical ability to transfer files directly.

Ingress and data movement activity is logged. The research group is accountable for unauthorized ingress associated with its project if identified during an audit.

Request architecture changes for software#

Role: Researcher / Data Analyst and Data Engineer

The default SDE VM configuration denies inbound and outbound connections. Some software may require network access for licensing, authentication, APIs, or other external services.

Any software installation that changes the underlying SDE architecture requires Information Security Office (ISO) approval. Examples include:

  • Opening ports or allowing internet access

  • Modifying firewall rules

  • Allowing software to retrieve tokens, authentication, or licensing information

  • Allowing connections to external APIs, services, or portals

These requests are handled by Northwestern IT. The request and approval are documented in a ticket for audit purposes.

Provide information for ISO review#

Role: Researcher / Data Analyst and Data Engineer

When requesting software that requires an architecture change, provide enough information for IT and ISO to complete their review without additional follow-up.

For example, provide:

  • Purpose: What will the library be used for?

  • External service: What service or portal will it access, and what will it retrieve?

  • Connection direction: Inbound, outbound, or bidirectional?

  • Data leaving the enclave: What information will be sent outside the SDE?

  • Data retrieved: What information will be retrieved?

  • Data sensitivity: Will any restricted or sensitive data leave the SDE?

Also provide the software/package name and version, target VM, external service or domain, required ports or protocols, and authentication or licensing requirements when applicable.

Important

The research group’s responsibility is to provide complete and accurate information for the request. Northwestern IT handles the technical review, ISO approval, and any resulting architecture changes.

Work with code in the SDE#

Role: Researcher / Data Analyst and Data Engineer

Once the approved code is inside the SDE, users can develop and modify it on the appropriate VM.

Researcher / Data Analysts generally work in their Workspace Project VMs.

Data Engineers perform processing, cleaning, and analysis work in the Data Ops VM.

You can:

  • Create and modify code on the appropriate VM

  • Use Git for local version control

  • Store working repositories on the VM

  • Save important code to an approved project bucket

Important

Keep important code in an approved project bucket rather than relying on the VM as the only copy. Code stored only on a VM may be lost if the VM is deleted.

Export code from the SDE#

Role: Researcher / Data Analyst and Data Engineer

Code can leave the SDE only through the approved egress process.

Step

Role

Action

1

Researcher / Data Analyst

Place the code in the designated Egress Dataprep bucket in the Data Lake Project.

2

Researcher / Data Analyst

Submit an egress request through the egress process.

3

Data Engineer

Review the request and complete the required approval and transfer process.

4

Data Engineer

Transfer approved code to the Data Egress Project.

5

Researcher / Data Analyst

Download the approved code to your managed endpoint.

Warning

Do not download, copy, or otherwise remove code directly from an SDE VM.

This applies to both Workspace Project VMs and the Data Ops VM. Code must leave the SDE through the approved egress process.