Skip to content

Commit aef3687

Browse files
committed
Add provision support for SwinB model files
1 parent 20ea55d commit aef3687

File tree

8 files changed

+48
-11
lines changed

8 files changed

+48
-11
lines changed

.dockerignore

+1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,5 @@
22
/.workspace-shell-dev
33
/.container_xauth
44
/HoloLens2-ResearchMode-Unity
5+
/model_files
56
/unity

.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
#
44
/.workspace-shell-dev/
55
/.container_xauth
6+
/model_files
67

78
# This .gitignore file should be placed at the root of your Unity project directory
89
#

README.md

+29-4
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,31 @@ report-back to the HoloLens2 platform.
7373
Workspace root: `./ros/`
7474

7575
System requirements
76+
* ansible
7677
* docker
7778
* docker-compose
7879

80+
Some files required from the `https://data.kitware.com` Girder service require
81+
authentication due to their protected nature.
82+
The environment variable `GIRDER_API_KEY` must be defined with a valid API key,
83+
otherwise an authentication token cannot be retrieved.
84+
85+
## Provision Files
86+
External large files should be provisioned by running the ansible tool:
87+
88+
ansible-playbook -i ansible/hosts.yml ansible/provision_files.yml
89+
90+
This may include large files for running the system, like ML model files, or
91+
other files required for building docker images.
92+
93+
This provisioning may require additional configuration and variables set in
94+
your environment in order to satisfy some permissions:
95+
* `GIRDER_API_KEY` will need to be set in order to acquire protected files from
96+
`data.kitware.com`.
97+
98+
The configuration that controls what is staged and where is located
99+
in the `ansible/roles/provision-files/vars/main.yml` file.
100+
79101
## Docker-based Workflow
80102
**Intention**: Use containerization to standardize development and runtime
81103
environment and practice.
@@ -84,9 +106,9 @@ environment and practice.
84106
configurations.
85107

86108
Docker functionality is located under the `./docker/` directory.
87-
* `ros2-base` provides a base environment that supports building and running
88-
our workspace.
89-
* `ros2-workspace-build` provides a build of our workspace.
109+
* `workspace-base-dev` provides a base environment that supports building and
110+
running our workspace.
111+
* `workspace-build` provides a build of our workspace.
90112

91113
### Building Docker Images
92114
Run `./angel-docker-build.sh`.
@@ -103,7 +125,7 @@ The definition of this service is found in the `docker/docker-compose.yml`
103125
configuration.
104126

105127
This will mount the `./ros/` subtree on the host system into the
106-
`/angel_workspace/src` directory in the run container.
128+
`/angel_workspace/src/` directory in the run container.
107129

108130
This "workspace" context additionally mounts build, install and log output
109131
directories to a spot on the host in order to:
@@ -115,6 +137,9 @@ The script `/angel_workspace/workspace_build.sh` is available to run.
115137
This script is used during the image build process, so using this script
116138
ensures that the same build method is performed.
117139

140+
Other directories and files are mounted into the container environment for
141+
development purposes and external file sharing.
142+
118143
This shell will **_NOT_** have the local installation sourced in order to
119144
facilitate further safe build actions.
120145
`ros2 run` actions should be performed in a separate shell from where builds

ansible/roles/provision-files/vars/main.yml

+10-2
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
stage_dirs:
55
# Example
66
#example: some/directory
7-
{} # REMOVE WHEN ADDING ENTRIES
7+
model_files: "model_files"
88

99
basic_downloads:
1010
# Example
@@ -21,7 +21,15 @@ girder_file_downloads:
2121
#- file_id: "111111111111111111111111"
2222
# sha512: "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
2323
# dest: "{{ stage_dirs.build_itt_base }}/some_file.txt"
24-
[] # REMOVE WHEN ADDING LIST CONTENT
24+
25+
# SwinB label list text file.
26+
- file_id: 62cc5fa7bddec9d0c4faa19f
27+
sha512: 3fbf340b9cb87a71dd3df81880b38ee414c47b43fbfb78c37220020fbdb8d0d0ff93d76b079aecd94c10317ad6764590fd2a98849338f7fb46f943825820aaf6
28+
dest: "{{ stage_dirs.model_files }}/swinb_coffee_task_labels.txt"
29+
# SwinB model weights checkpoint
30+
- file_id: 62cc5fbebddec9d0c4faa1b5
31+
sha512: cf991079ea52fdab602d46052925a06a40f60176271e7a098ee34bb9262af72be5f1925014f30faf27a59c1b9b46f4f1d8401add4b6e669b982ca3f5e9f1026b
32+
dest: "{{ stage_dirs.model_files }}/swinb_model_stage_base_ckpt_6.pth"
2533

2634
# List of git repositories to check out at a specific ref and then archive.
2735
# Destination files will be written as GZipped TAR files, so please suffix

docker/docker-compose.yml

+2
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ services:
5757
- ../ros:${ANGEL_WORKSPACE_DIR}/src
5858
- ../tmux:${ANGEL_WORKSPACE_DIR}/tmux
5959
- ../angel_system:${ANGEL_WORKSPACE_DIR}/angel_system
60+
- ../model_files:${ANGEL_WORKSPACE_DIR}/model_files
6061
- ./workspace_build.sh:${ANGEL_WORKSPACE_DIR}/workspace_build.sh
6162
- ./workspace_entrypoint.sh:${ANGEL_WORKSPACE_DIR}/workspace_entrypoint.sh
6263
- ./workspace_setenv.sh:${ANGEL_WORKSPACE_DIR}/workspace_setenv.sh
@@ -65,6 +66,7 @@ services:
6566
- ${WORKSPACE_SHELL_HOST_DIR}/install:${ANGEL_WORKSPACE_DIR}/install
6667
- ${WORKSPACE_SHELL_HOST_DIR}/log:${ANGEL_WORKSPACE_DIR}/log
6768
- ${WORKSPACE_SHELL_HOST_DIR}/stuff:${ANGEL_WORKSPACE_DIR}/stuff
69+
- ${WORKSPACE_SHELL_HOST_DIR}/cache/torch:/root/.cache/torch
6870
# X11 things
6971
- /tmp/.X11-unix:/tmp/.X11-unix
7072
# assume this file exists, should be created before running.

docker/workspace-build/Dockerfile

+1-3
Original file line numberDiff line numberDiff line change
@@ -82,11 +82,9 @@ RUN apt-get -y update \
8282
# Install tracked python dependencies
8383
COPY poetry.lock pyproject.toml "${ANGEL_WORKSPACE_DIR}"/
8484
RUN cd "${ANGEL_WORKSPACE_DIR}" \
85-
# TODO: Re-enable use of hashes once editable/git deps are removed. \
86-
# - Remove `--without-hashes` option. \
8785
# * Poetry extraneously finds matplotlib requiring setuptools-scm.
8886
# theoretically being "fixed" in matplotlib 3.6
89-
&& poetry export --without-hashes --dev -f requirements.txt | \
87+
&& poetry export --dev -f requirements.txt | \
9088
pip3 install --no-deps --user -r /dev/stdin \
9189
&& pip uninstall -y setuptools-scm
9290

model_files/README.rst

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
This directory is a target for provisioned files.
2+
Files you may add here `SHOULD NOT` be checked into Git.

ros/angel_system_nodes/configs/swinb_activity_det_config.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{
22
"type": "angel_system.impls.detect_activities.swinb.swinb_detect_activities.SwinBTransformer",
33
"angel_system.impls.detect_activities.swinb.swinb_detect_activities.SwinBTransformer": {
4-
"checkpoint_path": "angel_system/model_checkpoints/swinb_coffee_tasks.pth",
5-
"labels_file": "angel_system/labels/swinb_coffee_task_labels.txt",
4+
"checkpoint_path": "model_files/swinb_model_stage_base_ckpt_6.pth",
5+
"labels_file": "model_files/swinb_coffee_task_labels.txt",
66
"num_classes": 14,
77
"num_frames": 32,
88
"sampling_rate": 2,

0 commit comments

Comments
 (0)