Skip to content

Commit b2bae9b

Browse files
igawigaw
committed
docs: add howto to setup TLS connections
There are couple of step necessary to get TLS working nicely. Document it how this is done. Signed-off-by: Daniel Wagner <wagi@kernel.org>
1 parent 39b1db8 commit b2bae9b

File tree

1 file changed

+280
-0
lines changed

1 file changed

+280
-0
lines changed

‎TLS.md

+280
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
# How to Set Up TLS for NVMe-TCP
2+
3+
Enabling TLS for the NVMe-TCP transport requires a few configuration
4+
steps for both the kernel and userland.
5+
6+
## Kernel Configuration
7+
8+
To support TCP authentication and TLS encryption, enable the following
9+
kernel options:
10+
11+
- For DHCHAP authentication:
12+
`CONFIG_NVME_HOST_AUTH`
13+
14+
- For TLS transport encryption:
15+
`CONFIG_NVME_TCP_TLS`
16+
17+
These configuration option depend on another config option but these
18+
will be auto selected.
19+
20+
## Userland
21+
22+
For the userland configuration two components need to be configured.
23+
First, the tlshd TLS handshake daemon needs to be running and TLS keys
24+
need to be loaded into the kernel keystore.
25+
26+
### Setting Up `tlshd`
27+
28+
For TLS protocol support, which handles authentication and encryption,
29+
the kernel handles data encryption only, so userland support is required
30+
for the TLS handshake. The `tlshd` daemon implements the handshake
31+
process.
32+
33+
#### Requirements
34+
35+
Ensure `tlshd` includes the commit `311d9438b984` ("tlshd: always link
36+
.nvme default keyring into the session") - likely in `ktls-utils` version
37+
0.12. Alternatively, you can set the keyring manually in
38+
`/etc/tlshd.conf`:
39+
40+
```ini
41+
[authenticate]
42+
keyrings = .nvme
43+
```
44+
45+
#### Enable/start tlshd
46+
47+
No additional configuration is necessary for `tlshd`; simply start it as
48+
a daemon:
49+
50+
```bash
51+
systemctl enable --now tlshd
52+
```
53+
54+
### Loading Keys on Boot or Module Load
55+
56+
When the kernel is establishing a TCP connection with TLS, the NVMe
57+
subsystem loads keys from the kernel keystore. This means these keys
58+
must be available in the keystore before establishing a connection.
59+
60+
nvme-cli provides command line interfaces to create, import and export
61+
keys into the kernel keystore. Though it's not the only way to
62+
import/export keys. If there is another system component managing the
63+
keys, the following steps for creating and making the keys persistent
64+
over boot cycles are not necessary.
65+
66+
To stress this point, the nvme-cli is explicitly trying to avoid handling
67+
the keys, the only requirement is that the keys are present in the
68+
keystore.
69+
70+
#### Creating a New Key
71+
72+
```bash
73+
nvme gen-tls-key \
74+
--hostnqn nqn.2014-08.org.nvmexpress:uuid:befdec4c-2234-11b2-a85c-ca77c773af36 \
75+
--subsysnqn nqn.io-1 --hmac 1 --identity 1 --insert --keyfile /etc/nvme/tls-keys
76+
```
77+
78+
This command creates a new host key, inserts it into the kernel keyring,
79+
and appends the derived TLS PSK to the keyfile (`/etc/nvme/tls-keys`).
80+
81+
### Inserting an Existing Key
82+
83+
```bash
84+
nvme check-tls-key \
85+
--hostnqn nqn.2014-08.org.nvmexpress:uuid:befdec4c-2234-11b2-a85c-ca77c773af36 \
86+
--subsysnqn nqn.io-1 --identity 1 \
87+
--keydata NVMeTLSkey-1:01:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACtVQoZ: \
88+
--insert --keyfile /etc/nvme/tls-keys
89+
```
90+
91+
This command inserts the configured key (`--keydata`) into the kernel
92+
keyring and appends the derived TLS PSK to the keyfile.
93+
94+
### Loading Keys on Boot or Module Load
95+
96+
The kernel keyring does not persist keys, so userland must import keys
97+
into the keyring upon each boot or module load (for NVMe-TCP). The
98+
nvme-tcp module provides the `psk` type keystore, thus only when the
99+
nvme-tcp module is available it possible to load keys into the keystore:
100+
101+
```bash
102+
nvme tls --import --keyfile /etc/nvme/tls-keys
103+
```
104+
105+
The `70-nvmf-keys.rules` udev rule
106+
([source](https://github.com/linux-nvme/nvme-cli/blob/master/nvmf-autoconnect/udev-rules/70-nvmf-keys.rules.in))
107+
will load keys from `/etc/nvme/tls-keys` automatically.
108+
109+
```udev
110+
ACTION=="add", SUBSYSTEM=="module", KERNEL=="nvme_tcp", TEST=="@SYSCONFDIR@/tls-keys", RUN+="@SBINDIR@/nvme tls --import --keyfile @SYSCONFDIR@/tls-keys"
111+
```
112+
113+
### Recommendation for Handling TLS Keys
114+
115+
The `nvme connect` command also allows passing a TLS key directly via the
116+
command line or a JSON config file. Avoid this method in production
117+
environments, as it may expose keys.
118+
119+
### Establishing a Connection
120+
121+
Once the keys are in the keystore, add the `--tls` option to establish a
122+
secure connection:
123+
124+
```bash
125+
nvme connect --transport tcp --traddr 192.168.154.148 --trsvcid 4420 \
126+
--hostnqn nqn.2014-08.org.nvmexpress:uuid:befdec4c-2234-11b2-a85c-ca77c773af36 \
127+
--hostid befdec4c-2234-11b2-a85c-ca77c773af36 \
128+
--nqn nqn.io-1 --tls --dump-config --output-format json
129+
```
130+
131+
The resulting JSON output can be saved to simplify future connections:
132+
133+
```json
134+
[
135+
{
136+
"hostnqn": "nqn.2014-08.org.nvmexpress:uuid:befdec4c-2234-11b2-a85c-ca77c773af36",
137+
"hostid": "befdec4c-2234-11b2-a85c-ca77c773af36",
138+
"subsystems": [
139+
{
140+
"nqn": "nqn.io-1",
141+
"ports": [
142+
{
143+
"transport": "tcp",
144+
"traddr": "192.168.154.148",
145+
"trsvcid": "4420",
146+
"dhchap_key": "none",
147+
"tls": true
148+
}
149+
]
150+
}
151+
]
152+
}
153+
]
154+
```
155+
156+
Using this JSON file, you can connect with:
157+
158+
```bash
159+
nvme connect --config config.json
160+
```
161+
162+
## Setting Up the Target
163+
164+
The same steps for creating keys and importing/exporting keys to/from the
165+
kernel are necessary for the target as they are for the host (see above).
166+
167+
For the above example, you can use the `nvmetcli` config:
168+
169+
```json
170+
{
171+
"hosts": [
172+
{
173+
"nqn": "nqn.2014-08.org.nvmexpress:uuid:befdec4c-2234-11b2-a85c-ca77c773af36"
174+
}
175+
],
176+
"ports": [
177+
{
178+
"addr": {
179+
"adrfam": "ipv4",
180+
"traddr": "0.0.0.0",
181+
"treq": "not specified",
182+
"trsvcid": "4420",
183+
"trtype": "tcp",
184+
"tsas": "tls1.3"
185+
},
186+
"ana_groups": [
187+
{
188+
"ana": {
189+
"state": "optimized"
190+
},
191+
"grpid": 1
192+
}
193+
],
194+
"param": {
195+
"inline_data_size": "16384",
196+
"pi_enable": "0"
197+
},
198+
"portid": 0,
199+
"referrals": [],
200+
"subsystems": [
201+
"nqn.io-1"
202+
]
203+
}
204+
],
205+
"subsystems": [
206+
{
207+
"allowed_hosts": [
208+
"nqn.2014-08.org.nvmexpress:uuid:befdec4c-2234-11b2-a85c-ca77c773af36"
209+
],
210+
"attr": {
211+
"allow_any_host": "0",
212+
"cntlid_max": "65519",
213+
"cntlid_min": "1",
214+
"firmware": "6.8.0-rc",
215+
"ieee_oui": "0x000000",
216+
"model": "Linux",
217+
"pi_enable": "0",
218+
"qid_max": "128",
219+
"serial": "0c74361069d9db6c65ef",
220+
"version": "1.3"
221+
},
222+
"namespaces": [
223+
{
224+
"ana": {
225+
"grpid": "1"
226+
},
227+
"ana_grpid": 1,
228+
"device": {
229+
"nguid": "00000000-0000-0000-0000-000000000000",
230+
"path": "/dev/vdb",
231+
"uuid": "91fdba0d-f87b-4c25-b80f-db7be1418b9e"
232+
},
233+
"enable": 1,
234+
"nsid": 1
235+
}
236+
],
237+
"nqn": "nqn.io-1"
238+
}
239+
]
240+
}
241+
```
242+
243+
## Debugging tips
244+
245+
- Increase the debug log output in tlshd:
246+
```ini
247+
[debug]
248+
loglevel=9
249+
```
250+
251+
- To verify if any key is present you can look at the `/proc/keys` output:
252+
```bash
253+
cat /proc/keys | grep -i nvme
254+
```
255+
256+
- The keys description is the key identifier and is defined in the TCP
257+
transport specification (see the 'TLS PSK and PSK Identity Derivation'
258+
section). The format is `NVMe<version>R<hmac> <hostnqn> <subsynqn> <PSK digest>`
259+
260+
- The exported keys in the /etc/nvme/tls-keys file are one per line and
261+
the lines are formatted as `<identity> <PSK in interchange format>`. The
262+
`<PSK>` is the derive TLS PSK and not the retained nor the configured PSK.
263+
264+
- If several keys available in the keystore which match up to the `<PSK digest>`
265+
the first match will be used. If this is the wrong key, it can be revoked by
266+
```bash
267+
nvme tls --revoke <identity>
268+
```
269+
270+
- It's possible to provide a TLS key directly via the `nvme connect --tls
271+
--tls-key` command. If only the key is provided, nvme-cli assumes it is a
272+
configured PSK and thus does all the key transformation and creates the
273+
identity automatically. If the `--tls-key-identity` is also present
274+
nvme-cli assumes it is a derived TLS PSK and does not attempt
275+
transformation on it and inserts the key directly into the keystore.
276+
277+
- When the `nvme connect --tls-key` command is used, the `-vv` options
278+
will show the connect arguments passed to the kernel, including the key
279+
id numbers. These are in hex format and match with the output from
280+
`/proc/keys`.

0 commit comments

Comments
 (0)