Add Opentofu Worker #2

agrare · 2024-03-04T19:33:52Z

Adds an opentofu worker using the embedded_terraform role, note this will be the first non-ruby worker we'll have 🎉

TODO:

systemd service definition
check miq_worker status set to started
Pass environment variables for memcached / postgresql
Provide db password via podman/k8s secret
miq_worker record deleted when worker exits Mark non-rails workers as stopped manageiq#22941
podified deployment definition Updates for OpenTofu runner manageiq-pods#1067

Dependents:

rpm_spec updates Add opentofu-runner service manageiq-rpm_build#450

agrare · 2024-03-06T21:47:07Z

irb(main):001:0> OpentofuWorker.count
=> 1
irb(main):002:0> OpentofuWorker.first
=> 
#<OpentofuWorker:0x00007efe07752b08
 id: 33,
 guid: "30747dc1-40ae-4d3c-a3fd-29169d079f81",
 status: "started",
 started_on: nil,
 stopped_on: nil,
 last_heartbeat: Wed, 06 Mar 2024 21:45:41.655961000 UTC +00:00,
 pid: nil,
 queue_name: nil,
 type: "OpentofuWorker",
 percent_memory: 0.08,
 percent_cpu: 0.02,
 cpu_time: 1470.0,
 os_priority: 20,
 memory_usage: 300175360,
 memory_size: 486223872,
 uri: nil,
 miq_server_id: 1,
 sql_spid: nil,
 proportional_set_size: 285480000,
 unique_set_size: 284116000,
 system_uid: "opentofu-runner.service">

[root@manageiq-devel vmdb]# systemctl status opentofu-runner
● opentofu-runner.service
     Loaded: loaded (/usr/lib/systemd/system/opentofu-runner.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/opentofu-runner.service.d
             └─override.conf
     Active: active (running) since Wed 2024-03-06 16:45:43 EST; 35s ago
   Main PID: 9108 (ruby)
      Tasks: 1 (limit: 23130)
     Memory: 18.2M (high: 1.0G available: 1005.7M)
        CPU: 87ms
     CGroup: /system.slice/opentofu-runner.service
             └─9108 /usr/bin/ruby -e sleep

Mar 06 16:45:43 manageiq-devel.localdomain systemd[1]: Started opentofu-runner.service.

~~Need to handle systemd heartbeat:~~ Switching to Type=simple fixes this. We'll need to adjust when the actual loopback API is running.

app/models/opentofu_worker.rb

agrare · 2024-03-18T16:26:48Z

app/models/opentofu_worker.rb

+  def container_image_namespace
+    ENV["CONTAINER_IMAGE_NAMESPACE"]
+  end


This exists in the base class already:
https://github.com/ManageIQ/manageiq/blob/master/app/models/miq_worker/container_common.rb#L92-L94

app/models/opentofu_worker.rb

* [draft] add spec for .create_in_provider * Add EmbeddedTerraform Provider class * add ensure_managers * fixed .create_in_provider test * Add more tests * Use basename of dir, as template-name pre-fix, & full git repo url details as suffix * Add more test - templates-in-repo & update-in-provider

…t_source_and_payload Add configuration script source and payload

agrare · 2024-04-04T14:47:41Z

Okay I have the opentofu-runner.service being started by EvmServer via the OpentofuWorker class, passing the database password via a secret (we can add whatever other secret info we want here database password just a proof of concept)

[root@manageiq-devel vmdb]# systemctl status opentofu-runner.service
● opentofu-runner.service
     Loaded: loaded (/usr/lib/systemd/system/opentofu-runner.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-04-04 10:44:30 EDT; 32s ago
    Process: 11438 ExecStartPre=/bin/rm -f /tmp/opentofu-runner.service.cid (code=exited, status=0/SUCCESS)
   Main PID: 11439 (podman)
      Tasks: 28 (limit: 23124)
     Memory: 83.9M
        CPU: 2.709s
     CGroup: /manageiq.slice/opentofu-runner.service
             ├─ 6254 catatonit -P
             ├─11439 /usr/bin/podman run --conmon-pidfile /tmp/opentofu-runner.pid --cidfile /tmp/opentofu-runner.cid --cgroup-manager=cgroupfs --cgroups=no-conmon --log-driver=journald --root=/var/www/miq/vmdb/data/conta>
             ├─11448 /usr/bin/podman run --conmon-pidfile /tmp/opentofu-runner.pid --cidfile /tmp/opentofu-runner.cid --cgroup-manager=cgroupfs --cgroups=no-conmon --log-driver=journald --root=/var/www/miq/vmdb/data/conta>
             ├─11645 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 -e 4 --netns-type=path /tmp/podman-run-987/netns/netns-863ba440-0042-3999-b3b4-ec904858>
             ├─11648 /usr/bin/conmon --api-version 1 -c c276456a86cc1b9172db81003a15605d7458e9a9f9310010c8f8c142264e2a6b -u c276456a86cc1b9172db81003a15605d7458e9a9f9310010c8f8c142264e2a6b -r /usr/bin/crun -b /var/www/miq>
             └─11650 ruby /usr/src/app/./opentofu-runner.rb

Apr 04 10:44:32 manageiq-devel.localdomain podman[11448]: Copying blob sha256:5f74a64ac7702f1a3cf514af2d28600c186751c703bb08396d01631924d2b5d0
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: Copying config sha256:4c26793dbb863d854d7a16a2bb1a31fdc0f9ec49705b508a211ed12eb90f7bc3
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: Writing manifest to image destination
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: 2024-04-04 10:44:41.350721535 -0400 EDT m=+10.487458035 image pull 4c26793dbb863d854d7a16a2bb1a31fdc0f9ec49705b508a211ed12eb90f7bc3 docker.io/agrare/opentofu-runne>
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: 2024-04-04 10:44:41.536073634 -0400 EDT m=+10.672810125 container create c276456a86cc1b9172db81003a15605d7458e9a9f9310010c8f8c142264e2a6b (image=docker.io/agrare/o>
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: 2024-04-04 10:44:41.69561559 -0400 EDT m=+10.832352100 container init c276456a86cc1b9172db81003a15605d7458e9a9f9310010c8f8c142264e2a6b (image=docker.io/agrare/open>
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: 2024-04-04 10:44:41.700127055 -0400 EDT m=+10.836863545 container start c276456a86cc1b9172db81003a15605d7458e9a9f9310010c8f8c142264e2a6b (image=docker.io/agrare/op>
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: 2024-04-04 10:44:41.732550647 -0400 EDT m=+10.869287157 container attach c276456a86cc1b9172db81003a15605d7458e9a9f9310010c8f8c142264e2a6b (image=docker.io/agrare/o>
Apr 04 10:44:41 manageiq-devel.localdomain opentofu-runner[11648]: {"DATABASE_PASSWORD":"smartvm"}
Apr 04 10:44:41 manageiq-devel.localdomain podman[11448]: {"DATABASE_PASSWORD":"smartvm"}

agrare · 2024-04-04T14:49:24Z

For the systemd service I broadly followed https://www.redhat.com/sysadmin/podman-run-pods-systemd-services without some of the complexity of creating a Pod plus a Container

Fryguy · 2024-05-02T20:07:01Z

systemd/opentofu-runner.service

+User=manageiq
+Group=manageiq
+ExecStartPre=/bin/rm -f /tmp/%n.cid
+ExecStart=/usr/bin/podman run --conmon-pidfile %T/%N.pid --cidfile %T/%N.cid --cgroup-manager=cgroupfs --cgroups=no-conmon --log-driver=journald --replace --name=opentofu-runner --secret=opentofu-runner-secret --root=/var/www/miq/vmdb/data/containers/storage docker.io/agrare/opentofu-runner:latest


Suggested change

ExecStart=/usr/bin/podman run --conmon-pidfile %T/%N.pid --cidfile %T/%N.cid --cgroup-manager=cgroupfs --cgroups=no-conmon --log-driver=journald --replace --name=opentofu-runner --secret=opentofu-runner-secret --root=/var/www/miq/vmdb/data/containers/storage docker.io/agrare/opentofu-runner:latest

ExecStart=/usr/bin/podman run --conmon-pidfile %T/%N.pid --cidfile %T/%N.cid --cgroup-manager=cgroupfs --cgroups=no-conmon --log-driver=journald --replace --name=opentofu-runner --secret=opentofu-runner-secret --root=/var/www/miq/vmdb/data/containers/storage docker.io/manageiq/opentofu-runner:latest

miq-bot · 2024-05-02T20:11:00Z

Checked commits agrare/manageiq-providers-embedded_terraform@fc97155~...4e6881a with ruby 2.7.8, rubocop 1.56.3, haml-lint 0.51.0, and yamllint
2 files checked, 0 offenses detected
Everything looks fine. 🏆

agrare · 2024-05-02T20:12:57Z

TODO in a follow-up, check Terraform::Runner.available? and prevent the worker from starting up if it isn't available

agrare · 2024-05-02T20:15:42Z

TODO check if you can use an Environment Variable in an ExecStart to change the runner image label

agrare changed the title ~~Add Opentofu Worker~~ [WIP] Add Opentofu Worker Mar 4, 2024

Fryguy added the enhancement New feature or request label Mar 6, 2024

Fryguy assigned bdunne Mar 6, 2024

agrare mentioned this pull request Mar 12, 2024

Mark non-rails workers as stopped ManageIQ/manageiq#22941

Merged

1 task

agrare force-pushed the add_opentofu_worker branch from 447f3cf to b816f10 Compare March 14, 2024 16:34

agrare mentioned this pull request Mar 14, 2024

Add opentofu-runner service ManageIQ/manageiq-rpm_build#450

Merged

1 task

agrare force-pushed the add_opentofu_worker branch 2 times, most recently from 90a00db to 0ece888 Compare March 15, 2024 15:45

agrare commented Mar 15, 2024

View reviewed changes

app/models/opentofu_worker.rb Outdated Show resolved Hide resolved

agrare commented Mar 18, 2024

View reviewed changes

app/models/opentofu_worker.rb Outdated Show resolved Hide resolved

agrare commented Mar 18, 2024

View reviewed changes

app/models/opentofu_worker.rb Outdated Show resolved Hide resolved

agrare commented Mar 18, 2024

View reviewed changes

app/models/opentofu_worker.rb Show resolved Hide resolved

bdunne reviewed Mar 19, 2024

View reviewed changes

app/models/opentofu_worker.rb Outdated Show resolved Hide resolved

Fryguy mentioned this pull request Mar 26, 2024

Create an Embedded Terraform provider ManageIQ/manageiq#22956

Closed

40 tasks

jrafanie pushed a commit to jrafanie/manageiq-providers-embedded_terraform that referenced this pull request Mar 27, 2024

Merge pull request ManageIQ#2 from Adam-Grare/add_configuration_scrip…

65e1fe4

…t_source_and_payload Add configuration script source and payload

agrare force-pushed the add_opentofu_worker branch from 5dd21af to 3b99c79 Compare March 28, 2024 18:30

agrare and others added 7 commits April 3, 2024 13:49

Add opentofu worker

fc97155

Switch opentofu-runner to Type=simple

e1cfe5d

Add environment_variables for memcached/psql

2bc61fb

Embedding terraform runner image to the deployment

1645f17

Remove methods duplicated in base class

01dee92

Remove extra newline at end of class

fad16c3

Move maximum_workers_count to other class vars

c6eb7af

agrare force-pushed the add_opentofu_worker branch 2 times, most recently from 4177da6 to 2889114 Compare April 3, 2024 19:22

agrare force-pushed the add_opentofu_worker branch from 126c614 to 82e1ab5 Compare April 4, 2024 14:52

miq-bot added the wip label Apr 25, 2024

Fryguy reviewed May 2, 2024

View reviewed changes

agrare added 6 commits May 2, 2024 16:08

Add podman run to the opentofu-runner.service

58ab9e8

Create a podman secret when starting systemd unit

70b2a90

Add an OpentofuWorker::Runner

e9ec41f

Add --cgroup-manager=cgroupfs to fix warning

1b9ec65

Create podman secret as manageiq user

72e896b

Use data/containers/storage for image/secret

744b05b

Fryguy approved these changes May 2, 2024

View reviewed changes

agrare added 3 commits May 2, 2024 16:10

Use --replace when starting container

14c0841

Drop unit_config_file override

c03b925

Replace unit_environment_variables with hash

4e6881a

agrare force-pushed the add_opentofu_worker branch from 0f32a3a to 4e6881a Compare May 2, 2024 20:10

agrare changed the title ~~[WIP] Add Opentofu Worker~~ Add Opentofu Worker May 2, 2024

miq-bot removed the wip label May 2, 2024

Fryguy merged commit c386ccd into ManageIQ:master May 2, 2024
4 checks passed

agrare deleted the add_opentofu_worker branch May 2, 2024 20:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Opentofu Worker #2

Add Opentofu Worker #2

agrare commented Mar 4, 2024 •

edited by Fryguy

Loading

agrare commented Mar 6, 2024 •

edited

Loading

agrare Mar 18, 2024

agrare commented Apr 4, 2024

agrare commented Apr 4, 2024

Fryguy May 2, 2024

miq-bot commented May 2, 2024

agrare commented May 2, 2024

agrare commented May 2, 2024

	ExecStart=/usr/bin/podman run --conmon-pidfile %T/%N.pid --cidfile %T/%N.cid --cgroup-manager=cgroupfs --cgroups=no-conmon --log-driver=journald --replace --name=opentofu-runner --secret=opentofu-runner-secret --root=/var/www/miq/vmdb/data/containers/storage docker.io/agrare/opentofu-runner:latest
	ExecStart=/usr/bin/podman run --conmon-pidfile %T/%N.pid --cidfile %T/%N.cid --cgroup-manager=cgroupfs --cgroups=no-conmon --log-driver=journald --replace --name=opentofu-runner --secret=opentofu-runner-secret --root=/var/www/miq/vmdb/data/containers/storage docker.io/manageiq/opentofu-runner:latest

Add Opentofu Worker #2

Add Opentofu Worker #2

Conversation

agrare commented Mar 4, 2024 • edited by Fryguy Loading

agrare commented Mar 6, 2024 • edited Loading

agrare Mar 18, 2024

Choose a reason for hiding this comment

agrare commented Apr 4, 2024

agrare commented Apr 4, 2024

Fryguy May 2, 2024

Choose a reason for hiding this comment

miq-bot commented May 2, 2024

agrare commented May 2, 2024

agrare commented May 2, 2024

agrare commented Mar 4, 2024 •

edited by Fryguy

Loading

agrare commented Mar 6, 2024 •

edited

Loading