Configure gem install location via GEM_* (#402)

* Configure gem install location via GEM_*

The previous logic used the `BUNDLE_PATH` environment variable to direct bundler where to install gems. This had the side effect of adding additional paths to the directory structure, so the files wouldn't be in `<layer-path>` they would be in a `<layer-path>/ruby/<major>.<minor>.0/` directory such as `<layer-path>/ruby/3.4.0`. The classic buildpack handles this by shelling out to Ruby https://github.com/heroku/heroku-buildpack-ruby/blob/b3ccc41885135ae495c604a512b523c81241914d/lib/language_pack/ruby.rb#L157. This change diverges and takes an alternative approach, using the `GEM_HOME` and `GEM_PATH` environment variables to configure installation location rather than configuring bundler directly.

When `bundle install` is run, it will install into the first `GEM_PATH` (there can be multiple). This install will be direct (without any additional directories under it). This is what we want. This also allows us to remove `BUNDLE_BIN` which is no longer needed (bundler will install binaries into `GEM_PATH/bin`).

- `BUNDLE_DEPLOYMENT` does more than forces the `Gemfile.lock` to be frozen, it also installs gems into `vendor/bundle`, which we don't want. This has been changed to `BUNDLE_FROZEN=1`.

Unfortunately, removing `BUNDLE_PATH` causes the automatic clean after install logic (`BUNDLE_CLEAN=1`) to error with a warning saying that it is unsafe because there's no explicit path. To work around this, I added a manual call to `bundle clean --force` after the `bundle install`. This requires I also remove the `BUNDLE_CLEAN=1` environment variable.

Because this changes the structure of the underlying gems on disk, I need to clear the cache manually by updating the key to `v2`.

* Clear old files

Gems will now be installed directly into `<layer-path>` instead of `<layer-path>/ruby/X.Y.0`. This mechanism cleans previously installed gems left in the cache by checking the cache key of `v1`. This cache will also be evicted when the application changes Ruby version.

* Apply suggestions from code review

Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com>

---------

Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com>

Author: schneems

buildpacks/ruby/CHANGELOG.md

@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- Gem install behavior and configuration ([#402](https://github.com/heroku/buildpacks-ruby/pull/402))
+  - Gem install path is now configured with `GEM_HOME` and `GEM_PATH` instead of `BUNDLE_PATH`.
+  - Cleaning gems is now accomplished via running `bundle clean --force`. Previously it was accomplished by setting `BUNDLE_CLEAN=1`.
+  - The `BUNDLE_DEPLOYMENT=1` environment variable is changed to `BUNDLE_FROZEN=1`.
+  - The `BUNDLE_BIN` environment variable is no longer set.
+
 ## [5.1.0] - 2025-02-28
 
 ### Changed

buildpacks/ruby/src/layers/bundle_install_layer.rs

@@ -42,7 +42,7 @@ const SKIP_DIGEST_ENV_KEY: &str = "HEROKU_SKIP_BUNDLE_DIGEST";
 /// A failsafe, if a programmer made a mistake in the caching logic, rev-ing this
 /// key will force a re-run of `bundle install` to ensure the cache is correct
 /// on the next build.
-pub(crate) const FORCE_BUNDLE_INSTALL_CACHE_KEY: &str = "v1";
+pub(crate) const FORCE_BUNDLE_INSTALL_CACHE_KEY: &str = "v2";
 
 pub(crate) fn handle<W>(
     context: &libcnb::build::BuildContext<RubyBuildpack>,
@@ -99,6 +99,15 @@ where
                     fun_run::map_which_problem(error, cmd.mut_cmd(), env.get("PATH").cloned())
                 })
                 .map_err(RubyBuildpackError::BundleInstallCommandError)?;
+
+            bullet
+                .time_cmd(
+                    Command::new("bundle")
+                        .args(["clean", "--force"])
+                        .env_clear()
+                        .envs(&env),
+                )
+                .map_err(RubyBuildpackError::BundleInstallCommandError)?;
         }
         InstallState::Skip(checked) => {
             let bundle_install = style::value("bundle install");
@@ -144,6 +153,7 @@ pub(crate) struct MetadataV2 {
 }
 
 #[derive(Deserialize, Serialize, Debug, Clone, Eq, PartialEq, CacheDiff, TryMigrate)]
+#[cache_diff(custom = clear_v1)]
 #[try_migrate(from = MetadataV2)]
 #[serde(deny_unknown_fields)]
 pub(crate) struct MetadataV3 {
@@ -171,6 +181,14 @@ pub(crate) struct MetadataV3 {
     pub(crate) digest: MetadataDigest, // Must be last for serde to be happy https://github.com/toml-rs/toml-rs/issues/142
 }
 
+fn clear_v1(_new: &Metadata, old: &Metadata) -> Vec<String> {
+    if &old.force_bundle_install_key == "v1" {
+        vec!["Internal gem directory structure changed".to_string()]
+    } else {
+        Vec::new()
+    }
+}
+
 #[derive(thiserror::Error, Debug)]
 pub(crate) enum MetadataMigrateError {
     #[error("Could not migrate metadata {0}")]
@@ -252,20 +270,14 @@ fn layer_env(layer_path: &Path, app_dir: &Path, without_default: &BundleWithout)
         .chainable_insert(
             Scope::All,
             ModificationBehavior::Override,
-            "BUNDLE_PATH", // Directs bundler to install gems to this path.
+            "GEM_HOME", // Tells bundler where to install gems, along with GEM_PATH
             layer_path,
         )
-        .chainable_insert(
-            Scope::All,
-            ModificationBehavior::Override,
-            "BUNDLE_BIN", // Install executables for all gems into specified path.
-            layer_path.join("bin"),
-        )
         .chainable_insert(Scope::All, ModificationBehavior::Delimiter, "GEM_PATH", ":")
         .chainable_insert(
             Scope::All,
             ModificationBehavior::Prepend,
-            "GEM_PATH", // Tells Ruby where gems are located. Should match `BUNDLE_PATH`.
+            "GEM_PATH", // Tells Ruby where gems are located.
             layer_path,
         )
         .chainable_insert(
@@ -283,13 +295,7 @@ fn layer_env(layer_path: &Path, app_dir: &Path, without_default: &BundleWithout)
         .chainable_insert(
             Scope::All,
             ModificationBehavior::Override,
-            "BUNDLE_CLEAN", // After successful `bundle install` bundler will automatically run `bundle clean`
-            "1",
-        )
-        .chainable_insert(
-            Scope::All,
-            ModificationBehavior::Override,
-            "BUNDLE_DEPLOYMENT", // Requires the `Gemfile.lock` to be in sync with the current `Gemfile`.
+            "BUNDLE_FROZEN", // Requires the `Gemfile.lock` to be in sync with the current `Gemfile`.
             "1",
         );
     // CAREFUL: Changes to these ^^^^^^^ environment variables
@@ -306,14 +312,7 @@ fn display_name(cmd: &mut Command, env: &Env) -> String {
     fun_run::display_with_env_keys(
         cmd,
         env,
-        [
-            "BUNDLE_BIN",
-            "BUNDLE_CLEAN",
-            "BUNDLE_DEPLOYMENT",
-            "BUNDLE_GEMFILE",
-            "BUNDLE_PATH",
-            "BUNDLE_WITHOUT",
-        ],
+        ["BUNDLE_FROZEN", "BUNDLE_GEMFILE", "BUNDLE_WITHOUT"],
     )
 }
 
@@ -439,12 +438,10 @@ mod test {
 
         let actual = commons::display::env_to_sorted_string(&env);
         let expected = r"
-BUNDLE_BIN=layer_path/bin
-BUNDLE_CLEAN=1
-BUNDLE_DEPLOYMENT=1
+BUNDLE_FROZEN=1
 BUNDLE_GEMFILE=app_path/Gemfile
-BUNDLE_PATH=layer_path
 BUNDLE_WITHOUT=development:test
+GEM_HOME=layer_path
 GEM_PATH=layer_path
         ";
         assert_eq!(expected.trim(), actual.trim());
@@ -510,6 +507,27 @@ platform_env = "c571543beaded525b7ee46ceb0b42c0fb7b9f6bfc3a211b3bbcfe6956b69ace3
         let deserialized: Metadata = toml::from_str(&toml_string).unwrap();
 
         assert_eq!(metadata, deserialized);
+
+        let old = Metadata {
+            ruby_version: ResolvedRubyVersion("3.5.3".to_string()),
+            os_distribution: OsDistribution {
+                name: "ubuntu".to_string(),
+                version: "20.04".to_string(),
+            },
+            cpu_architecture: "amd64".to_string(),
+            force_bundle_install_key: "v1".to_string(),
+            digest: MetadataDigest::new_env_files(
+                &context.platform,
+                &[&context.app_path.join("Gemfile")],
+            )
+            .unwrap(),
+        };
+
+        let diff = old.diff(&old);
+        assert_eq!(
+            vec!["Internal gem directory structure changed".to_string()],
+            diff
+        );
     }
 
     #[test]

buildpacks/ruby/tests/integration_test.rs

@@ -81,7 +81,8 @@ fn test_default_app_ubuntu20() {
             assert_contains!(context.pack_stderr, "# Heroku Ruby Buildpack");
             assert_contains!(
                 context.pack_stderr,
-                r#"`BUNDLE_BIN="/layers/heroku_ruby/gems/bin" BUNDLE_CLEAN="1" BUNDLE_DEPLOYMENT="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_PATH="/layers/heroku_ruby/gems" BUNDLE_WITHOUT="development:test" bundle install`"#);
+                r#"`BUNDLE_FROZEN="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_WITHOUT="development:test" bundle install`"#
+            );
 
             assert_contains!(context.pack_stderr, "Installing puma");
 
@@ -90,7 +91,7 @@ fn test_default_app_ubuntu20() {
         let command_output = context.run_shell_command(
             indoc! {"
                 set -euo pipefail
-                printenv | sort | grep -vE '(_|HOME|HOSTNAME|OLDPWD|PWD|SHLVL|SECRET_KEY_BASE)='
+                printenv | sort | grep -vE '(_|^HOME|HOSTNAME|OLDPWD|PWD|SHLVL|SECRET_KEY_BASE)='
 
                 # Output command + output to stdout
                 export BASH_XTRACEFD=1; set -o xtrace
@@ -101,13 +102,11 @@ fn test_default_app_ubuntu20() {
         assert_empty!(command_output.stderr);
         assert_eq!(
             formatdoc! {"
-                BUNDLE_BIN=/layers/heroku_ruby/gems/bin
-                BUNDLE_CLEAN=1
-                BUNDLE_DEPLOYMENT=1
+                BUNDLE_FROZEN=1
                 BUNDLE_GEMFILE=/workspace/Gemfile
-                BUNDLE_PATH=/layers/heroku_ruby/gems
                 BUNDLE_WITHOUT=development:test
                 DISABLE_SPRING=1
+                GEM_HOME=/layers/heroku_ruby/gems
                 GEM_PATH=/layers/heroku_ruby/gems:/layers/heroku_ruby/bundler
                 JRUBY_OPTS=-Xcompile.invokedynamic=false
                 LD_LIBRARY_PATH=/layers/heroku_ruby/binruby/lib
@@ -190,11 +189,10 @@ fn test_default_app_ubuntu20() {
             assert_eq!(
                 r"
       $ echo $PATH
-      /layers/heroku_ruby/gems/ruby/<x.y.z>/bin:/workspace/bin:/layers/heroku_ruby/gems/bin:/layers/heroku_ruby/bundler/bin:/layers/heroku_ruby/binruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+      /layers/heroku_ruby/gems/bin:/workspace/bin:/layers/heroku_ruby/bundler/bin:/layers/heroku_ruby/binruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
       $ which -a rake
-      /layers/heroku_ruby/gems/ruby/<x.y.z>/bin/rake
-      /workspace/bin/rake
       /layers/heroku_ruby/gems/bin/rake
+      /workspace/bin/rake
       /layers/heroku_ruby/binruby/bin/rake
       /usr/bin/rake
       /bin/rake
@@ -203,7 +201,7 @@ fn test_default_app_ubuntu20() {
       /usr/bin/ruby
       /bin/ruby
             ".trim(),
-        Regex::new(r"/layers/heroku_ruby/gems/ruby/\d+\.\d+\.\d+/bin").unwrap().replace_all(&rake_output, "/layers/heroku_ruby/gems/ruby/<x.y.z>/bin").trim()
+        rake_output.trim()
 );
 
             let command_output = rebuild_context.run_shell_command(
@@ -240,7 +238,8 @@ fn test_default_app_ubuntu22() {
             assert_contains!(context.pack_stderr, "# Heroku Ruby Buildpack");
             assert_contains!(
                 context.pack_stderr,
-                r#"`BUNDLE_BIN="/layers/heroku_ruby/gems/bin" BUNDLE_CLEAN="1" BUNDLE_DEPLOYMENT="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_PATH="/layers/heroku_ruby/gems" BUNDLE_WITHOUT="development:test" bundle install`"#);
+                r#"`BUNDLE_FROZEN="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_WITHOUT="development:test" bundle install`"#
+            );
 
             assert_contains!(context.pack_stderr, "Installing puma");
         },
@@ -259,7 +258,8 @@ fn test_default_app_latest_distro() {
             assert_contains!(context.pack_stderr, "# Heroku Ruby Buildpack");
             assert_contains!(
                 context.pack_stderr,
-                r#"`BUNDLE_BIN="/layers/heroku_ruby/gems/bin" BUNDLE_CLEAN="1" BUNDLE_DEPLOYMENT="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_PATH="/layers/heroku_ruby/gems" BUNDLE_WITHOUT="development:test" bundle install`"#);
+                r#"`BUNDLE_FROZEN="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_WITHOUT="development:test" bundle install`"#
+            );
 
             assert_contains!(context.pack_stderr, "Installing puma");
 
@@ -340,7 +340,7 @@ DEPENDENCIES
             assert_contains!(context.pack_stderr, "# Heroku Ruby Buildpack");
             assert_contains!(
                 context.pack_stderr,
-                r#"`BUNDLE_BIN="/layers/heroku_ruby/gems/bin" BUNDLE_CLEAN="1" BUNDLE_DEPLOYMENT="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_PATH="/layers/heroku_ruby/gems" BUNDLE_WITHOUT="development:test" bundle install`"#
+                r#"`BUNDLE_FROZEN="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_WITHOUT="development:test" bundle install`"#
             );
             assert_contains!(context.pack_stderr, "Ruby version `3.1.4-jruby-9.4.8.0` from `Gemfile.lock`");
             });
@@ -361,7 +361,8 @@ fn test_ruby_app_with_yarn_app() {
             assert_contains!(context.pack_stderr, "# Heroku Ruby Buildpack");
             assert_contains!(
                 context.pack_stderr,
-                r#"`BUNDLE_BIN="/layers/heroku_ruby/gems/bin" BUNDLE_CLEAN="1" BUNDLE_DEPLOYMENT="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_PATH="/layers/heroku_ruby/gems" BUNDLE_WITHOUT="development:test" bundle install`"#);
+                r#"`BUNDLE_FROZEN="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_WITHOUT="development:test" bundle install`"#
+            );
             }
         );
 }

docs/application_contract.md

@@ -38,7 +38,7 @@ Once an application has passed the detect phase, the build phase will execute to
       - `Gemfile.lock`
       - User configurable environment variables.
     -To always run `bundle install` even if there are changes if the environment variable `HEROKU_SKIP_BUNDLE_DIGEST=1` is found.
-  - We will always run `bundle clean` after a successful `bundle install` via setting `BUNDLE_CLEAN=1` environment variable.
+  - We will always run `bundle clean` after a successful `bundle install`.
   - We will always cache the contents of your gem dependencies.
       - We will always invalidate the dependency cache if your distribution name or version (operating system) changes.
       - We will always invalidate the dependency cache if your CPU architecture (i.e. amd64) changes.
@@ -71,11 +71,8 @@ Once an application has passed the detect phase, the build phase will execute to
   - `SECRET_KEY_BASE=${SECRET_KEY_BASE:-<generate a secret key>}` - In Rails 4.1+ apps a value is needed to generate cryptographic tokens used for a variety of things. Notably this value is used in generating user sessions so modifying it between builds will have the effect of logging out all users. This buildpack provides a default generated value. You can override this value.
   - `BUNDLE_WITHOUT=development:test` - Tells bundler to not install `development` or `test` groups during `bundle install`. You can override this value.
 - Environment variables modified - In addition to the default list this is a list of environment variables that the buildpack modifies:
-  - `BUNDLE_BIN=<bundle-path-dir>/bin` - Install executables for all gems into specified path.
-  - `BUNDLE_CLEAN=1` - After successful `bundle install` bundler will automatically run `bundle clean` to remove all stale gems from previous builds that are no longer specified in the `Gemfile.lock`.
-  - `BUNDLE_DEPLOYMENT=1` - Requires `Gemfile.lock` to be in sync with the current `Gemfile`.
+  - `BUNDLE_FROZEN=1` - Requires `Gemfile.lock` to be in sync with the current `Gemfile`.
   - `BUNDLE_GEMFILE=<app-dir>/Gemfile` - Tells bundler where to find the `Gemfile`.
-  - `BUNDLE_PATH=<bundle-path-dir>` - Directs bundler to install gems to this path
   - `DISABLE_SPRING="1"` - Spring is a library that attempts to cache application state by forking and manipulating processes with the goal of decreasing development boot time. Disabling it in production removes significant problems [details](https://devcenter.heroku.com/changelog-items/1826).
   - `GEM_PATH=<bundle-path-dir>` - Tells Ruby where gems are located.
   - `MALLOC_ARENA_MAX=2` - Controls glibc memory allocation behavior with the goal of decreasing overall memory allocated by Ruby [details](https://devcenter.heroku.com/changelog-items/1683).