Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Followup from #10040 - Support uv compiled requirements files #10688

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
bc29be7
Support python uv as pip-compile compatible replacement
avilaton Jun 19, 2024
823600b
Update requirements.txt
avilaton Jul 16, 2024
6fbb577
Update python/helpers/requirements.txt
avilaton Jul 17, 2024
94bb96b
Update python/spec/fixtures/requirements/pip_compile_uv_header.txt
avilaton Jul 18, 2024
2a63fb6
Bump uv. New uv_pip_compile_options_from_compiled_file method and big…
albertferras-vrf Aug 18, 2024
1fef8ee
Update python/helpers/requirements.txt
avilaton Aug 22, 2024
703cbe4
Update python/lib/dependabot/python/file_updater/pip_compile_file_upd…
avilaton Aug 22, 2024
408fcef
Update python/lib/dependabot/python/file_updater/pip_compile_file_upd…
avilaton Aug 22, 2024
a5ec78a
Rename uv_pip_compile_file_updater_spec.rb to pip_compile_file_update…
avilaton Aug 22, 2024
26caba4
Update python/lib/dependabot/python/file_updater/pip_compile_file_upd…
avilaton Sep 9, 2024
e1913f0
Update python/lib/dependabot/python/file_updater/pip_compile_file_upd…
avilaton Sep 9, 2024
a659f80
Update python/lib/dependabot/python/file_updater/pip_compile_file_upd…
avilaton Sep 9, 2024
5f26e1e
Update python/helpers/requirements.txt
avilaton Sep 12, 2024
f31dc26
Update python/lib/dependabot/python/update_checker/pip_compile_versio…
avilaton Sep 12, 2024
0d7312e
Update pip_compile_version_resolver.rb
avilaton Sep 12, 2024
03d0ef9
Add missing functions
avilaton Sep 23, 2024
ee616b2
Remove legacy pip stuff I don't mean to mess with
avilaton Sep 23, 2024
8101dde
[BUGFIX] Resolve missing regex const definition
imajes Sep 27, 2024
fa5519d
[BUGFIX] Update as per comments from RazerM
imajes Sep 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions python/helpers/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ plette==2.1.0
poetry==1.8.3
# TODO: Replace 3p package `toml` with 3.11's new stdlib `tomllib` once we drop support for Python 3.10.
toml==0.10.2
uv==0.4.9

# Some dependencies will only install if Cython is present
Cython==3.0.10
Original file line number Diff line number Diff line change
Expand Up @@ -96,13 +96,14 @@ def compile_new_requirement_files
def compile_file(filename)
# Shell out to pip-compile, generate a new set of requirements.
# This is slow, as pip-compile needs to do installs.
options = pip_compile_options(filename)

options, command = pip_compile_options(filename)
options_fingerprint = pip_compile_options_fingerprint(options)

name_part = "pyenv exec pip-compile " \
name_part = "#{command} " \
"#{options} -P " \
"#{dependency.name}"
fingerprint_name_part = "pyenv exec pip-compile " \
fingerprint_name_part = "#{command} " \
"#{options_fingerprint} -P " \
"<dependency_name>"

Expand Down Expand Up @@ -453,10 +454,16 @@ def pip_compile_options(filename)
options += pip_compile_index_options

if (requirements_file = compiled_file_for_filename(filename))
options += pip_compile_options_from_compiled_file(requirements_file)
if requirements_file.content.include?("uv pip compile")
options += uv_pip_compile_options_from_compiled_file(requirements_file)
command = "pyenv exec uv pip compile"
else
options += pip_compile_options_from_compiled_file(requirements_file)
command = "pyenv exec pip-compile"
end
end

options.join(" ")
[options.join(" "), command]
end

def pip_compile_options_from_compiled_file(requirements_file)
Expand All @@ -483,6 +490,30 @@ def pip_compile_options_from_compiled_file(requirements_file)
options
end

def uv_pip_compile_options_from_compiled_file(requirements_file)
options = ["--output-file=#{requirements_file.name}"]

options << "--no-emit-index-url" if requirements_file.content.include?("index-url http")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would cause dependabot to strip an existing index-url.

see #10040 (comment)

Suggested change
options << "--no-emit-index-url" if requirements_file.content.include?("index-url http")
options << "--emit-index-url" if requirements_file.content.include?("index-url http")

probably needs a test generated with --emit-index-url to make sure dependabot doesn't remove it

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah good catch!


options << "--generate-hashes" if requirements_file.content.include?("--hash=sha")

options << "--no-annotate" unless requirements_file.content.include?("# via ")

options << "--pre" if requirements_file.content.include?("--pre")

options << "--no-strip-extras" if requirements_file.content.include?("--no-strip-extras")

if requirements_file.content.include?("--no-binary") || requirements_file.content.include?("--only-binary")
options << "--emit-build-options"
end

# options skips the resolver option, as it has no effect for uv.

options << "--universal" if requirements_file.content.include?("--universal")

options
end

def pip_compile_index_options
credentials
.select { |cred| cred["type"] == "python_index" }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ class PipCompileVersionResolver
PYTHON_PACKAGE_NAME_REGEX = /[A-Za-z0-9_\-]+/
RESOLUTION_IMPOSSIBLE_ERROR = "ResolutionImpossible"
ERROR_REGEX = /(?<=ERROR\:\W).*$/
RESOLVER_REGEX = /(?<=--resolver=)(\w+)/

attr_reader :dependency
attr_reader :dependency_files
Expand Down Expand Up @@ -91,12 +92,12 @@ def fetch_latest_resolvable_version_string(requirement:)
def compile_file(filename)
# Shell out to pip-compile.
# This is slow, as pip-compile needs to do installs.
options = pip_compile_options(filename)
options, command = pip_compile_options(filename)
options_fingerprint = pip_compile_options_fingerprint(options)

run_pip_compile_command(
"pyenv exec pip-compile -v #{options} -P #{dependency.name} #{filename}",
fingerprint: "pyenv exec pip-compile -v #{options_fingerprint} -P <dependency_name> <filename>"
"#{command} -v #{options} -P #{dependency.name} #{filename}",
fingerprint: "#{command} -v #{options_fingerprint} -P <dependency_name> <filename>"
)

return true if dependency.top_level?
Expand All @@ -110,8 +111,8 @@ def compile_file(filename)
# update_not_possible.
write_original_manifest_files
run_pip_compile_command(
"pyenv exec pip-compile #{options} #{filename}",
fingerprint: "pyenv exec pip-compile #{options_fingerprint} <filename>"
"#{command} #{options} #{filename}",
fingerprint: "#{command} #{options_fingerprint} <filename>"
)

true
Expand Down Expand Up @@ -201,12 +202,12 @@ def check_original_requirements_resolvable
write_temporary_dependency_files(update_requirement: false)

filenames_to_compile.each do |filename|
options = pip_compile_options(filename)
options, command = pip_compile_options(filename)
options_fingerprint = pip_compile_options_fingerprint(options)

run_pip_compile_command(
"pyenv exec pip-compile #{options} #{filename}",
fingerprint: "pyenv exec pip-compile #{options_fingerprint} <filename>"
"#{command} #{options} #{filename}",
fingerprint: "#{command} #{options_fingerprint} <filename>"
)
end

Expand Down Expand Up @@ -251,7 +252,16 @@ def pip_compile_options(filename)
options << "--output-file=#{requirements_file.name}"
end

options.join(" ")
if (requirements_file = compiled_file_for_filename(filename))
if requirements_file.content.include?("uv pip compile")
options += uv_pip_compile_options_from_compiled_file(requirements_file)
command = "pyenv exec uv pip compile"
else
command = "pyenv exec pip-compile"
end
end

[options.join(" "), command]
end

def pip_compile_index_options
Expand All @@ -277,6 +287,32 @@ def run_pip_compile_command(command, fingerprint:)
run_command(command, fingerprint: fingerprint)
end

def uv_pip_compile_options_from_compiled_file(requirements_file)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this implementation needs to be updated to match the one in pip_compile_file_updater.rb

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha

options = ["--output-file=#{requirements_file.name}"]

options << "--no-emit-index-url" unless requirements_file.content.include?("index-url http")

options << "--generate-hashes" if requirements_file.content.include?("--hash=sha")

options << "--no-annotate" unless requirements_file.content.include?("# via ")

options << "--pre" if requirements_file.content.include?("--pre")

options << "--no-strip-extras" if requirements_file.content.include?("--no-strip-extras")

if requirements_file.content.include?("--no-binary") || requirements_file.content.include?("--only-binary")
options << "--emit-build-options"
end

if (resolver = RESOLVER_REGEX.match(requirements_file.content))
options << "--resolver=#{resolver}"
end

options << "--universal" if requirements_file.content.include?("--universal")

options
end

def python_env
env = {}

Expand Down
Loading
Loading