Broken capstone install on Apple Silicon Macs (arm64)

[atcuno]

Using pip to install Volatility with the provided requirements.txt file breaks on Apple Silicon Macs due to a long standing capstone bug where the amd64 binary of capstone is installed even on arm64 systems.

The current answer to work around it is here:

capstone-engine/capstone#2147 (comment)

which is running this command to install capstone:

pip install --pre --no-binary capstone capstone

And there is a long trail of tickets about the problem, but the latest from the developer team is that it will be properly fixed in 5.0.2, which was made in a comment from April:

capstone-engine/capstone#2221 (comment)

But 5.0.2 is still not yet released by capstone.

So currently, we will be guaranteed a broken Volatility install on Apple Silicon systems, plus presumably other ARM64 systems people might have, like Linux on ARM.

I am not well versed at all in writing complex requirements.txt, so I am not sure if we can replicate the "--pre --no-binary capstone capstone" logic inside of it for some time until capstone releases 5.0.2. If we don't have a workaround though, we will likely be receiving many bug reports soon as people starting using Volatility 3 more in the run up to the October event. In the workshop I ran this week, out of the ~40 students, around half of them had the Apple Silicon Macs and had to manually run the command with --pre after uninstalling the version our requirements.txt installs.

[atcuno]

@digitalisx @branchvincent @gcmoreira @eve-mem @npetroni Tagging people here who have more pip experience than I do

[gcmoreira]

@atcuno It seems you can add those options to requirements.txt, i.e.:

$ cat requirements.txt 
--pre
--no-binary capstone
capstone>=3.0.5

That will include those options in the __pip-runner__.py line:

(venv) $ strace -f -s99999 pip install --dry-run -r requirements.txt 2>&1 | grep pip
...
[pid 211809] execve("/home/user/venv/bin/python", ["/home/user/venv/bin/python", "/home/user/venv/lib/python3.12/site-packages/pip/__pip-runner__.py", "install", "--ignore-installed", "--no-user", "--prefix", "/tmp/pip-build-env-bg38cq3q/overlay", "--no-warn-script-location", "--no-binary", "capstone", "--only-binary", ":none:", "-i", "https://pypi.org/simple", "--pre", "--", "setuptools>=40.8.0"], 0x3253850 /* 80 vars */ <unfinished ...>

I haven't tested that, could you check if that works on Mac?

On the other hand, I think we should limit the pre-releases to capstone only. The pre-release-version docs says:

If a Requirement specifier includes a pre-release or development version (e.g. >=0.0.dev0) then pip will allow pre-release and development versions for that requirement.

So, I think we could avoid the --pre if the requirements.txt capstone line is i.e.:

capstone>=3.0.5.dev0

EDIT: The above will also include the 3.0.5 pre-releases, such as 3.0.5rc2. See more in the next post.

hope that helps

[gcmoreira]

To test the pre-release setting, I did the following:
First, I checked the capstone pip package history. The available versions >= 3.0.5 are:

• 3.0.5
• 3.0.5.post1
• 4.0.0
• 4.0.1
• 4.0.2
• 5.0.0rc2
• 5.0.0rc4
• 5.0.0.post1
• 5.0.1

To simulate that the lastest available version (including the pre-releases) is i.e. 5.0.0rc4, I filter out the last two versions (5.0.0.post1 and 5.0.1).

$ pip install --dry-run "capstone>=3.0.5,<5.0.0.post1" | grep "Would install"
Would install capstone-4.0.2

$ pip install --dry-run --pre "capstone>=3.0.5,<5.0.0.post1" | grep "Would install"
Would install capstone-4.0.2

Hm that's weird. In this case --pre doesn't seem to work, or for some reason, filtering out the post1 also removes all the 5.0.0 versions .. that doesn't look right to me, maybe a pip bug? anyway, let's remove also the post1 from the list:

$ pip install --dry-run "capstone>=3.0.5,<5.0.1,!=5.0.0.post1" | grep "Would install"
Would install capstone-4.0.2

$ pip install --dry-run --pre "capstone>=3.0.5,<5.0.1,!=5.0.0.post1" | grep "Would install"
Would install capstone-5.0.0rc4

That worked. As it's expected, the --pre includes the pre-releases.

Let's try if removing the --pre and using the workaround in the previous post has the same behavior:

$ pip install --dry-run "capstone>=3.0.5.dev0,<5.0.1,!=5.0.0.post1" | grep "Would install"
Would install capstone-5.0.0rc4

Even though the 3.0.5.dev0 version doesn't exist as such, it worked as expected.

Using just .dev or .pre also does the trick.

$ pip install --dry-run "capstone>=3.0.5.dev,<5.0.1,!=5.0.0.post1" | grep "Would install"
Would install capstone-5.0.0rc4

$ pip install --dry-run "capstone>=3.0.5.pre,<5.0.1,!=5.0.0.post1" | grep "Would install"
Would install capstone-5.0.0rc4

To sum up, as the >=3.0.5.pre will also include the 3.0.5 pre-releases such as3.0.5rc2 which wasn't included in the current >=3.0.5, IMO we should do the following in requirements.txt:

...
--no-binary capstone
capstone>3.0.5.pre
...

That way, we can avoid to activate pre-releases for all modules, and the behaviour will be the same as --pre + capstore>=3.0.5 + --no-binary capstone.

[gcmoreira]

The following also proves that the --no-binary capstone in requirements.txt works:

$ pip install --dry-run --debug -vvv -r requirements.txt 2>&1 | grep "Skipping.*binaries"
  Skipping link: No binaries permitted for capstone: https://files.pythonhosted.org/packages/85/10/acee17a5e42f370306aafa732b5d370e736d5a9f258820a6d680d2f68803/capstone-3.0.5rc2-py2-none-manylinux1_i686.whl (from https://pypi.org/simple/capstone/)
  Skipping link: No binaries permitted for capstone: https://files.pythonhosted.org/packages/fd/33/d1fc2d01b85572b88c9b4c359f36f88f8c32f2f0b9ffb2d21cd41bad2257/capstone-3.0.5rc2-py2-none-manylinux1_x86_64.whl (from https://pypi.org/simple/capstone/)
  Skipping link: No binaries permitted for capstone: https://files.pythonhosted.org/packages/92/78/0bb5949fdd0cc45302fdb7d2e390ed7ba165a0a3df292541cb4ba1a23ea6/capstone-3.0.5rc2-py2-none-win32.whl (from https://pypi.org/simple/capstone/)
  Skipping link: No binaries permitted for capstone: https://files.pythonhosted.org/packages/07/6d/5a05b17e65ada4e6940df19f82d248820114fd7a2590
...

[ikelos]

Has this been resolved by merging #1157?

[atcuno]

This is fixed. Thanks @ikelos and @gcmoreira