yarn2nix #20637
Comments
NeQuissimus
added
0.kind: enhancement
|
Nice :) |
|
What happens if |
|
Would you mind if yarn2nix was a go/rust binary instead of a ruby script? Or a nodejs script, since we are anyway working with the node ecosystem. Having to pull in ruby to work on node just doesn't feel good. I can volunteer for part of the effort. Nice excuse to try out something new too, IMO. |
|
I think it would be more maintainable to use yarn's parser directly. Here I made a small program that turns the yarn.lock into JSON: https://github.com/numtide/yarnlock2json . The URL still has to be split to extract the sha from the fragment. |
|
@moretea do you need help with this? This finally seems like a long-term viable approach… Use Yarn to get a fixed set of deps, create the deps, and let yarn do the node_modules build. I agree with @zimbatm that we should use yarn's own parser. I had a look at the yarn2nix and it seems very simple, so perhaps it can be built directly on top off zimbatm's script? Also, perhaps each yarn package should be a separate nix expression, so that they get reused between projects? I think it would mean walking the dep tree an building each dep in turn, creating packages as you go, and linking them into the |
|
Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore Regarding the node_modules folder, it's not practical to use symlinks in there unfortunately. If you search for it you'll see that others are trying to tackle the issue but in short it's due to two things: (1) some modules actually rely on the node_modules structure by doing things like |
|
Ideally we want the source to be fetched by nix but it's not clear to me
how to plug that into yarn yet. Maybe we can reproduce yarn's file layout
and ignore yarn install but I suspect there are some hidden complexities
there.
I was thinking that we can reproduce the cache dir layout. After all, we
have the full source available.
Another solution would be to work with yarn to see if we can pass them the
list of source derivations.
Actually that might work too, maybe with some sort of plugin system if we
can't come up with something that upstream would want to support. What
would that look like, ideally?
Regarding the node_modules folder, it's not practical to use symlinks in
there unfortunately.
I was not referring to that; AFAICT, Yarn uses its cache dir to store
tarballs and build products. These build products are then copied into
node_modules, so most likely the build product folders can be symlinks.
So the node_modules folder is a bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.
|
|
Alright that makes sense. I have a notion that this same problem is quite
general to all package managers. The difficulty for nix is that we don't
want to deal with dependency resolution, only work over a resolved set. We
want to fetch the dependencies but not deal with installing them. But
provide library dependencies for C extensions. And finally link the result
so the package can use them.
So that means nix and $lang package manager are quite intertwined. If we
can figure out the right interfaces for each of these steps, it will be
much easier to communicate with $lang and ask them to insert the proper
hooks where we need them.
For now my best idea is that $lang provides a lockfile that contains the
dependencies, where to fetch then including the SHA. $lang developer
shouldn't have to commit the generated nix code. Nix knows how to read that
lockfile and prefetch the dependencies from that. After that it's a bit
fuzzy and probably varies. Ideally we can build each dependency in
isolation but it's not always possible.
If there is a package repository like rubygems or npn we should push for
having the SHA included to make things faster. The SHA must be calculated
the same way as nix though.
Hmm that's it for now. Not really helping this issue in particular.
…On Fri, 16 Dec 2016, 20:36 Wout Mertens, ***@***.***> wrote:
>
> Ideally we want the source to be fetched by nix but it's not clear to me
> how to plug that into yarn yet. Maybe we can reproduce yarn's file layout
> and ignore yarn install but I suspect there are some hidden complexities
> there.
>
I was thinking that we can reproduce the cache dir layout. After all, we
have the full source available.
> Another solution would be to work with yarn to see if we can pass them
the
> list of source derivations.
>
Actually that might work too, maybe with some sort of plugin system if we
can't come up with something that upstream would want to support. What
would that look like, ideally?
> Regarding the node_modules folder, it's not practical to use symlinks in
> there unfortunately.
>
I was not referring to that; AFAICT, Yarn uses its cache dir to store
tarballs and build products. These build products are then copied into
node_modules, so most likely the build product folders can be symlinks.
So the node_modules folder is a bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#20637 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAMsNQzJh7dXxWHjTU2aEtlAIQiRCYkks5rIvZXgaJpZM4K6DHj>
.
|
Not at all. It's just a prototype at the moment and I just like to use Ruby to prototype these kinds of things.
Awesome. I actually know little about javascript; I just want to package some stuff.
I tried to do exactly that, but yarn still insists on writing random files in that cache directory. When I tried to fill it with the correct packages, it still failed because of missing packages.
It's a bit more subtile than that. The directory structure of
Fully agreed. Luckily a I would very much appreciate any help with this project ;) |
|
It's good to see that we are all pretty much on the same page even if the
details are still fuzzy. I might have some time allocated over the holiday
period, I will know next Tuesday.
…On Fri, 16 Dec 2016, 22:23 Maarten Hoogendoorn, ***@***.***> wrote:
@phanimahesh <https://github.com/phanimahesh> Would you mind if yarn2nix
was a go/rust binary instead of a ruby script?
Not at all. It's just a prototype at the moment and I just like to use
Ruby to prototype these kinds of things.
@zimbatm <https://github.com/zimbatm> I think it would be more
maintainable to use yarn's parser directly.
Awesome. I actually know little about javascript; I just want to package
some stuff.
@zimbatm <https://github.com/zimbatm> Ideally we want the source to be
fetched by nix but it's not clear to me how to plug that into yarn yet.
Maybe we can reproduce yarn's file layout and ignore yarn install but I
suspect there are some hidden complexities there.
@wmertens <https://github.com/wmertens> I was thinking that we can
reproduce the cache dir layout. After all, we
have the full source available.
I tried to do exactly that, but yarn still insists on writing random files
in that cache directory. When I tried to fill it with the correct packages,
it still failed because of missing packages.
@wmertens <https://github.com/wmertens> So the node_modules folder is a
bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.
It's a bit more subtile than that. The directory structure of node_modules
is a bit weird; npm/yarn support having multiple versions of different
packages in there. This in itself is not a weird thing to do; Cargo
supports the same features.
The problem is that the directory names do not have a version number.
Having multiple dependency versions is handled by having another
node_modules directory in a $ROOT_DIR/node_modules/some_package/
directory. Unfortunately, I could not find any documentation on the exact
logic of how these decisions are made.
@zimbatm <https://github.com/zimbatm> For now my best idea is that $lang
provides a lockfile that contains the
dependencies, where to fetch then including the SHA. $lang developer
shouldn't have to commit the generated nix code. Nix knows how to read that
lockfile and prefetch the dependencies from that. After that it's a bit
fuzzy and probably varies. Ideally we can build each dependency in
isolation but it's not always possible.
Fully agreed. Luckily a yarn.lock file contains all that!
------------------------------
I would very much appreciate any help with this project ;)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#20637 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAMsEHQWABoIju_cUb-C0BOMxQmidxUks5rIw9hgaJpZM4K6DHj>
.
|
|
I started to patch It was actually a lot less work than I anticipated. I'm closing this issue and proposing that we move the discussion to that PR. |
|
Hmm, the approach of node2nix is entirely different: basically it's yarn
for Nixpkgs, building node_modules itself.
Looking at the huge list of issues yarn has had, it seems like that is very
hard to do correctly.
i think the approach of running yarn in offline mode is much more
maintainable.
…On Sat, Dec 17, 2016, 4:30 PM Maarten Hoogendoorn ***@***.***> wrote:
I started to patch node2nix to support importing a yarn.lock file in
svanderburg/node2nix#28 <svanderburg/node2nix#28>.
I'm closing this issue and proposing that we move the discussion to that
PR.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#20637 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AADWlofvaFJfPOuPS91etznaQKDk2kWkks5rJAArgaJpZM4K6DHj>
.
|
|
Maybe so, but my painpoints were in getting yarn to use the files downloaded with fetchurl, which are all solved by node2nix already. It is something we can explore in parallel to only depending on yarn. |
|
Agreed. Yarn is a bunch of good ideas, implemented extremely poorly. We shouldn't trust yarn to do things right. If it can be convinced to use pre-downloaded files, that's what we should be doing. If it isn't supported, we should request/contribute that. |
|
I've rewritten I'm abusing the offline cache feature of yarn by generating a directory with @zimbatm @offlinehacker @wmertens @phanimahesh could you take a look? I gathered that you are packaging node apps. |
|
Nice! Looking good |
|
Nice, I really like the solution as it's simple, now all we need is |
|
Integrating yarn2nix into nixpkgs doesn't look all that difficult; I've already integrated it into our own package overrides. Does someone already have work in a branch to do this? If not, I'd love it if we could get a PR open to contribute yarn2nix and a separate [EDIT] I'm happy to help :) |
|
@ixmatus it's not that hard, the main reason not to merge was to be able to experiment without polluting the mainline. the project is in a pretty good state but still needs a bit of polish. |
|
@zimbatm Whats the current state of this? |
|
@flokli @Profpatsch's approach looks way better than the hacks that we do in this repository. If I would need to package a yarn thing again, I would probably take a look at his solution first. If you can use @Profpatsch's version, I'd go for that :) |
|
Please write issues if it takes you too long to get https://github.com/Profpatsch/yarn2nix running. I’m more than happy to help with packaging, to get feedback on whether the interface is any good and whether it actually works for all/most use cases. Sadly, the whole npm ecosystem is a bit of a mess and there’s lots and lots of edge cases everywhere you look. |
|
With #35340 merged, I think we can close this issue |
|
Unfortunately yarn2nix depends on Import From Derivation (IDF) to work, and it's not enabled on ofborg and hydra. yarn2nix can be used from nixpkgs but other packages in nixpkgs can't use it. See #35334 |
|
I wonder if we could add yarn2nix stuff to nixpkgs but disallow hydra builds? As a stepping stone. |
|
Can't yarn2nix be modified to generate its .nix expressions like node2nix, emacs2nix & cabal2nix? That is much preferred than disabling hydra. I think that is the right way to do things in Nixpkgs & it has worked very well for cabal2nix at least. |
|
agreed |
|
I realize that this has been removed from nixpkgs. However, would it make sense to use https://github.com/Profpatsch/yarn2nix by @Profpatsch instead? I'm getting started with nix and happen to be a bit confused by the two yarn2nix packages. I'm quite familiar with node/npm/yarn though. |
|
Hey @jbmusso - I think that project is less complete the last time I tested things. I'd recommend using yarn2nix by moretea. Maybe @Profpatsch is willing to rename his project to avoid confusion? |
|
Thanks for the recommendation, I'll experiment with moretea's package then. Since it's mostly JS, I may be able to patch things there if required - can't do that in Haskell yet. Unsure about what Profpatsch's package could be, maybe Also, thanks for building these tools. |
|
Still not documented. |
I guess I could, but no idea how much effort this would be. I’ll add a warning at the top of the readme for now (I don’t have much headspace for it right now sorry). |
|
Update: I’m trying to link to it but there is no documentation I can see anywhere #108138 (comment) |
|
So The Current State as far as I can see it:
|
|
I mean I could link to that code, but in that case I’d just as well have people try my repo first, which has an actual readme and an issue tracker … |
|
I updated the readme to link to this issue instead, so that people can see the developments over time (ideally we would update this issue when something new happens in this space). |
|
@Profpatsch's yarn2nix is compliant with my provisional NixOS/rfcs#109 I believe. I don't know whether the one in Nixpkgs is. |
Facebook recently published yarn, a package manager that behaves like Bundler and Cargo by having a lock file that describes the exact versions of all transitive dependencies.
I investigated the
yarn.lockformat, and was pleasantly surprised that it actually contains both direct download links to (and SHA1 hashes of) the.tgzfiles that make up packages.I'm very much open to merge this with @svanderburg's
node2nix, but I'm not sure that that's appropriate.Progress
The text was updated successfully, but these errors were encountered: