Latest / Full Time Nix / Nix 2.23.0 release with Jacek Galowicz

Transcript

Nix 2.23.0 release with Jacek Galowicz · Full Time Nix

0:03
Hello and welcome to Full Time Nix.
0:06
This is an episode about release notes of Nix 2.23.0 with Jacek Galowicz.
0:18
Hi, Jacek.
0:20
Hey, Shahar how are you?
0:23
Pretty good, pretty good.
0:25
Well, let's talk about it.
0:29
Yes, let's go.
0:35
There are two new features.
0:40
One is a new built-in.
0:43
and as all built-ins it can be found under the global scope `builtins` attrset and
0:53
it's called warn.
0:54
So it'll be `builtins.warn`.
0:57
What does it do?
1:00
Yeah, so when you use `builtins.warn`, it has a very similar semantics to
1:06
`builtins.trace`.
1:07
Basically, `builtins.trace` is a function that takes two parameters.
1:11
The first one is a string that will be printed on the user's shell.
1:16
It will not go into the program itself.
1:19
It will not change how your program behaves.
1:23
and the second parameter to it will be some kind of value that goes from
1:27
somewhere somewhere else and will just be forwarded while your message is printed on
1:32
the terminal only visible to the user.
1:37
Developers use this for deprecation warnings or just printf style debugging or
1:43
whatever else.
1:46
Generally, traces don't really do anything and it's
1:50
relatively difficult for the end user if you are triggering a warning somewhere
1:55
that is disguised as a trace kind of to find out where it comes from and we kind
2:05
of lately got this new debugging features in Nix and you can start a debugger on
2:12
traces but then the problem is if you're
2:15
.
2:17
you just want to catch the one specific trace that is a warning about something
2:22
that you're doing wrong because of deprecation or whatever.
2:27
Yeah, you get in trouble distinguishing that.
2:30
So they introduced `builtins.warn`, which is kind of a special trace, which is meant
2:36
to be only used for actual warnings, warnings that users ought to fix.
2:44
it is now possible to tell the debugger that it should only trigger not on traces
2:49
but only on warnings or even abort on warnings.
2:53
So you could either as a developer say, OK, you activate the option debugger on
3:00
warn and then your debugger will jump into the code and give you the context as soon
3:06
as it sees a warning being triggered.
3:09
You could also in your CI,
3:11
or whatever else, set your Nix builds with the option `--abort-on-warn` and then it
3:19
will go red in front of your face when your code triggers a warning.
3:23
So this is very nice because all the traces could still be in there.
3:27
They wouldn't produce any noise because you can already distinguish between
3:32
tracing and warning, right?
3:35
Also, the warning messages have a little bit of a different
3:41
route of character.
3:44
it's easier to grab for warnings or whatever because of their fixed format.
3:50
So this is in general useful for the debugging mix developer and also for
3:55
people who simply want to get rid of all warnings for example after an update or
4:02
similar.
4:04
That is great.
4:04
Sorry, what is output character?
4:09
Mean.
4:09
I mean, not output character as a character, but the general character of
4:14
the outputs being printed.
4:18
Before that, you had `builtins.trace` with maybe a warning prefix, maybe not.
4:24
So there was no real guarantee that all the warning output that comes from faces
4:32
being used would look similar.
4:35
There was no guarantee, right?
4:36
Now...
4:37
with `builtins.warn` being a built-in primitive, they will have exactly the same
4:45
output formatting.
4:47
So you will have a nicer time simply distinguishing that.
5:26
from.
5:28
This feature certainly makes it even easier because you don't get false
5:33
positives.
5:34
Everything that is not a warning is not being triggered any longer as long as
5:38
developers use this warn feature.
5:53
I asked for this on this course.
5:57
nice.
6:06
Anyway, thank you, who did this?
6:17
Robert H.
6:21
Hensing.
6:23
Thank you Robert.
6:28
new command nix env shell as the new nix shell so what is a nix space shell what is
6:41
that the existing command
6:45
generally want to start a new shell with specific packages installed, you could
6:52
write like, Nix space, shell space, and then nixpkgs, hashtag, some package name.
7:00
Then you would be launched into a new shell that seems like this package is
7:05
installed there.
7:06
And when you exit that shell, it's gone again.
7:08
It's one of the very nice killer features of Nix, right?
7:12
I mean, people who don't use Nix, they do Docker run something, something, and then
7:19
they simply start Docker images where one or multiple tools are installed in.
7:24
If you want to customize that, you have to write your own Docker image.
7:30
And that is not really composable.
7:32
You cannot combine two Docker images easily that have all the tools that you
7:36
like.
7:37
With the Nix shell, this is different.
7:39
You simply line up all the...
7:41
packages that you would like to have in your shell.
7:44
Launch the command.
7:45
Boom, you're in a shell now that has all these packages installed, right?
7:50
Also, not to underestimate that or not to forget, Nix.
7:56
I mean, you could do this from any Linux distro, right?
7:59
Many people are on Fedora or Ubuntu or whatever.
8:01
Maybe they don't have this package, but Nix gives you access to the biggest and
8:06
freshest open source package repository in the world, right?
8:09
This is two features at the same time.
8:11
One is these ephemeral shells.
8:15
You go into this shell, have this package, you exit the shell and it's gone again,
8:19
right?
8:19
At least it looks like this.
8:21
And access to this huge package collection, right?
8:25
So that is what nix shell generally does.
8:29
And it's not only for ad hoc invocations,
8:43
Right?
8:45
Generally, when we look at the old commands, nix-shell, not nix -space
8:52
-shell, but nix-shell, it was kind of meant to be used in two different
8:56
scenarios.
8:57
One is with the list of packages on the command line, so you get an ad hoc shell
9:02
that you just define on the command line.
9:05
The other was to source a shell.nix file or similar.
9:11
where someone already prepared all the projects packages that you want to work
9:15
on, then you wouldn't have any other command -line parameters.
9:20
With Flakes, this developed a little bit in a different direction.
9:24
So we have two different commands for doing these two things.
9:28
One is for an ad hoc shell, where you simply line up all the package names on
9:33
the command line, you use `nix shell` for that.
9:37
If you want to start a development environment from a Flake project, you
9:41
would go with the next develop command.
9:46
So, Nix Develop will look into the Dev Shells section of your Flake and without
9:53
any parameters, select the default one.
9:56
Otherwise, you could also select a specific one on the command line so that
10:00
you can have different developer environments per project if that is
10:04
useful.
10:05
And then it would simply launch that.
10:07
And that's two separate commands now, right?
10:12
Correct, I wonder why it was renamed, but I also, right, but there is a Nix, so
10:20
there's the old Nix hyphen shell command, which does not work with Flakes, and it
10:28
works with ad hoc invocations, and otherwise it looks for, by default, a
10:33
shell.nix, but it can be arbitrary file.
10:38
And with Flakes or,
10:42
Flakes and nix-command experimental features.
10:45
There is Nix space shell, which is only equivalent to the Nix hyphen shell ad hoc
10:59
invocation.
11:01
Accepted.
11:03
The equivalent to the shell.nix usage of the classic nix-shell is nix develop.
11:16
Yeah, so when you look at how `nix-shell`'s feature with ad -hoc shells
11:24
was implemented, I mean, you use the nix-shell command and then dash p and then
11:30
list of packages.
11:31
It would kind of take all your package names, put them into a bigger Nix
11:35
expression and launch that as it's kind of ad -hoc Nix shell or shell.nix.
11:40
So it would kind of create one on the fly and then simulate this behavior.
11:46
The problem was there that it selects the nixpkgs for you where the packages come
11:56
from.
11:57
You could not really easily change that without tampering with environment
12:02
variables.
12:04
And the new Nix shell, Nix space shell, it waits for Flake input and then the package
12:13
input from the Flake.
12:14
You write Nix shell.
12:17
nixpkgs, hashtag, some package name.
12:20
And then this command line defines not only the package that you want, but also
12:25
the policy decision, where should this package come from?
12:28
So you could mix up different nixpkgs commits on the command line.
12:33
You want package A in very new and package B in very old.
12:37
Or you could simply access completely
12:41
different third -party flakes.
12:43
So you have some hobby repository somewhere and tell someone, hey, you can
12:46
simply run my package from the command line without cloning my flake.
12:51
So you would write nix shell github colon shahar slash saharsproject hashtag
13:00
shahar's package name.
13:02
And boom, that's it, right?
13:03
So the old nix shell command couldn't do this.
13:08
Or not without a lot of hassle.
13:11
It was trying to do this wouldn't be a nice user experience with the old command.
13:16
And with the new command, it's really easy, right?
13:19
So it's more powerful, actually, much more powerful.
13:26
Yes.
13:27
And what is this `nix env shell`?
13:32
Is it supposed to replace nix space shell?
13:36
So it's nix space env space shell.
13:41
Exactly.
13:41
So if you look at this alone in isolation, it looks like a completely random rename,
13:48
but that's not what it is.
13:50
There's a concept behind that.
13:52
And the whole idea is that we have so many different Nix commands, right?
13:57
Nix shell this, sorry, Nix space this, Nix space that.
14:02
The developers tried to make it more intuitive by...
14:08
giving these subcommands a common scheme where you know, okay, I want to do
14:13
something with development environments.
14:16
So there are multiple Nix, space, dev, space, something commands, or I want to do
14:21
something with package installations or building or running.
14:25
Then there's a Nix space, package space, something, something subcategory.
14:31
So they are kind of subdividing the commands which kind of randomly piled up
14:37
over the years.
14:38
into new categories that make more sense because they're more coherent.
14:42
So for example, there will be `nix dev shell` and `nix dev run` and `nix dev
14:52
print-env`.
14:53
So these things all have to do something with development shells that are from a
14:59
flake.
15:02
But you can also run `nix env shell` in contrast to `nix dev shell`.
15:10
That's better
15:12
shell` will be kind of ad hoc shells.
15:15
Everything with `nix env` will be something like an ad hoc thingy.
15:19
Everything with `nix dev` will be something with prepared development
15:22
environments from Flakes, right?
15:24
So this totally makes sense.
15:27
And the other commands will be like NixPackageBuild, NixPackageRun,
15:31
NixPackageInfo, NixPackageEdit.
15:35
And that will be something that will be like per package do something with one
15:39
specific package.
15:41
And yeah, so they're kind of there's a lot of discussion going on.
15:45
So I went through all the GitHub issues where they were discussing how stuff
15:50
should be named.
15:51
Of course, naming is one of the hard problems in IT.
15:56
People put like five people in a room to discuss how something should be named and
16:02
you get 10 different opinions out of it.
16:05
So I think we will.
16:09
have more discussion about that in the GitHub issues and forums, how this should
16:13
be.
16:13
Some people will like it, some people won't, but in general, I'm looking forward
16:17
to this improved structure, right?
16:27
Yes, it sounds right.
16:30
It sounds right to me and I also trust that the intelligent beings who tend to
16:39
this project are making reasonable decisions.
16:47
Alright, so that's a small and yet part of a greater improvement change.
16:55
absolutely.
16:56
So one other note.
16:58
They will not remove the `nix shell` command because of `nix env shell`.
17:03
So both will coexist for a while, I guess.
17:07
So this is not a breaking change where everyone who updates to Nix 23.0 will have
17:15
to rewrite all their commands.
17:17
That is not the case.
17:20
They are simply added at this point.
17:26
and I hope that it will eventually be removed because this is the time to do
17:32
this all of these are supposed to be experimental it's just that we're we took
17:41
our time
17:42
Yeah, of course.
17:43
But I mean, commands like nix develop, nix shell, nix build with the spaces in
17:48
between and not the dashes.
17:51
Yeah, so many people use it already.
17:53
And of course, this is officially experimental, but it's experimental for so
17:59
long already that people ignore that.
18:02
So our experimental semantics are not really not equally useful everywhere right
18:09
now.
18:11
Mm -hmm.
18:12
It totally makes sense that they are not deprecating the nix shell, nix develop,
18:17
nix build and nix run commands too quickly.
18:21
But eventually you are right, eventually the whole interface will look chaotic and
18:27
noisy if you keep all of this, right?
18:32
Yeah, I'll leave it up to the Nix team.
18:38
Absolutely.
18:39
Yeah, they will figure it out and do it well.
18:46
So there are some more improvements.
18:48
One is to the consistency.
18:55
It's about making a nix build with the keep going flag consistent with the
19:03
nix-build with the keep going flag.
19:08
How is it not consistent?
19:11
So first, the use case for the keep going flag is usually when you're building
19:19
something on your laptop and that rebuilds many, many different derivations at the
19:25
same time.
19:26
For example, if you updated your inputs and then rebuilt, of course, everything is
19:31
going to be rebuilt usually.
19:33
And then without the keep going flag, the whole build will simply abort on the first
19:39
error.
19:41
Mm -hmm.
19:41
And maybe there are 10 errors and you want different packages and you want to see
19:46
them all.
19:48
Typically what's also the case when you are, let's say you have like two packages
19:52
building at the same time and one builds for an hour and will be completely fine.
20:00
And the other one builds for like 10 minutes until it breaks.
20:06
And without keep going the
20:09
whole build will break after 10 minutes after the first error, while the other
20:13
package, which you could simply let continue, is aborted too.
20:18
So on the next nix build, the longer package build will be completely
20:23
restarted, and that might not be necessary.
20:27
So if you run Nix build with the keep going parameter, it will simply print all
20:32
the errors that it finds on the way, so you can on your other screen start to
20:37
fix problems and bugs and at the same time all the builds that are completely fine
20:43
will continue to build, right?
20:44
So you're kind of not wasting time.
20:48
While you're waiting anyway, you can fix the bugs of the builds that don't work,
20:52
right?
20:52
So that's very useful of a feature.
20:55
Typically, I don't enable it in CIs because I want CIs to get red pretty
21:00
quickly and then abort so they have the resources for other builds that are not
21:04
going red.
21:05
while at home while debugging stuff, I'm using the keep going flag to just keep the
21:11
builds that work building while I fix the problems, right?
21:15
But now it was like this feature existed in old times already with the old
21:19
`nix-build` command and with the new `nix build` command, they still have it, but
21:26
they are slightly, slightly different.
21:28
So for example, if you have like a...
21:34
output derivations that fail to build.
21:38
Nix is basically comparing hashes and hash mismatches are displayed in terms of the
21:45
error message.
21:47
And maybe you have like three, four, five different hash mismatches that you want to
21:52
fix quickly.
21:55
But then you can't because it's already aborting after the first one.
21:59
So the...
22:02
old `nix-build` command showed you all of them, the new one didn't and now this is
22:08
fixed so this will be as useful as the old one.
22:16
Sorry, so what was the difference?
22:19
There was a difference in what it keeps going and what it fails on with regards to
22:26
this particular hash mismatch?
22:30
In case of hash mismatches, there were cases where not all of them are displayed
22:34
that were found.
22:36
It's only with regards to what is displayed, you say.
22:43
Yeah, kind of.
22:46
In general, the whole idea of keep going is simply print all the errors that you
22:54
see, but also keep going with everything else.
22:57
And this promise of really keep going during a build was, so to say, not
23:04
completely followed by the new nix build command.
23:08
So sometimes it was the case that you run nix space build keep going, then it
23:12
displays you some kind of hash mismatch.
23:15
And then, okay, then you fix it.
23:17
Then you run it again, and then it will come up with a new hash mismatch that
23:21
could have been shown on the first run already, but didn't.
23:24
Of course, that's irritating, that's stupid, but now they fixed that.
23:30
so some things were...
23:37
triggering a complete failure short -circuiting everything in spite of the
23:46
flag keep going and now it's fixed now they will not cause the whole build to
23:53
fail it will keep going
23:55
say keep going now, like really keep going.
23:59
The promise wasn't met completely right now it is.
24:03
And it's kind of, it's not forgetting to print all of the errors that I've seen.
24:11
There were simply corner cases in which this was the case and some cases in which
24:15
it wasn't.
24:17
I didn't really experience this myself, so I'm just reading from the issue
24:25
description on GitHub and the pull requests what was the case.
24:28
And it really seems like there were simply a few special cases, like with fixed
24:34
output derivations, where it didn't keep up the promise.
24:41
It's good to know.
24:47
I mean, if you don't know, then you can already forget.
24:52
Because it's now in parity now, right?
24:57
I just would like to thank the author of this fix.
25:04
And so whoever you are, thank you.
26:00
Okay, by the way with this feature flag keep going, this keep going flag, errors
26:11
are being printed, there is no summary, right?
26:15
It's not like you get a summary of errors at the end, right?
26:20
Yeah, that's true.
26:21
I mean you get a summary of all the, I mean a list of packages that failed
26:26
obviously.
26:29
And for each of them you can run the show log command and then it will give you the
26:36
whole log of this specific package built in isolation, which is very useful because
26:41
typically they are just mixed on the terminal, right?
26:45
But if you really want to understand a more complex error
26:49
then you want to see the package build log in isolation which you always could.
26:55
It's just another command.
26:57
okay great JSON format update modify `nix derivation add` and `show` `nix derivation
27:09
add`, `nix derivation show` JSON format.
27:13
What is `nix derivation add` and `nix derivation show`?
27:19
So, `nix derivation show` is a command which allows you to get a more human
27:28
readable form of a Nix Derivation, right?
27:31
When you evaluate a Nix expression, it will kind of, yeah, interpret your Nix
27:37
script and create a data structure.
27:41
This data structure is called a Derivation.
27:44
And when you build a package, this Derivation is actually
27:49
the thing that is being built, right?
27:51
Your Nix expression is full of variables and whatever, you can update the nixpkgs
27:55
inputs and whatnot, but the derivation is completely fixed, completely static,
28:00
contains no variables any longer whatsoever.
28:05
All the Nix paths are set there.
28:07
And this is the format that kind of doesn't know about the Nix language any
28:13
longer.
28:14
It's just the pure static data structure.
28:16
And this data structure can be sent to other remote builders, for example, and
28:21
they will be able to build stuff.
28:24
So you could, in theory, even create different programming, use different
28:32
programming languages to create a derivation structure, right?
28:35
You wouldn't completely have to use Nix.
28:37
So this is a useful abstraction layer.
28:43
Now the question is in what format are these derivations defined?
28:48
How to read them, how to create them and so on, right?
28:52
And below the code, when you look at the derivation file, it is implemented in so
29:00
-called ATerm format.
29:01
ATerm is a very old serialization format.
29:06
It is halfway human readable.
29:09
So it's not really binary, it's ASCII.
29:13
But nowadays people use JSON to describe all kinds of data structures because
29:19
that's a very nice language agnostic thing.
29:22
All the programming languages have JSON serialization and deserialization
29:25
libraries and so on.
29:26
So the `nix derivation show` command gives you a very nice JSON object from a
29:33
derivation.
29:35
The thing is that this
29:37
feature has been used in the past by humans to simply read format and search
29:43
derivation structures.
29:45
So it was fine if this was not completely valid JSON.
29:51
But they are going to change this.
29:53
So they want to have it like real JSON by following the spec.
29:59
And in this case, what they did is they revised the output
30:07
to better conform their own JSON guidelines.
30:11
For example, when you have content address derivations, the content address
30:19
derivations specify the output hash that comes there and the used hash algorithm
30:27
and also the kind of serialization that they did.
30:32
So that's very much content addressable.
30:36
storage details here.
30:39
The thing is that they had like their own format of describing a content addressable
30:44
derivation and they're kind of pulling this apart to be more composed.
30:51
In the past they put multiple different kinds of information into the same string
30:56
and separated them by columns and now they are trying to make this individual JSON
31:01
fields.
31:02
So it becomes easier to programmatically
31:05
do something with the derivation files.
31:08
This is a pretty much deep below the hood change.
31:14
So normal users who don't really play around with derivation files will not
31:19
notice this.
31:21
But it will enable tooling that reads derivation files to do something else with
31:28
that to have easier implementations because of this improved format, right?
31:37
Sounds like it.
31:40
So thank you.
31:47
John Ericson.
31:52
for this change.
31:55
Warnings, warn on unknown settings anywhere in the command line.
32:06
Right, so this is extremely useful.
32:12
With Flakes this is not really relevant because Flakes don't really take command
32:18
line like options on the command line.
32:21
Flakes are so hermetic that you cannot really change the output by something on
32:27
the command line.
32:28
But this is relevant for the old non -flakes commands.
32:32
So for example when you run nix-build or nix-instantiate on something
32:41
on a Nix expression that takes a parameter on the top level, you could provide this
32:48
parameter on the command line, like a flag, a string, or whatever.
32:54
And you could simply run nix-build and then your Nix expression file, and then
33:00
`--option`, and parameter name and parameter value, right?
33:06
And it would simply inject that into your Nix script.
33:09
and then evaluate it and that's a very useful feature in general that kind of got
33:14
lost with Flakes.
33:15
I hope Flakes will have this in the future too.
33:19
However, if your option has a typo in it, Nix would simply do nothing with that,
33:30
right?
33:31
So it's like with JavaScript or similar stuff when you have a typo in a variable
33:35
name, it will simply invent a new one.
33:38
and you have like undefined values and so on.
33:41
And with this change, Nix actually warns you.
33:44
So it sees, sorry, I only have parameters A and B and C.
33:49
And the one that you provided is unknown to this set.
33:53
So this is an error now.
33:55
And this way you will never ever struggle any longer with expressions that for some
34:01
magic reasons don't work while you for hours just had a typo somewhere and didn't
34:06
notice it.
34:10
nice i guess Cole had such an incident enough something to make this PR so thank
34:22
you Cole Helbling for fixing this for us.
34:39
Hehehehe
34:44
What does that mean?
34:45
Right, so for example in flakes, flakes have a very specific format, right?
34:53
You have output categories like packages, devShells, nixosConfigurations,
34:58
nixosModules, overlays, blah blah blah.
35:02
That list will get longer over time, I guess.
35:06
However, for example, if you run...
35:11
yeah, so schema will make them even user -definable.
35:14
So the list will grow longer.
35:17
However, for example, let's stay on the basic case when you run nix build
35:21
myPackageName.
35:25
And your package is on the nix level, not really a derivation.
35:31
Then, yeah, it won't work.
35:33
Nix will tell you, sorry, this is not a derivation.
35:37
And if you ask, OK, what is it else?
35:42
Because you're
35:43
Of course, you didn't do this on purpose.
35:45
It's kind of a mistake, right?
35:47
Then it wasn't really easy to find out what it was.
35:50
Maybe you are like pointing at an attribute set and forgot to add the sub
35:57
attribute or something like this.
35:59
Then nix won't really tell you.
36:01
Even if you open your flake in the REPL just to browse on the command line to your
36:07
package path, then it will tell you, sorry, this is not a derivation.
36:10
And you're like, okay.
36:12
How the hell am I supposed to debug this now?
36:15
Well, you have to put it outside of the packages into some undefined, unspecified,
36:24
and then it'll tell you.
36:25
Of course, that's an idea.
36:29
However, even without doing that, the flake command will now print you that this
36:38
is not a derivation, but it will also print you what it found otherwise.
36:44
So you will see, this is not a path, but for example, a set or a list or whatever.
36:50
And then you will most probably recognize what you did wrong when you see this
36:55
output.
36:55
So this is another improvement that will make the next programmer's life easier.
37:02
Mm -hmm.
37:05
That's very nice.
39:30
So thank you to whomever fixed this quality of life improvement.
39:39
Thank you.
39:42
fetchTree now fetches git repositories shallowly by default.
39:51
I'm not familiar with fetchTree.
39:53
What is fetchTree?
39:56
Yeah, so fetchTree basically is able to clone Git repositories.
40:03
The thing is that Git repositories can be regarded as a tree of Git repositories
40:12
again, but then also as a tree of commits.
40:14
So this deep or shallow cloning of Git repositories can have two different
40:20
meanings on two different axes.
40:23
Mm -hmm.
40:23
is, if you check out a Git repository that, for example, has sub -modules, then
40:28
you maybe want to check out the sub -modules too.
40:31
So this is a yes -no decision.
40:35
And that's useful in different scenarios.
40:38
The other thing is, when you check out a Git repository, you may want to check it
40:44
out with all its branches and all its history, which is typically the case when
40:48
you Git clone something, like without Nix even.
40:50
If you just use Git.
40:52
And do `git clone `, it will download the whole history.
40:55
For example, on nixpkgs, this takes a very long time because it's such a huge
41:00
repository.
41:01
Same with the Linux kernel, for example.
41:05
And the thing is, you can tell Nix to please check out all the sub -modules, yes
41:11
or no.
41:13
But you weren't really able to tell Nix or...
41:17
it was was wasn't told by default to check out the whole history yes or not it was
41:22
checking out the whole history so typically all the nix nixpkgs and so on
41:30
which reference some github repository they wouldn't even use the git protocol
41:38
but download via HTTPS
41:42
And GitHub has this nice feature that when you provide a certain URL to a repository
41:48
and a commit, it will give you a tarball which doesn't even contain all the Git
41:54
information any longer.
41:55
So they basically used this GitHub feature as an easy way to have a shallow clone of
42:02
just the latest commit of a Git repository.
42:05
And the whole idea of
42:08
this change in fetchTree is now that you basically get the same very quick checkout
42:13
of just the latest commit with Git by default, which will accelerate many
42:22
derivations that already reference Git repositories because all the history that
42:31
happened before the latest commit is most of the time not relevant to the build,
42:36
right?
42:40
That's correct.
42:45
for this podcast, FullTimeNix for a different format episode.
42:58
In fact, it's very similar.
43:00
It's news, but it's news for nixpkgs instead of Nix.
43:08
We have been working on a program that will generate the script, or at least a
43:18
list of packages that are new since last week in nixpkgs and their descriptions and
43:25
their maintainers.
43:30
And for that, we want indeed to, well, we want to test the, both in testing and in
43:38
implementation, we want to clone nixpkgs.
43:48
and we don't want to clone the entire history.
43:53
But also we don't want only a depth of one latest commit.
43:59
We want a depth that is possibly by date because we know the date, but that would
44:08
have to kind of be resolved.
44:10
You just can't know that.
44:12
It's not exposed.
44:13
You have to get the commits and then examine them.
44:17
But details aside, we would need an arbitrary
44:23
depth.
45:04
So it's just that this little change in the default can have a huge impact on the
45:09
checkout performance of your repositories or your Nix builds.
45:14
But it's actually not really a revolution going on.
45:19
Yes.
45:20
And my point is only adjacent and not really related, but I thought it would be
45:28
interesting.
45:30
So thank you.
45:33
David Hauer
45:36
David Hauer for optimizing this for us.
45:46
should save AWS a lot of money.
45:50
Mm -hmm.
45:58
Store Object Info JSON Format now uses null rather than omitting fields.
46:09
Yeah, so this is a general thing in JSON parsing or deserializing JSON into your
46:17
data structures.
46:19
Generally, when you have some kind of optional field that is set, yes or no, for
46:24
example, I don't know, some extra information in your JSON object, there are
46:29
two ways to provide this when there is no information.
46:33
So if you have some field that is called, for example, extraField,
46:36
and it contains a string.
46:39
You can either, when you don't have this information and you create a JSON object,
46:43
say extraField, colon, null.
46:46
Then you say, okay, this field has no content.
46:50
Or you simply don't provide the extra fields.
46:55
So that's from, if your JSON structure generally has like 10 fields, you leave
47:01
one of them empty, then you have just nine fields and you don't provide this field at
47:06
all.
47:07
This is an annoyance for the programmer who is going to deserialize your JSON.
47:14
Because the question is, when this extra field is not there, is this an error
47:19
because someone provided a different kind of JSON object?
47:23
Or would you just say, okay, because this field is optional, we ignore the missing
47:30
existence of this field?
47:32
And the next team decided that wherever they create JSON objects, every field that
47:40
has no value will exist as a field and its value will be null.
47:48
This is not a huge change, but it will make all the code that will later
47:53
deserialize JSON again simpler to deal with because you can assume that all the
47:59
fields will be there.
48:00
It's just that their value will be distinguished if it's usefully set or not.
48:08
So this is something that the users will not really find relevant most of the time,
48:13
but everyone who is programmatically involved with dealing with JSON objects
48:18
from Nix tooling, they will experience a quality of life improvement.
48:28
This PR is by John Ericson.
48:30
Thank you, John Ericson
48:36
And the last one is large path warnings.
48:40
Introduce warnings for large paths.
48:43
What are large paths?
48:54
Yeah, so generally this feature is also useful, especially for beginners.
48:59
For example, something that I experience a lot when I'm helping organizations to
49:06
adopt Nix is that many people will try to Nixify huge projects and then many
49:12
projects are, I don't know, I mean, one gigabyte of source code is something that
49:18
happens in real life.
49:20
And then, of course, in 1 gigabyte of source code, you will have multiple
49:24
packages that simply live in the same repository.
49:29
And then when you want to build these packages, what happens very often is
49:33
people use the src attribute and make derivation and just put in huge folders
49:40
that contain much, much more than the actual package source code.
49:47
Imagine you have a Nix package whose source attribute points to 1GB source
49:53
folder and you change the readme for example.
49:58
Then you have one gigabyte of changed source folder.
50:01
And if you rebuild this Nix derivation, it will copy the whole one gigabyte of source
50:07
code into your Nix store again and rebuild it all due to relevant or irrelevant
50:14
changes.
50:15
So this is generally a code smell in projects where the answer is better
50:21
project structure with subfolders and or source filtering.
50:27
But generally, what you see as a user is your builds are extremely slow.
50:34
And then the question is, why are they so slow?
50:36
Even without knowing the project structure, you want to know why they are
50:39
so slow.
50:41
And what you can do now is there's a new option.
50:44
It's called `--warn-large-path-threshold`.
50:50
And it takes one parameter, the size of the...
50:53
megabytes to warn after and if you say for example warn large path threshold 100m, it
51:01
Nix will warn you whenever it copies more than 100 megabytes of folder content into
51:08
the Nix store and then you can see I didn't intend to copy so much.
51:13
Thank you for this warning line right now and then you will see which derivation
51:19
copies so much stuff into the store.
51:22
So this is
51:23
Imagine for example you have a repository with, I don't know, for example a video
51:28
folder with 10 gigabytes of videos and somehow you are accidentally copying this
51:34
all to the store, right?
51:35
There was...
51:37
You could look into your nix store paths if they contain stuff that you wouldn't
51:42
suspect to be useful but that would be very laborious and with this new warning
51:47
it will become very easy to check if you are accidentally copying too much stuff,
51:52
right?
51:54
Yes, is there a default for this or is it off?
52:00
It's off by default.
52:02
Hmm.
52:03
I mean, if you really intend to copy so much stuff into the nix store, then it's
52:08
perfectly fine to do that, right?
52:12
Rarely.
52:14
Rarely is.
52:15
but legitimate and it's completely thinkable that this is legitimate in
52:21
specific cases, right?
52:23
Most of the time, of course, it's not and now it's I mean, it's easy to simply add
52:28
this warning flag, get a few warnings printed and then you tell people like look
52:34
We are not filtering in this package now is copying 500 megabytes of source code
52:39
into the nix store and after 10 rebuilds with
52:44
little irrelevant changes you have five gigabytes of wasted space on your disk and
52:51
that's unnecessary let's fix that right so it's easier to find out which is the
52:58
package that actually does that
53:01
I kind of am like fantasizing at the moment that Nix would...
53:13
read somehow read all the files
53:22
and try to somehow intelligently figure out that they're all source code or maybe
53:34
otherwise tiny.
53:41
To help with preventing this, I know that we're supposed to be engineers and we're
53:49
supposed to be careful about this.
53:51
But...
53:54
Sometimes you don't even know what exactly you have in your tree because projects are
54:00
so big.
54:01
Mm.
54:03
Maybe there are assets hidden where you didn't know that they are.
54:09
Maybe Nix can help you and say, you know what?
54:14
I noticed you have videos in your source.
54:23
and warn you for that.
54:25
So that would be a warning by file type instead.
54:29
That would be most probably very useful to many, but I'm not sure if this is the
54:34
right layer to do this.
54:35
I mean, if you are not filtering your source code at all, Nix will simply take
54:43
it all and use it.
54:44
And if you would add intelligence to Nix, that actually looks through the whole
54:49
folder to inspect content of each file.
54:52
This could be extremely slow then, because this would be a very file system
54:58
intensive.
54:58
workload.
55:01
Of course, you could have extra tooling that simply evaluates all your Nix
55:05
expressions and then goes through the list of derivations which lists up all the
55:10
paths that they're going to use.
55:13
And then it could kind of scan these paths and see what they contain and then warn
55:22
you in specific cases.
55:24
But I guess even this
55:26
I mean, you would need some kind of heuristic, right?
55:28
Because if you are not the build system which actually uses this, it will be hard
55:34
to distinguish what will be useful and what not.
55:37
Of course, the build system could tell you, hey, look, I used the following 100
55:42
files and the rest of the 1 ,000 files in this repository hasn't been touched by
55:47
anyone.
55:48
But that would also not be trivial.
55:50
You would have to kind of trace every processes.
55:55
file system operations to see what's actually used.
55:59
So something like this exists, but this is very invasive tooling.
56:05
I think it's a...
56:07
Okay.
56:08
track, read the whole tree after the build is done and say, well, because file
56:17
systems can keep track of what file has been read, when's the last time it's been
56:23
read.
56:25
I see.
56:26
Yeah.
56:26
Okay.
56:26
True.
56:28
However, just because something hasn't really been used already, the question is,
56:34
if you change some kind of build flag, will the file then be used or not?
56:43
So this is something that is, they're like, from the outside domain, what the
56:50
build system does with which file could look like completely random.
56:55
And then the question is, if the tooling has too many false positives, then it will
57:02
be annoying to use, right?
57:06
And then all the nice ideas and intentions don't lead to a nice user experience,
57:11
while maybe potentially making all the builds slower, right?
57:15
So that's not too easy to solve.
57:21
Yes, so this PR adds a simple tool that we can explicitly choose to use and this is
57:35
by Elco Dolstra.
57:37
Thank you, Elco.
57:41
Yeah, this is nice and easy to use, right?
57:42
You can say, OK, 6 megabytes are too much, 600 megabytes are too much, 6 gigabytes
57:49
are too much.
57:52
For example, when you work with VirtualBox, VirtualBox has a huge source
57:56
code folder.
57:58
And the build system doesn't make it easy to just build portions of that.
58:04
So you typically end up building the whole thing.
58:08
And it simply has huge source closures that you're going to live with.
58:15
With other projects, you could define completely different thresholds.
58:20
And you as the developer typically know what threshold you want to use on which
58:25
project.
58:25
And that's simply easy to do.
58:29
So they are only providing the mechanics of warning after a specific threshold.
58:35
But this policy has to be defined by you ad hoc on each project, which is
58:41
completely fine, right?
58:46
Yeah, well, Nix is very useful and there are always ways to make it more useful and
58:52
more pleasant to use.
58:57
Right.
59:03
That's all, isn't it?
59:05
Yes, that's all.
59:10
So I'd like to thank all the Nix developers and you, Jacek, for sharing
59:20
these updates and explaining them to us.
59:24
And I'm looking forward to the next Nix release.
59:30
Yes, thank you for having me.
59:32
It was a pleasure again.
59:35
Be well, Jacek.
59:36
Everyone, thank you for listening.
59:38
Have a nice day.