Rust Coreutils 0.12.0 Release:
We are happy to announce the release of Rust Coreutils 0.12.0 - a release focused on compatibility fixes for our Ubuntu friends.
Ubuntu ships uutils coreutils as its default coreutils implementation, and this cycle was driven by what their users and maintainers reported: the bugs they asked us to prioritize are all fixed here, along with a long tail of smaller divergences from GNU. Few of these are glamorous, but they are exactly what breaks a script that has worked for twenty years.
Alongside that, we removed the last C dependency of expr (onig is replaced by the pure-Rust fancy-regex), continued the Windows port (nice, nohup and hostid now build there, and uucore's filesystem layer was rewritten with safe Win32 wrappers) and landed another round of security hardening on ls, sort and install.
Also, some contributions had carried comments and code over from GNU coreutils, against our own rule
that uutils is an original implementation. We reviewed the tree and rewrote what we found, in comments and in the few functions that followed the C too closely; no behaviour change. The GNU coreutils maintainers have been notified.
GNU coreutils 9.12 was released during our cycle (kudos to the team!), and some of the work went that way too. Chasing compatibility tends to turn up things worth fixing or improving on both sides.
We also made it official: a utility being significantly slower than GNU is now treated as a bug, not a wishlist item. If you hit one, please report it as a bug.
We are now at 653 passing, 21 failing, 0 errors on the GNU test suite.
GNU Test Suite Compatibility:
Note: the suite grew by 7 tests (backport of GNU tests) and 8 more tests are skipped on our reference runner, so the raw pass count is flat while the failures went down again. Counting only the tests that actually run, we are at 653/674 (96.9%), our best ratio so far, and the second release in a row with no erroring test. We are comparing against Coreutils 9.11.
Highlights:
-
Ubuntu Compatibility - The issues Ubuntu asked us to fix first are all fixed: data loss in a parallel install -D(#12355),duon 32-bit architectures (#11848),lsACL dereferencing (#13234),cp -Rname collisions (LP#2167118) andchmod -Rthrough symlinks (LP#2167122)
- A whole family of options now accept a hyphen-leading value, like GNU does: csplit --suffix-format,paste -d,join -t,stat --format/--printf/-cand all ofnumfmt's options. A-in a delimiter or a format string is data, not a flag
- rmno longer segfaults on very deeply nested directories
- sort -napplies to the key instead of the whole line, echoes a disorder line verbatim instead of reformatting it as a number, and no longer keeps every input file open while sorting
- basenametrailing-separator handling for dot components,- realpathexit status in quiet mode,- lnkeeping the destination when a forced link fails,- csplitalways creating the final split even when empty,- fmt -x/- -X,- printftreating a- --past the format as an argument,- testrejecting an- -a/- -othat ends the expression,- echo -eno longer expanding- \u,- \Uand- \"
- ls:- diredfixes, a tie in the time sort broken by the name, name sort for- -u/- -cin long format, capability markers that respect the dereference config;- dirand- vdirmatch the GNU default quoting
- cp:- --no-preserveis retained over a later- --preserve, and- --attributes-onlygives the destination the source's type
- Error messages and exit codes aligned with GNU in uniq,chgrp,factor,date,expr,od,tty,tsortandpathchk
- truncatefinally implements- --io-blocks, and- dateaccepts multiple- --reference
- The issues Ubuntu asked us to fix first are all fixed: data loss in a parallel
-
Security Hardening - ls: report- stat()failures on directory entries instead of silently showing wrong metadata
- sort: temporary files are created with mode 0600 instead of the umask, the- -ooutput copy is kept private too, and a failure to truncate the output file is now reported
- install: the fd-based destination is created exclusively and at 0600
- rm,- chmod,- chown,- chgrp:- --preserve-rootcompares- (st_dev, st_ino)rather than the path
- chconwalks directories with- DirFdinstead of- fts
- SECURITY.mdnow requires a boundary crossing, not just a GNU divergence, for a report to be treated as a vulnerability
-
Robustness - Panics and overflows fixed in stat(out-of-range octal escape),ls(a color style with no attributes),split(a huge-n l/K/Nchunk count),truncate(a size abovei64::MAXno longer creates the file),cksum(an over-length BLAKE2b digest in a check file) anduucore(invalid input lines)
- tsorthandles invalid UTF-8 input and flushes stdout properly;- datereports invalid UTF-8 lines in- -finput;- truncatesupports non-UTF-8 paths for- --reference
- Panics and overflows fixed in
-
Dependency Diet - exprreplaces- onigwith fancy-regex, removing a C dependency and the CI hacks that came with it. As a side effect,- exprswitched to leftmost-longest regex matching, matching GNU
- mvreplaces- fs_extrawith std directory sizing;- fts-sysis gone;- cksumuses aws-lc instead of vendored OpenSSL on MSVC, so a Perl install is no longer needed to build on Windows
- bstrdefault features disabled,- uucore's hand-rolled process functions dropped in favor of- rustix
-
Cross-platform Reach - Windows: nice(with a Priority Class mapping),nohupandhostidnow build,dirnamehandles backslashes, user locale detection landed, checksum paths use forward slashes, andmv,sync,hostname,touchandrmmoved to safe std APIs
- uucoreon Windows:- fsextrewritten with safe Win32 wrappers, real free space and mount paths;- dfresolves paths by mount prefix
- New and improved targets: aarch64-unknown-linux-ohos, AIX (fsext), Haiku (fs), Hurd (signals), NetBSD (safe-traversal), plus widerprocess,utmpxandentriessupport
- WASI: cpattribute preservation and reflink handling,mktemptemporary directories,tailintegration coverage
- macOS: tailstdin-redirect, pid and follow-name tests re-enabled
- Windows:
-
Performance - sortgained benchmarks for spilling, checking and shared prefixes;- dubenchmarks now warm the OS page cache before timing
- cpavoids resolving the cwd for absolute recursive copies;- tsortgives the OS a sequential access hint
- A significant performance gap with GNU is now stated in the README as a bug, and we handle it as one
-
Contributions: This release landed 191 pull requests from 16 new contributors
Call to Action:
Try it in your browser - Online Playground powered by WebAssembly
Help us translate - Contribute to Rust Coreutils on Weblate
Sponsor us on GitHub to accelerate development: github.com/sponsors/uutils
What's Changed
basename
- basename: fix trailing-separator handling for dot components by @wangyusheng1985 in #14365
chcon
- chcon: walk directories with DirFd instead of fts by @sylvestre in #14413
chgrp
- rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411
- chgrp: strip error when changing group of file fails by @Devel08 in #14403
chmod
- rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411
- chmod: fix clippy lints by @xtqqczze in #14447
- chmod: do not check umask on skipped symlinks during recursion by @okaythi in #14512
- chmod: gate by target support by @xtqqczze in #14585
chown
- rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411
cksum
- cksum: use aws-lc instead of vendored openssl on MSVC to avoid perl by @oech3 in #14559
- cksum: do not abort on an over-length BLAKE2b digest in a check file by @Dijaa in #14508
cp
- cp: avoid resolving cwd for absolute recursive copies by @varranvar in #14336
- fix(wasi): cp attribute preservation and reflink handling by @eduardomourar in #13685
- cp: fix copying entries sharing parent directory name with -R . by @okaythi in #14530
- cp: retain --no-preserve over later --preserve by @wtcpython in #14555
- cp: --attributes-only must give the destination the source's type by @sylvestre in #14622
csplit
- csplit: accept a hyphen-leading --suffix-format value by @arbelonson-source in #14320
- csplit: always create the final split, even when empty by @yuefdev in #14565
date
- date: report invalid UTF-8 lines in -f file/stdin input by @alexchen-sys in #14280
- date: wrap errors with thiserror by @Devel08 in #14333
- date: exit with 1 for expected file, got directory by @Devel08 in #14435
- date: rewrite test to remove thread::sleepby @cakebaker in #14535
- date: allow multiple --referenceby @cakebaker in #14546
dd
- dd: simplify windows stdin in match guard by @hlsxx in #14347
- dd: fix broken man page formatting by @hlsxx in #14348
dir
- dir, vdir: match GNU default quoting by @ChihweiLHBird in #14538
dirname
du
- du: warm OS page cache before timing in benchmarks by @sylvestre in #13958
- du: fix EOVERFLOW on 32-bit Linux by @sylvestre in #14548
echo
- echo: don't expand \u, \U and " in -e by @SichenLiang in #14475
env
- env: trace the environment changes under -v by @Socialpranker in #14401
- Revert "env: trace the environment changes under -v" by @cakebaker in #14493
expr
- expr: replace onig with fancy-regex by @wtcpython in #14329
- expr: re-enable test_regex_caret on Windows arm64 by @wtcpython in #14429
- expr: exit with on a write error by @Devel08 in #14500
- expr: switch to leftmost-longest regex matching behavior by @keith-hall in #14527
factor
fmt
- fmt: honor -x and -X instead of always matching prefixes exactly by @MaxFreedomPollard in #14417
head
- head, tail: reject multiplier suffixes that GNU rejects by @kennywillbe in #14228
hostid
- hostid: remove unuseful lock() by @oech3 in #14426
- hostid: build for Windows & remove unsafe by @oech3 in #14427
hostname
- mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
- hostname: gate by target support by @xtqqczze in #14612
install
- install: fix EEXIST race losing files in parallel install -D by @eyupcanakman in #12429
- test_install.rs: add test case for eCryptfs by @oech3 in #14545
- install: create the fd-based destination exclusively and at 0600 by @sylvestre in #14625
join
- join: accept a hyphen-leading -t separator, quote it in the error by @arbelonson-source in #14318
link
ln
- ln: keep the destination when a forced link fails by @nagendramohan in #14574
logname
ls
- ls: fix panic on a color style with no attributes by @harshasiddartha in #14373
- ls: avoid accessing by index by @oech3 in #14380
- ls: apply dir and vdir format defaults in Config::from by @ChihweiLHBird in #14450
- ls: break a tie in the time sort with the name by @luantaraschi in #14049
- ls: fix two dired bugs by @sylvestre in #14499
- fix(ls): ensure ACL and capability markers respect dereference config by @sachinskss in #13368
- ls: use name sort for -u/-c in long format by @ChihweiLHBird in #14573
- ls: report stat() failures on directory entries (GHSA-fhfr-qj69-pq5m) by @sylvestre in #14553
mknod
mktemp
- fix(mktemp): add wasi platform support for temporary directory handling by @eduardomourar in #13786
mv
- mv: avoid accessing by index by @oech3 in #14379
- mv: test broken symlink file/dir as target by @hlsxx in #10739
- mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
- mv: replace fs_extra with std directory sizing by @wtcpython in #14432
- generalize some mv code for ln, cp by @sylvestre in #14623
nice
- nice: prepare supporting Windows by @oech3 in #14452
- nice: map Windows Priority Class by @oech3 in #14457
nohup
- nohup: simplify by .into() by @oech3 in #14476
- nohup: build for Windows by @oech3 in #14461
- nohup: move platform code into platform/{unix,windows}.rs by @sylvestre in #14525
numfmt
- numfmt: accept hyphen-leading -d/--suffix/--padding values by @arbelonson-source in #14322
- numfmt: accept hyphen-leading values for the remaining options by @sylvestre in #14362
- numfmt: zero-pad the number, not the unit suffix by @Socialpranker in #14393
od
- od: say the skip ran past the end of the combined input by @Socialpranker in #14399
paste
- paste: accept a hyphen-leading -d delimiter list by @arbelonson-source in #14321
pathchk
- pathchk: reject empty path in portability mode by @cakebaker in #14506
- pathchk: remove #![allow(unused_must_use)]by @cakebaker in #14507
pinky
- pinky: make it more rust idiomatic by @sylvestre in #14424
pr
- pr: three small refactorings by @cakebaker in #14534
printf
- printf: honor POSIXLY_CORRECT for the character-constant warning by @SichenLiang in #14416
- printf: treat a --past the format as an argument by @Socialpranker in #14394
ptx
- ptx: make the code more rust idiomatic + fix some comments by @sylvestre in #14595
realpath
- realpath: fix exit status in quiet mode by @ChihweiLHBird in #14442
rm
- rm, chmod, chown, chgrp: compare (st_dev, st_ino) for --preserve-root by @sylvestre in #14411
- mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
- rm: fix segfault on very deeply nested directories by @sylvestre in #14554
sort
- sort: apply -n to the key, not the whole line by @Socialpranker in #14331
- sort: add benchmarks for spilling, checking and shared prefixes by @sylvestre in #14367
- sort: echo the disorder line verbatim, not reformatted as a number by @Socialpranker in #14389
- sort: do not hold all inputs open while sorting by @Hekzory in #14423
- sort: simplify get_unit by @oech3 in #14505
- sort: gate tests correctly for BSD by @xtqqczze in #14533
- sort: report a failure to truncate the -o output file by @sylvestre in #14617
- sort: create temporary files with mode 0600 instead of the umask by @sylvestre in #10647
- sort: keep the output-file copy private too by @sylvestre in #14624
split
- split: fix overflow on a huge -n l/K/N chunk count by @harshasiddartha in #14374
stat
- stat: accept a hyphen-leading --format/--printf/-c value by @arbelonson-source in #14323
- stat: fix overflow on out-of-range octal escape by @harshasiddartha in #14371
stty
sync
- mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
tail
- head, tail: reject multiplier suffixes that GNU rejects by @kennywillbe in #14228
- test(tail): fix follow name truncate failing on macos by @eduardomourar in #14438
- tail: enable working stdin-redirect and pid tests on macOS by @wtcpython in #14433
test
- test: reject a -a/-othat ends the expression by @Socialpranker in #14395
- test: Honor PATHEXT on Windows by @lhecker in #14481
touch
- mv, sync, hostname, touch, rm: use safe std APIs on Windows by @wtcpython in #14445
truncate
- truncate: reject a size above i64::MAX instead of creating the file by @AlejandroCoronadoN in #13902
- fix(truncate): implement --io-blocks by @ThomasHabets in #13432
- truncate: group fifo-related tests in mod by @cakebaker in #14385
- truncate: support non-UTF8 paths for --referenceby @cakebaker in #14386
- truncate: remove unreachable code by @cakebaker in #14422
tsort
- tsort stdout flushing fix and several error messages handling ameliorations by @anastygnome in #14384
- tsort: handle invalid utf8 in input by @anastygnome in #14464
- tsort: add sequential file access hint on windows and where available on unix by @anastygnome in #14463
tty
- tty: add missing newline to output by @ChrisX101010 in #10252
- tty: avoid exit & show write error by @oech3 in #14366
uniq
- uniq: drop the "Could not open" prefix from open errors by @abhijeetvichare76 in #14262
uptime
- uptime: accept hour or minute in pretty-print tests by @ChihweiLHBird in #14547
vdir
- dir, vdir: match GNU default quoting by @ChihweiLHBird in #14538
who
- who: make the code more rust idiomatic by @sylvestre in #14410
uucore
- uucore: update fsext to build on aix by @xtqqczze in #14430
- uucore: fix a fsext test by @xtqqczze in #14446
- uucore: Avoid panic on invalid input lines by @lhecker in #14470
- uucore: use libc implementations of major,minor, andmakedevby @xtqqczze in #14471
- uucore: Implement user locale detection for Windows by @lhecker in #14477
- uucore: Use forward slashes for checksum paths on Windows by @lhecker in #14469
- uucore: Fix strip_errno doctest for Windows by @lhecker in #14482
- uucore: remove process functions by @xtqqczze in #11158
- uucore: os_str_from_bytes does not need to alloc by @lhecker in #14466
- uucore(windows): rewrite fsext with safe Win32 wrappers, real free space and mount paths; df(windows): resolve paths by mount prefix by @nikolalukovic in #14455
- uucore: support fs feature for haiku by @xtqqczze in #14570
- uucore: support process feature for more platforms by @xtqqczze in #14579
- chore(uucore): fix clippy lints for fsext feature by @xtqqczze in #14581
- uucore: fix test_crlf_across_blocks not compiling on Windows by @abhijeetvichare76 in #14254
- uucore: support signals feature for hurd by @xtqqczze in #14454
- uucore: fix entries feature by @xtqqczze in #14601
- uucore: support utmpx feature for more platforms by @xtqqczze in #14584
- uucore: escape embedded locale content instead of using a raw string by @sylvestre in #14616
- uucore: support safe-traversal feature for netbsd by @xtqqczze in #14606
Testing
- tests: don't pipe input into commands that exit before reading it by @Socialpranker in #14431
Code Quality & Cleanup
- fix: redundant_homepage cargo lint by @xtqqczze in #14440
- fix: missing_lints_inheritance cargo lint by @xtqqczze in #14441
- fix outdated comments about onig by @oech3 in #14448
- fix: needless_range_loop clippy lint by @xtqqczze in #14444
- fix: unused_dependencies cargo lint by @xtqqczze in #14443
- simplify matches! and inequalities in match by @oech3 in #14502
- remove some matches! & use .get() by @oech3 in #14503
- use .is_root() of rustix::process by @oech3 in #14519
- fix: missing_const_for_thread_local clippy lint by @xtqqczze in #14462
- chore: add missing newlines to source header by @xtqqczze in #14572
- chore: fix clippy missing_const_for_thread_local lint by @xtqqczze in #14613
CI & Build
- Revert "remove rust-version from fuzz" by @xtqqczze in #14105
- fetch-gnu.sh: fix hardcoded pathes to GNU test binary by @oech3 in #14332
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #14335
- Cargo.toml: remove unused fts-sysby @cakebaker in #14425
- CICD.yml: remove a hack for onig by @oech3 in #14428
- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #14439
- openbsd.yml: remove C toolchain by @oech3 in #14449
- ci: fix style_empty_assertionsjob by @cakebaker in #14451
- Cargo.toml: remove timeout & kill from feat_require_unix_core by @oech3 in #14453
- freebsd.yml: fix warning by @oech3 in #14488
- Fix the GNU reference + send a notification to coreutils-tracking for data refresh by @sylvestre in #14492
- github copilot: Create code review guidelines for uutils/coreutils by @sylvestre in #14510
- cargo: use workspace inheritance for dev-dependencies by @xtqqczze in #14517
- CICD.yml, test_openssl: drop non-*sum & avoid release build by @oech3 in #14473
- CI: fix artifact name for Cygwin by @oech3 in #14494
- Cargo.toml: reconsider feat_{wasm, common_core,Tier1} for wasi by @oech3 in #14543
- CICD.yml: publish binaries with static openssl by @oech3 in #14539
- CI: remove brew install by @oech3 in #14557
- fix(build): adjust check for cross-platform building by @xtqqczze in #14568
- CICD.yml: drop unnecessay setup, simplify by @oech3 in #14528
- fix cspell by @oech3 in #14580
- GnuTests: autoreconf for portability by @oech3 in #14575
- chore(cargo): use workspace inheritance for dependencies by @xtqqczze in #14596
- cargo: avoid safe_traversal feature on unsupported platforms by @xtqqczze in #14588
- deny.toml: remove windows-sys 0.59and related crates from skip list by @cakebaker in #14611
Documentation
- README.md: clear statement wrt performances by @sylvestre in #14345
- uudoc: improve loop by @oech3 in #14361
- CONTRIBUTING.md: update link to DL GNU coreutils by @oech3 in #14608
- SECURITY.md: require a boundary crossing, not just a GNU divergence by @sylvestre in #14615
Platform Support
- tests: enable tail integration coverage on WASI by @DePasqualeOrg in #13805
- coreutils: replace /proc/ by /dev/ by @oech3 in #12553
- ohos: add aarch64-unknown-linux-ohos support (fsext/hostid/date) by @YodonTan in #14288
Dependency Updates
- build(deps): update rust crates by @xtqqczze in #14338
- fix(deps): refactor rustix/nix package configuration by @xtqqczze in #14337
- chore(deps): update rust crate rstest to 0.27.0 by @renovate[bot] in #14418
- chore(deps): update vmactions/freebsd-vm action to v1.5.6 by @renovate[bot] in #12977
- chore(deps): update rust crate console to v0.16.6 by @renovate[bot] in #14491
- Revert "chore(deps): update vmactions/freebsd-vm action to v1.5.6 (#12977)" by @xtqqczze in #14513
- chore(deps): update vmactions/freebsd-vm action to v1.5.5 by @xtqqczze in #14516
- chore(deps): update rust crate jiff to v0.2.37 by @renovate[bot] in #14521
- chore(deps): update rust crate textwrap to v0.16.4 by @renovate[bot] in #14486
- chore(deps): update rust crate clap to v4.6.7 by @renovate[bot] in #14566
- chore(deps): update rust crate clap_complete to v4.6.10 by @renovate[bot] in #14567
- chore(deps): update rust crate clap_complete to v4.6.11 by @renovate[bot] in #14582
- chore(deps): update ICU derive crates by @xtqqczze in #14587
- chore(deps): update vmactions/freebsd-vm action to v1.5.6 by @xtqqczze in #14589
- chore(deps): update rust crate platform-info to v2.2.0 by @renovate[bot] in #14597
- deps: disable default features of bstrby @cakebaker in #14605
Version Management
- Prepare version 0.12.0 by @sylvestre in #14578
New Contributors
- @kennywillbe made their first contribution in #14228
- @ChrisX101010 made their first contribution in #10252
- @wangyusheng1985 made their first contribution in #14365
- @harshasiddartha made their first contribution in #14371
- @varranvar made their first contribution in #14336
- @SichenLiang made their first contribution in #14416
- @YodonTan made their first contribution in #14288
- @Hekzory made their first contribution in #14423
- @MaxFreedomPollard made their first contribution in #14417
- @ChihweiLHBird made their first contribution in #14450
- @lhecker made their first contribution in #14470
- @okaythi made their first contribution in #14530
- @keith-hall made their first contribution in #14527
- @sachinskss made their first contribution in #13368
- @yuefdev made their first contribution in #14565
- @Dijaa made their first contribution in #14508
Full Changelog: 0.11.0...0.12.0