Modify

Opened 7 years ago

Closed 7 years ago

Last modified 4 years ago

#8951 closed defect (worksforme)

toolchain prepare target spawns multiple unpacking of each package

Reported by: dirtyfreebooter <openwrt-devel@…> Owned by: developers
Priority: low Milestone: Barrier Breaker 14.07
Component: toolchain Version: Trunk
Keywords: gcc, uclibc, parallel, build, tar, gzcat, bzcat, multiple Cc:

Description

Doing a parallel build, in the toolchain target, when packages are prepared and untar'd, the build process spawns multiple instances of the unpacking (tar/gzcat/bzcat) simultaneously. It results in processes simply overwriting each other. Most times it doesn't result in file corruption, but on rare occasion, it can, causing strange errors from the build. Plus it just adds extra time to build, stress on the disk, etc.

Both gcc and uClibc suffer from this. As a note, I always see 3 instances of each unpacking no matter what is specified as -j or in menuconfig for parallel jobs.

I am willing to do work here, but can someone maybe point me to where in the makefile structures the prepare dependencies are handled for the toolchain? Thanks.

Attachments (0)

Change History (7)

comment:1 Changed 7 years ago by dirtyfreebooter <openwrt-devel@…>

Let me add that I now see this happens for all packages, not just the toolchain targets...

$ ps axww | grep tar | egrep "gzcat|gunzip|gzip|bzcat|bunzip|bzip"
25632 s004  SN+    0:00.01 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/dropbear-0.52.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/dropbear-0.52/.. -xf - 
25663 s004  SN+    0:00.05 gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/dropbear-0.52.tar.gz
25664 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/dropbear-0.52.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/dropbear-0.52/.. -xf - 
25667 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/dropbear-0.52.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/dropbear-0.52/.. -xf - 
25803 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/busybox-1.17.3.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/busybox-1.17.3/.. -xf - 
25811 s004  SN+    0:00.22 bzcat /Volumes/OpenWrt/DIR825/trunk/dl/busybox-1.17.3.tar.bz2
25813 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/busybox-1.17.3.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/busybox-1.17.3/.. -xf - 
25815 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/busybox-1.17.3.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/busybox-1.17.3/.. -xf - 
27392 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/ncurses-5.7.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/libncurses/ncurses-5.7/.. -xf - 
27399 s004  SN+    0:00.05 gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/ncurses-5.7.tar.gz
27400 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/ncurses-5.7.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/libncurses/ncurses-5.7/.. -xf - 
27401 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/ncurses-5.7.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/libncurses/ncurses-5.7/.. -xf - 
32075 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/openssl-1.0.0d.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/openssl-1.0.0d/.. -xf - 
32081 s004  SN+    0:00.02 gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/openssl-1.0.0d.tar.gz
32082 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/openssl-1.0.0d.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/openssl-1.0.0d/.. -xf - 
32083 s004  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; gzip -dc /Volumes/OpenWrt/DIR825/trunk/dl/openssl-1.0.0d.tar.gz | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/target-mips_r2_uClibc-0.9.32/openssl-1.0.0d/.. -xf - 

comment:2 Changed 7 years ago by nbd

  • Resolution set to fixed
  • Status changed from new to closed

the toolchain spawning multiple instances for the same package was fixed in r24969

comment:3 Changed 7 years ago by dirtyfreebooter <openwrt-devel@…>

  • Resolution fixed deleted
  • Status changed from closed to reopened

This problem has come back, but in a slightly different form. I am using r26172 and the above is happening. It is happening for more than just toolchain/gcc target, from the above it is happening to me on every package. Also it seems to spawn 3 instances no matter the -j<n> argument.

$ ps axww | grep tar | egrep "gzcat|gunzip|gzip|bzcat|bunzip|bzip"
35900 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/host/u-boot-2010.12/.. -xf - 
35906 s000  RN+    0:00.32 bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2
35907 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/host/u-boot-2010.12/.. -xf - 
35909 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/host/u-boot-2010.12/.. -xf -

and later in the build

$ ps axww | grep tar | egrep "gzcat|gunzip|gzip|bzcat|bunzip|bzip"
25079 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/binutils-2.20.1.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/toolchain-mips_r2_gcc-linaro_uClibc-0.9.32/binutils-2.20.1/.. -xf - 
25080 s000  SN+    0:00.00 bash -c bzcat /Volumes/OpenWrt/DIR825/trunk/dl/linux-2.6.37.3.tar.bz2 | /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/toolchain-mips_r2_gcc-linaro_uClibc-0.9.32 -xf -
25082 s000  SN+    0:00.15 bzcat /Volumes/OpenWrt/DIR825/trunk/dl/binutils-2.20.1.tar.bz2
25083 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/binutils-2.20.1.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/toolchain-mips_r2_gcc-linaro_uClibc-0.9.32/binutils-2.20.1/.. -xf - 
25084 s000  RN+    0:00.03 bzcat /Volumes/OpenWrt/DIR825/trunk/dl/linux-2.6.37.3.tar.bz2
25085 s000  RN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/binutils-2.20.1.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/toolchain-mips_r2_gcc-linaro_uClibc-0.9.32/binutils-2.20.1/.. -xf - 
25093 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/gcc-linaro-4.5-2011.02-0.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/toolchain-mips_r2_gcc-linaro_uClibc-0.9.32/gcc-linaro-4.5-2011.02-0/.. -xf - 
25094 s000  SN+    0:00.02 bzcat /Volumes/OpenWrt/DIR825/trunk/dl/gcc-linaro-4.5-2011.02-0.tar.bz2
25095 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/gcc-linaro-4.5-2011.02-0.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/toolchain-mips_r2_gcc-linaro_uClibc-0.9.32/gcc-linaro-4.5-2011.02-0/.. -xf - 
25096 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/gcc-linaro-4.5-2011.02-0.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/toolchain-mips_r2_gcc-linaro_uClibc-0.9.32/gcc-linaro-4.5-2011.02-0/.. -xf -

It is now happening to every package unpacking, unless I am reading the output of ps wrong.

comment:4 Changed 7 years ago by nbd

  • Resolution set to worksforme
  • Status changed from reopened to closed

I don't think it's possible to prevent make from parallelizing the unpacking, while keeping the parallel build feature for other stuff without making the parallel parts less effective.

If it's unpacking multiple things at the same time, then that's not a bug, please only reopen this ticket if this actually results in build breakage.

comment:5 Changed 7 years ago by dirtyfreebooter <openwrt-devel@…>

Its not multiple packages, its the same packages being unpacked multiple times. (I understand multiple unpacking of different packages is needed for a much efficient parallel build.) This leads to race condition and some times file corruption that will cause builds to fail (one unpacking writing the same files as another simultaneously). Since its a race condition, usually re-building solves the issue.

Again, maybe I misreading the output, but are there not 3 `bzcat | gnutar' going here?

$ ps axww | grep tar | egrep "gzcat|gunzip|gzip|bzcat|bunzip|bzip"
35900 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/host/u-boot-2010.12/.. -xf - 
35906 s000  RN+    0:00.32 bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2
35907 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/host/u-boot-2010.12/.. -xf - 
35909 s000  SN+    0:00.00 bash -c . /Volumes/OpenWrt/DIR825/trunk/include/shell.sh; bzcat /Volumes/OpenWrt/DIR825/trunk/dl/u-boot-2010.12.tar.bz2 | trapret 2 /usr/bin/gnutar -C /Volumes/OpenWrt/DIR825/trunk/build_dir/host/u-boot-2010.12/.. -xf -

or is all that something else? Each is getting its own PID? I see each one of those PIDs 35900, 35907, 35909 with a bzcat piped into gnutar. I don't know why 35906 is just bzcat on its own? I guess I could use lsof during the build to see what is writing what exactly.

comment:6 Changed 7 years ago by nbd

I think you're misreading that one. There are three shell processes, but only one bzcat (and probably one tar process not covered by your egrep expression)
The number of shell processes here is completely irrelevant, it's simply an artifact of how the shell handles expressions internally.

comment:7 Changed 4 years ago by jow

  • Milestone changed from Attitude Adjustment 12.09 to Barrier Breaker 14.07

Milestone Attitude Adjustment 12.09 deleted

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.