ldd leaves the machine unresponsive

[Anton Shterenlikht]

Just updated to ia64 r205248

If my problem is due to my mis-configuration,
I apologise in advance.

I run this shell script after each upgrade
and 'make delete-old-libs' to check
if any shared objects need to be rebuilt:

<start script>

#!/bin/sh

for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done

<end script>

After the upgrade to r205248, the script
freezes at seemingly random points.

I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

On the serial console I cannot get the prompt
after entering the root password.

I have top(1) running interactively in another window.
The sh process is in "getblk" state, and ignores kill -9.
But there's no ldd process.

And shutdown requests are also ignored:

# shutdown -r now
Shutdown NOW!
shutdown: [pid 8019]
#
and nothing happens after that

So I have to do a cold reset via MP.

On ia64 r204322, this script causes no problems.

Please advise

many thanks
anton

[Marcel Moolenaar]

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>
After the upgrade to r205248, the script
freezes at seemingly random points.

The script is pretty disk intensive. How long do you wait before
giving up on output?

[Anton Shterenlikht]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

Post by Anton Shterenlikht
On the serial console I cannot get the prompt
after entering the root password.

See above.

Post by Anton Shterenlikht
I have top(1) running interactively in another window.
The sh process is in "getblk" state, and ignores kill -9.
But there's no ldd process.
# shutdown -r now
Shutdown NOW!
shutdown: [pid 8019]
#
and nothing happens after that
So I have to do a cold reset via MP.
On ia64 r204322, this script causes no problems.
Please advise

The above edited script should help to limit disk usage and too many open
processes that causes your machine to bog down like that. This script does
have its limitations and there is one bug in it... Ill let you figure out
how to get rid of that bug but it really does not effect the intended
output so I left it alone and sent error output to fd/2.
The limitations you'll find is how many files that ldd(1) or file(1) can
handle at one time. But if you specify specific paths like already in
SEARCHPATH then you will most likely never see this unless the files in
/*bin grow to be over max number of files that file(1) or ldd(1) can
handle at one time. Shortly said... use direct paths or short globs like
above.

Post by Anton Shterenlikht
many thanks
anton

A final note you might want to just install sysutils/libchk and run that.
Standard Disclaimer: NONE OF THIS CONTAINED HEREIN "THIS MESSAGE" EXCUSES
ANY OF THE UNEXPLAINED DISK LOCKING THAT IS GOING ON AND THE INFORMATION
FOR WHICH IT MAY CONTAIN BECOMING UNAVAILABLE AT ANY POINT IN TIME DURING
THE ORIGINAL RUN OF THE FIRST SCRIPT OR THE SECOND SCRIPT THAT WAS POSTED
EITHER AS A ATTACHMENT OR IN-LINE.
;) JK!
Good Luck.

many thanks, this is very helpful

I don't seem to have this lockup anymore.
Don't know what was happening. I've run
it now several times on 3 different ia64
current (different revisions) boxes, with
disks of different speed, and can't reproduce.
My script was very crude, of course.
I'll try sysutils/libchk

thanks again
anton

[Garrett Cooper]

Post by Anton Shterenlikht

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
       echo $file
       ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
      echo $file
      ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
         for ifelf in `find $spath -type f`; do
                 ldd `file $ifelf | grep dynamically | cut -f1 -d:`
         done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

Post by Anton Shterenlikht
On the serial console I cannot get the prompt
after entering the root password.

See above.

Post by Anton Shterenlikht
I have top(1) running interactively in another window.
The sh process is in "getblk" state, and ignores kill -9.
But there's no ldd process.
# shutdown -r now
Shutdown NOW!
shutdown: [pid 8019]
#
and nothing happens after that
So I have to do a cold reset via MP.
On ia64 r204322, this script causes no problems.
Please advise

The above edited script should help to limit disk usage and too many open
processes that causes your machine to bog down like that. This script does
have its limitations and there is one bug in it... Ill let you figure out
how to get rid of that bug but it really does not effect the intended
output so I left it alone and sent error output to fd/2.
The limitations you'll find is how many files that ldd(1) or file(1) can
handle at one time. But if you specify specific paths like already in
SEARCHPATH then you will most likely never see this unless the files in
/*bin grow to be over max number of files that file(1) or ldd(1) can
handle at one time. Shortly said... use direct paths or short globs like
above.

Post by Anton Shterenlikht
many thanks
anton

A final note you might want to just install sysutils/libchk and run that.
Standard Disclaimer: NONE OF THIS CONTAINED HEREIN "THIS MESSAGE" EXCUSES
ANY OF THE UNEXPLAINED DISK LOCKING THAT IS GOING ON AND THE INFORMATION
FOR WHICH IT MAY CONTAIN BECOMING UNAVAILABLE AT ANY POINT IN TIME DURING
THE ORIGINAL RUN OF THE FIRST SCRIPT OR THE SECOND SCRIPT THAT WAS POSTED
EITHER AS A ATTACHMENT OR IN-LINE.
;) JK!
Good Luck.

many thanks, this is very helpful
I don't seem to have this lockup anymore.
Don't know what was happening. I've run
it now several times on 3 different ia64
current (different revisions) boxes, with
disks of different speed, and can't reproduce.
My script was very crude, of course.
I'll try sysutils/libchk

FWIW I've been seeing some performance issues with iir(4) and mfi(4)
backed UFS2 with softupdate filesystems on my new machine with some
other drivers loaded on my system [a PCI based em(4) card and
nvidia-driver enabled card -- which uses GIANT locking still].

Machine is Core i7 on an ASUS W6T Professional MB, 12GB RAM, with
debug symbols, ddb, kgdb, anti-reslock contention manager, (no
witness) etc.

I don't have much other than that to provide at this time, but it
might help to see if and when there's an overlap in the drivers noted
here.

Thanks,
-Garrett

[jhell]

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.

Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.

#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done

The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.

It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.

=== Script starts here ===
#!/bin/sh

SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"

trap 'exit 1' 2

check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}

check_libs 2>/dev/null
=== Script ends here ===

The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.

Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.

So with the about stated:
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in

ZFS file-system perhaps ?

Post by Anton Shterenlikht
On the serial console I cannot get the prompt
after entering the root password.

See above.

Post by Anton Shterenlikht
I have top(1) running interactively in another window.
The sh process is in "getblk" state, and ignores kill -9.
But there's no ldd process.
# shutdown -r now
Shutdown NOW!
shutdown: [pid 8019]
#
and nothing happens after that
So I have to do a cold reset via MP.
On ia64 r204322, this script causes no problems.
Please advise

The above edited script should help to limit disk usage and too many open
processes that causes your machine to bog down like that. This script does
have its limitations and there is one bug in it... Ill let you figure out
how to get rid of that bug but it really does not effect the intended
output so I left it alone and sent error output to fd/2.

The limitations you'll find is how many files that ldd(1) or file(1) can
handle at one time. But if you specify specific paths like already in
SEARCHPATH then you will most likely never see this unless the files in
/*bin grow to be over max number of files that file(1) or ldd(1) can
handle at one time. Shortly said... use direct paths or short globs like
above.

Post by Anton Shterenlikht
many thanks
anton

A final note you might want to just install sysutils/libchk and run that.

Standard Disclaimer: NONE OF THIS CONTAINED HEREIN "THIS MESSAGE" EXCUSES
ANY OF THE UNEXPLAINED DISK LOCKING THAT IS GOING ON AND THE INFORMATION
FOR WHICH IT MAY CONTAIN BECOMING UNAVAILABLE AT ANY POINT IN TIME DURING
THE ORIGINAL RUN OF THE FIRST SCRIPT OR THE SECOND SCRIPT THAT WAS POSTED
EITHER AS A ATTACHMENT OR IN-LINE.

;) JK!

Good Luck.

- --

jhell

[Anton Shterenlikht]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

I've no ZFS.

I'm seeing very similar behaviour now with csup:

( I do csup -L2 /root/ports-supfile, where

# cat /root/ports-supfile
*default host=cvsup.uk.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=. delete use-rel-suffix compress

ports-all
# )

top(1) shows:

last pid: 1160; load averages: 0.00, 0.06, 0.07 up 0+00:10:53 15:05:52
81 processes: 3 running, 61 sleeping, 17 waiting
CPU 0: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 23M Active, 19M Inact, 75M Wired, 136K Cache, 34M Buf, 5900M Free
Swap: 2780M Total, 2780M Free

PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 0 2 171 ki31 0K 64K RUN 0 20:18 198.00% idle
11 0 17 -48 - 0K 544K WAIT 0 0:01 0.00% intr
1118 1001 1 96 0 12800K 3920K CPU0 0 0:00 0.00% top
4 0 1 -8 - 0K 32K - 1 0:00 0.00% g_down
1158 0 4 -8 0 43672K 6296K biowr 0 0:00 0.00% csup


which stays in biowr state indefinitely.

I can issue kill -9 or kill -HUP from top(1),
which makes csup change state to STOP, but
nothing else happens.

As before, I can't log in from other terminals
and have to do a cold reset. I've reinstalled
on another disk, so not sure what's going on.

I think rm(1) is also extremely slow, but
maybe I'm imagining things.

many thanks
anton

[jhell]

Post by Anton Shterenlikht

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

I've no ZFS.
( I do csup -L2 /root/ports-supfile, where
# cat /root/ports-supfile
*default host=cvsup.uk.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=. delete use-rel-suffix compress
ports-all
# )
last pid: 1160; load averages: 0.00, 0.06, 0.07 up 0+00:10:53 15:05:52
81 processes: 3 running, 61 sleeping, 17 waiting
CPU 0: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 23M Active, 19M Inact, 75M Wired, 136K Cache, 34M Buf, 5900M Free
Swap: 2780M Total, 2780M Free
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 0 2 171 ki31 0K 64K RUN 0 20:18 198.00% idle
11 0 17 -48 - 0K 544K WAIT 0 0:01 0.00% intr
1118 1001 1 96 0 12800K 3920K CPU0 0 0:00 0.00% top
4 0 1 -8 - 0K 32K - 1 0:00 0.00% g_down
1158 0 4 -8 0 43672K 6296K biowr 0 0:00 0.00% csup
which stays in biowr state indefinitely.
I can issue kill -9 or kill -HUP from top(1),
which makes csup change state to STOP, but
nothing else happens.
As before, I can't log in from other terminals
and have to do a cold reset. I've reinstalled
on another disk, so not sure what's going on.
I think rm(1) is also extremely slow, but
maybe I'm imagining things.
many thanks
anton

I would post up the contents of your make.conf & your kernel config & your
dmesg somewhere so it can be evaluated.

Regards,

[Anton Shterenlikht]

Post by jhell

Post by Anton Shterenlikht

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

I've no ZFS.
( I do csup -L2 /root/ports-supfile, where
# cat /root/ports-supfile
*default host=cvsup.uk.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=. delete use-rel-suffix compress
ports-all
# )
last pid: 1160; load averages: 0.00, 0.06, 0.07 up 0+00:10:53 15:05:52
81 processes: 3 running, 61 sleeping, 17 waiting
CPU 0: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 23M Active, 19M Inact, 75M Wired, 136K Cache, 34M Buf, 5900M Free
Swap: 2780M Total, 2780M Free
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 0 2 171 ki31 0K 64K RUN 0 20:18 198.00% idle
11 0 17 -48 - 0K 544K WAIT 0 0:01 0.00% intr
1118 1001 1 96 0 12800K 3920K CPU0 0 0:00 0.00% top
4 0 1 -8 - 0K 32K - 1 0:00 0.00% g_down
1158 0 4 -8 0 43672K 6296K biowr 0 0:00 0.00% csup
which stays in biowr state indefinitely.
I can issue kill -9 or kill -HUP from top(1),
which makes csup change state to STOP, but
nothing else happens.
As before, I can't log in from other terminals
and have to do a cold reset. I've reinstalled
on another disk, so not sure what's going on.
I think rm(1) is also extremely slow, but
maybe I'm imagining things.
many thanks
anton

I would post up the contents of your make.conf & your kernel config & your
dmesg somewhere so it can be evaluated.

When I reinstalled 8.0 from a CD,
I updated source with csup, that worked.
However, after upgrading to current, I can't get
any luck with csup. The important bit is that
I don't really know what revision this is.

I've no /etc/make.conf

kernel config:
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/UZI

dmesg:
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/dmesg.boot

many thanks for your help
anton

[Anton Shterenlikht]

Post by Anton Shterenlikht

Post by jhell

Post by Anton Shterenlikht

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

I've no ZFS.
( I do csup -L2 /root/ports-supfile, where
# cat /root/ports-supfile
*default host=cvsup.uk.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=. delete use-rel-suffix compress
ports-all
# )
last pid: 1160; load averages: 0.00, 0.06, 0.07 up 0+00:10:53 15:05:52
81 processes: 3 running, 61 sleeping, 17 waiting
CPU 0: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 23M Active, 19M Inact, 75M Wired, 136K Cache, 34M Buf, 5900M Free
Swap: 2780M Total, 2780M Free
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 0 2 171 ki31 0K 64K RUN 0 20:18 198.00% idle
11 0 17 -48 - 0K 544K WAIT 0 0:01 0.00% intr
1118 1001 1 96 0 12800K 3920K CPU0 0 0:00 0.00% top
4 0 1 -8 - 0K 32K - 1 0:00 0.00% g_down
1158 0 4 -8 0 43672K 6296K biowr 0 0:00 0.00% csup
which stays in biowr state indefinitely.
I can issue kill -9 or kill -HUP from top(1),
which makes csup change state to STOP, but
nothing else happens.
As before, I can't log in from other terminals
and have to do a cold reset. I've reinstalled
on another disk, so not sure what's going on.
I think rm(1) is also extremely slow, but
maybe I'm imagining things.
many thanks
anton

I would post up the contents of your make.conf & your kernel config & your
dmesg somewhere so it can be evaluated.

When I reinstalled 8.0 from a CD,
I updated source with csup, that worked.
However, after upgrading to current, I can't get
any luck with csup. The important bit is that
I don't really know what revision this is.
I've no /etc/make.conf
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/UZI
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/dmesg.boot

Marcel, this might be of some interest.

I managed to csup /usr/src, probably because
there was not too many updates from 3 days ago.
I proceeded with updating the system, but
had a freeze again in single user at the very
beginning of 'make installworld'.

Now I've reinstalled 8.0-CURRENT-200906
snapshot and have no issues with csup,
just completed downloading the ports tree.
It seems something is wrong with csup(1),
or pehaps disk i/o, in the recent ia64 updates.

I'll try building svn from ports and update
via svn, will report the results.

many thanks
anton

[Anton Shterenlikht]

Post by Anton Shterenlikht

Post by Anton Shterenlikht

Post by jhell

Post by Anton Shterenlikht

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

I've no ZFS.
( I do csup -L2 /root/ports-supfile, where
# cat /root/ports-supfile
*default host=cvsup.uk.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=. delete use-rel-suffix compress
ports-all
# )
last pid: 1160; load averages: 0.00, 0.06, 0.07 up 0+00:10:53 15:05:52
81 processes: 3 running, 61 sleeping, 17 waiting
CPU 0: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 23M Active, 19M Inact, 75M Wired, 136K Cache, 34M Buf, 5900M Free
Swap: 2780M Total, 2780M Free
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 0 2 171 ki31 0K 64K RUN 0 20:18 198.00% idle
11 0 17 -48 - 0K 544K WAIT 0 0:01 0.00% intr
1118 1001 1 96 0 12800K 3920K CPU0 0 0:00 0.00% top
4 0 1 -8 - 0K 32K - 1 0:00 0.00% g_down
1158 0 4 -8 0 43672K 6296K biowr 0 0:00 0.00% csup
which stays in biowr state indefinitely.
I can issue kill -9 or kill -HUP from top(1),
which makes csup change state to STOP, but
nothing else happens.
As before, I can't log in from other terminals
and have to do a cold reset. I've reinstalled
on another disk, so not sure what's going on.
I think rm(1) is also extremely slow, but
maybe I'm imagining things.
many thanks
anton

I would post up the contents of your make.conf & your kernel config & your
dmesg somewhere so it can be evaluated.

When I reinstalled 8.0 from a CD,
I updated source with csup, that worked.
However, after upgrading to current, I can't get
any luck with csup. The important bit is that
I don't really know what revision this is.
I've no /etc/make.conf
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/UZI
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/dmesg.boot

Marcel, this might be of some interest.
I managed to csup /usr/src, probably because
there was not too many updates from 3 days ago.
I proceeded with updating the system, but
had a freeze again in single user at the very
beginning of 'make installworld'.
Now I've reinstalled 8.0-CURRENT-200906
snapshot and have no issues with csup,
just completed downloading the ports tree.
It seems something is wrong with csup(1),
or pehaps disk i/o, in the recent ia64 updates.
I'll try building svn from ports and update
via svn, will report the results.

An update:

1. reinstalled from 8.0-CURRENT-200906

2. installed the ports tree via csup(1)

3. installed svn(1) from ports

4. updated src with svn.
Both svn and csup worked fine here.

5. rebuilt and reinstalled kernel and world as
usual to r205403.

6. rebooted.
The kernel config file:
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/UZI

dmesg:
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/dmesg.boot

ifconfig -a:
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/ifconfig-a


7. tried to update the src again with svn and got stuck.
All I can issue is CTRL/T, which shows for svn:

mech-as221# svn co svn://svn.freebsd.org/base/head/ /usr/src/

load: 0.00 cmd: svn 888 [biord] 8008.53r 0.09u 0.30s 0% 13992k
load: 0.00 cmd: svn 888 [biord] 8009.53r 0.09u 0.30s 0% 13992k
load: 0.00 cmd: svn 888 [biord] 8015.07r 0.09u 0.30s 0% 13992k

in another ssh session I was running gstat(8) which showed
zero activity in the disk.

and in yet another ssh session I tried to launch top:

mech-as221# top
load: 0.00 cmd: csh 915 [ufs] 6146.33r 0.00u 0.00s 0% 5008k
load: 0.00 cmd: csh 915 [ufs] 6147.15r 0.00u 0.00s 0% 5008k

and on the serial console:

load: 0.00 cmd: getty 828 [ufs] 8129.90r 0.00u 0.00s 0% 2560k
load: 0.00 cmd: getty 828 [ufs] 8130.70r 0.00u 0.00s 0% 2560k

but the shell prompt never appears.
I've waited maybe 2-3 hours.


Can't do much else, but a cold reboot.

many thanks
anton

[Anton Shterenlikht]

Marcel

Post by Anton Shterenlikht
1. reinstalled from 8.0-CURRENT-200906
2. installed the ports tree via csup(1)
3. installed svn(1) from ports
4. updated src with svn.
Both svn and csup worked fine here.
5. rebuilt and reinstalled kernel and world as
usual to r205403.
6. rebooted.
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/UZI
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/dmesg.boot
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/ifconfig-a
7. tried to update the src again with svn and got stuck.
mech-as221# svn co svn://svn.freebsd.org/base/head/ /usr/src/
load: 0.00 cmd: svn 888 [biord] 8008.53r 0.09u 0.30s 0% 13992k
load: 0.00 cmd: svn 888 [biord] 8009.53r 0.09u 0.30s 0% 13992k
load: 0.00 cmd: svn 888 [biord] 8015.07r 0.09u 0.30s 0% 13992k
in another ssh session I was running gstat(8) which showed
zero activity in the disk.
mech-as221# top
load: 0.00 cmd: csh 915 [ufs] 6146.33r 0.00u 0.00s 0% 5008k
load: 0.00 cmd: csh 915 [ufs] 6147.15r 0.00u 0.00s 0% 5008k
load: 0.00 cmd: getty 828 [ufs] 8129.90r 0.00u 0.00s 0% 2560k
load: 0.00 cmd: getty 828 [ufs] 8130.70r 0.00u 0.00s 0% 2560k
but the shell prompt never appears.
I've waited maybe 2-3 hours.

On reboot I did
# cd /usr/obj
# chflags -R noschg *
# rm -rf *

I monitor disk activity with iostat(8) and gstat(8),
and system activity with top(1) from 3 separate ssh sessions.
For about 5-10 sec iostat(8) and gstat(8) show significant
disk activity. After that both iostat and gstat show
zero disk activity. top(1) shows:

[skip]
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 0 2 171 ki31 0K 64K RUN 0 28:40 198.00% idle
11 0 17 -48 - 0K 544K WAIT 0 0:02 0.00% intr
893 0 1 96 0 12800K 4008K CPU0 0 0:00 0.00% top
918 0 1 -4 0 11592K 2424K getblk 0 0:00 0.00% rm
[skip]

rm never exits (well.. within 20 minutes).

kill -9 918 (issued from top(1)) makes no effect.
No new ssh logins are possible, and the existing
ssh sessions and the serial line don't show the
shell prompt.

It seems the problems I've had in the last
several days with ldd/csup/svn/rm have the
same root cause.

I'm just not sure if it's someting simple
that I've messed up, or something went wrong
in current..

many thanks for your help
anton

[Marcel Moolenaar]

Post by Anton Shterenlikht
It seems the problems I've had in the last
several days with ldd/csup/svn/rm have the
same root cause.

I'm aware of the issue. Give me a few days to fix it. I have a
fix under test, but it exposes other problems...

[jhell]

Post by Anton Shterenlikht

Post by Anton Shterenlikht

Post by Anton Shterenlikht

Post by jhell

Post by Anton Shterenlikht

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Anton Shterenlikht
Just updated to ia64 r205248
If my problem is due to my mis-configuration,
I apologise in advance.
I run this shell script after each upgrade
and 'make delete-old-libs' to check
<start script>
#!/bin/sh
for file in `find /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/libexec /usr/local -name "*"`
do
echo $file
ldd $file >> /root/ldd_results 2> /dev/zero
done
<end script>

This will probably do closer to what you actually would want to look for.
Writing to /dev/zero ... I don't know never tried it since /dev/null is
usually the standard place to throw trash.
#!/bin/sh
for file in `find /*bin /usr/*bin /usr/lib* /usr/local/*bin -type f` do
echo $file
ldd $file >>/root/ldd_results 2>/dev/null
done
The problem with your script is that it finds most files that it can not
or is not useful to run ldd on and leaves you junk in return.
It might be more useful if you searched for dynamically linked ELF
binaries to run ldd against like the following.
=== Script starts here ===
#!/bin/sh
SEARCHPATH="/*bin /usr/*bin /usr/lib* /usr/local/*bin"
trap 'exit 1' 2
check_libs() {
for spath in $SEARCHPATH; do
for ifelf in `find $spath -type f`; do
ldd `file $ifelf | grep dynamically | cut -f1 -d:`
done
done
}
check_libs 2>/dev/null
=== Script ends here ===
The above will find all type ELF * that are dynamically linked within the
SEARCHPATH variable and run ldd on them and print the results to stdout.
Obviously since you are going to have thousands of files being questioned,
stdout is not going to be useful.
save the script to: checklibs.sh
run with: "sh checklibs.sh >/root/checklibs_output"
or: "script /root/checklibs_output checklibs.sh"

Post by Anton Shterenlikht
After the upgrade to r205248, the script
freezes at seemingly random points.

Unneeded disk usage & execution.

Post by Anton Shterenlikht
I can still ssh to the machine (using keys), i.e.
I see the welcome message, but cannot get to the console prompt.

Of course... to many open files or processes in wait. SSH already has the
information it needs loaded into memory, that's why you can get sort-of-in
ZFS file-system perhaps ?

I've no ZFS.
( I do csup -L2 /root/ports-supfile, where
# cat /root/ports-supfile
*default host=cvsup.uk.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=. delete use-rel-suffix compress
ports-all
# )
last pid: 1160; load averages: 0.00, 0.06, 0.07 up 0+00:10:53 15:05:52
81 processes: 3 running, 61 sleeping, 17 waiting
CPU 0: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 23M Active, 19M Inact, 75M Wired, 136K Cache, 34M Buf, 5900M Free
Swap: 2780M Total, 2780M Free
PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 0 2 171 ki31 0K 64K RUN 0 20:18 198.00% idle
11 0 17 -48 - 0K 544K WAIT 0 0:01 0.00% intr
1118 1001 1 96 0 12800K 3920K CPU0 0 0:00 0.00% top
4 0 1 -8 - 0K 32K - 1 0:00 0.00% g_down
1158 0 4 -8 0 43672K 6296K biowr 0 0:00 0.00% csup
which stays in biowr state indefinitely.
I can issue kill -9 or kill -HUP from top(1),
which makes csup change state to STOP, but
nothing else happens.
As before, I can't log in from other terminals
and have to do a cold reset. I've reinstalled
on another disk, so not sure what's going on.
I think rm(1) is also extremely slow, but
maybe I'm imagining things.
many thanks
anton

I would post up the contents of your make.conf & your kernel config & your
dmesg somewhere so it can be evaluated.

When I reinstalled 8.0 from a CD,
I updated source with csup, that worked.
However, after upgrading to current, I can't get
any luck with csup. The important bit is that
I don't really know what revision this is.
I've no /etc/make.conf
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/UZI
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/dmesg.boot

Marcel, this might be of some interest.
I managed to csup /usr/src, probably because
there was not too many updates from 3 days ago.
I proceeded with updating the system, but
had a freeze again in single user at the very
beginning of 'make installworld'.
Now I've reinstalled 8.0-CURRENT-200906
snapshot and have no issues with csup,
just completed downloading the ports tree.
It seems something is wrong with csup(1),
or pehaps disk i/o, in the recent ia64 updates.
I'll try building svn from ports and update
via svn, will report the results.

1. reinstalled from 8.0-CURRENT-200906
2. installed the ports tree via csup(1)
3. installed svn(1) from ports
4. updated src with svn.
Both svn and csup worked fine here.
5. rebuilt and reinstalled kernel and world as
usual to r205403.
6. rebooted.
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/UZI
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/dmesg.boot
http://seis.bris.ac.uk/~mexas/freebsd/ia64/rx2600/uzi/ifconfig-a
7. tried to update the src again with svn and got stuck.

:
: mech-as221# svn co svn://svn.freebsd.org/base/head/ /usr/src/
:

Not that this has anything to do with your problem but might turn into a
problem if it is not your intent. But the above command will update your
src to 9.0-CURRENT

If it is your intention to stay on 8.0-* you might want to use one of the
following:

For checkout: (8.0-STABLE)
svn co svn://svn.freebsd.org/base/stable/8/ /usr/src/

For checkout: (8.0-RELEASE-pN)
svn co svn://svn.freebsd.org/base/releng/8.0/ /usr/src/

For updating: (after initial checkout)
cd /usr/src ;svn update [-r%REVISION%]


Then again maybe this is the problem... If you updated to current and did
not rebuild your ports tree then that might be a factor of the problem.

Regards,

[Marcel Moolenaar]

Not that this has anything to do with your problem but might turn into a problem if it is not your intent. But the above command will update your src to 9.0-CURRENT

That is his intend.

Anton: I fixed a bunch of things yesterday. You should be able to
upgrade to HEAD. I have one more fix pending that's needed for
preemption, but you should not hold off an upgrade for that...

FYI,