UKC

Starting to run out of ideas on this so thought someone might be able to help.
I have a root ssh session open to a Centos 6 server and have changed the password as follows by running passwd.
[root@s1 etc]# passwd
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

All ok to here, but when I try to open another ssh session to test the updated password 'Permission denied, please try again.' is returned. There is normally no problem with multiple ssh sessions running.

I'm a bit worried, as should my internet drop I'll be locked out!

Thanks,

In reply to rockxk:

Root blocked from ssh? It should be!

Check /etc/ssh/sshd_config

PermitRootLogin no

Failing that, check /var/log/secure... Is it actually rejecting the password?

Failing that, Pam might be blocking it? Unlikely really but worth a look:

pam_tally2 --user root --reset

In reply to rockxk:

Btw, I wasn't joking about direct root login, especially as you suggest you have an internet facing server. Once you have it sorted, block it from remote access and use a proper method. A decent sudo profile preferably, but failing that a normal user that can su to root would be an improvement.

In reply to snoop6060:

>Root blocked from ssh? It should be!
yeah, sure :0 there is an open ssh session as root and to confirm: PermitRootLogin yes

Yup, the password is being rejected... (have tried retyping it obvs)
Dec 10 17:10:55 localhost sshd[24367]: Failed password for root from x.x.x.x port 40932 ssh2

Thanks,

In reply to snoop6060:


Agreed, it's certainly on the to-do list

In reply to rockxk:

What's pam_tally2 saying?

The only other thing I can think of is some sort of char encoding issue. Or you are mistyping the password. Make it simple as a test?

In reply to rockxk:

pam_tally2 --user root --reset
Login Failures Latest failure From
root 0

The number of failures on root isn''t incrementing...

In reply to snoop6060:

Yeah I'm trying a very simple password now.

Ironically given you're other advice, I've created a second root user with the below and this doesn't work either.

useradd -ou 0 -g 0 root1

So /etc/shadow is being updated with passwd but could there be a hash type mismatch or something?

In reply to rockxk:

if you login as another user (not uid 0), change passwd and then try to login again with the same uid do you get the same issue?

In reply to rockxk:

Can you SSH in as another user and then SU to root.

If so you have another methods of accessing if there is a problem.
You can also check if the root password has been updated.

In reply to herbe_rouge:


Creating a user with useradd and passwd gives the same problem with ssh yeah.

In reply to rockxk:

so if you log out of both ssh sessions and then login back in with the non uid 0 account, which passwd takes effect? and if the updated, can you then instantiate a second ssh with that uid?

In reply to herbe_rouge:

I'm trying to get a second ssh session and don't want to kill the existing session - that's the only way to make any changes atm.

New ssh logons with the existing root user, new root user (root1), or a new standard user all fail on the password check. Logging on with root on the old password fails too.

In reply to rockxk:

yep understand that but you can logout of the non-root account i.e. log out of the non-root account session and logged back in, then, which passwd is active and does it still throw up the same error if you try a parallel ssh session? Sorry, this is a cumbersome way to troubleshoot....

In reply to herbe_rouge:

Thanks, appreciate the advice.

I'm not able to login to it...

It was created simply with:
adduser tester11
passwd ...

to give:
tester11:x:500:500::/home/tester11:/bin/bash
tester11:$6$BCGwBjbY$NkWWHONssFwxepK5DxWyMkPPSdY2JZvc8b.G1qNL8TyB3Xw4TTy7n5Yc7Pntf6Iv2DS8LZErE/6mcIHFsOqiL.:16779:0:99999:7:::

In reply to rockxk:

Ok, so the issue is not related to logging in as root (which you really should not do in future, when this is fixed edit /etc/ssh/sshd_config and set PermitRootLogin to no). Thus the issue is passwd under ssh. Since adduser works fine under ssh I suspect the issue maybe centos nuanced - google is your friend(ish). I haven't used an RPM distro for years and have to feed the kids but will try to think about this later, sorry, that's not much help......

In reply to rockxk:

I would first echo others comments about root and ssh. I would go further and say any internet facing machine should be key login only, no password login enabled. Have a look in /var/log/auth.log (or maybe different for your distro) to count how times a minute someone from a China IP address tries to login as root.

I learnt this the hard way. I had root login disabled on a machine, reasonably secure password but by brute force (even with fail2ban) someone got in and was bitcoin mining on the machine.

For your problem, im guessing connecting a screen and bypassing ssh isn't an option? Are you sure you havent accidentally changed something in sshd? Ive seen misleading responses from failed ssh logins. If you try with the wrong port or username, do you get a different response?

In reply to Matt_b:


Local support have connected a screen but there is no video output unfortunately (this has worked in the past but rebooting the server isn't really an option).

The wrong username still reports 'Permission denied, please try again.' Other (unfirewalled) ports don't return anything.

In reply to rockxk:

No, I've not changed anything in sshd (presumably the existing ssh session would have died if changes were to take effect anyway)

I'd actually be keen to create a backdoor in case the session dies as having to rebuild the box would be problematic to say the least.

Thanks for all the help

In reply to rockxk:

Has anything changed in nsswitch.conf? Are you definitely using passwd files, and not some other service? You should have a line like passwd: files in /etc/nsswitch.conf

In reply to rockxk:

>Yup, the password is being rejected... (have tried retyping it obvs)
>Dec 10 17:10:55 localhost sshd[24367]: Failed password for root from x.x.x.x port 40932 ssh2

I've just tried logging into one of my boxes as root, when I know that sshd disallows it, and get the same message as you.
# grep -i root /etc/ssh/sshd_config
PermitRootLogin no

/var/log/secure
unix_chkpwd[4122]: password check failed for user (root)
...
sshd[4113]: Failed password for root from x.x.x.x port 50177 ssh2

When I try to log in as a user with the wrong passwd I get
Failed password for ...

When I try logging in as a user with the correct password, but /bin/false as a shell it, accepts the password, then kicks me out.
Accepted password for ...
...
pam_unix(sshd:session): session closed for user ...

When you say:
>there is an open ssh session as root and to confirm: PermitRootLogin yes

How are you seeing that? Is that by typing: who or w ?
Also if you just changed the PermitRootLogin line, you need to restart sshd, with: service sshd restart.
Restarting ssh wont drop your session.

You might want to check selinux config
cat /etc/selinux/config
You're looking for the SELINUX= bit. If it says enforcing, then you may need to disable selinux, which you can do as root with:
echo 0 >/selinux/enforce

Depending on if you can gain physical access to the box, you can always reset the root passwd at boot time. Might be an issue if it's on the other side of the world though ...

In reply to Nexonen:


Yup, as below:

passwd: files
shadow: files
group: files

In reply to rockxk:

can you get physical ownership and edit the grub line?

In reply to rockxk:
Can you su to another user (a non-root user), then su to root, you will be prompted for the root password, see if that works. Use exit (twice) to get back to your original login.

# su tester11
$ su root
(Enter root password ...)
# exit
$ exit

Also, double check you haven't got MaxSessions in /etc/sshd_config set to 1 or anything silly?

In reply to rockxk:

>I'd actually be keen to create a backdoor in case the session dies as having to rebuild the box would be problematic to say the least.

You can bind a shell to a port, even a root shell or if you have another unix box to hand you can get your broken box to send a shell back to your good box. You need to sort the firewall so the good box doesn't block it.
google for "netcat shell on port" or "netcat reverse shell"

If you don't have netcat installed then:
yum install nc

Before you do that though, I'd have a check to see if sshd is running and I'd work at trying to ssh to a user account.
In your /etc/ssh/sshd_config file have a look for
PasswordAuthentication yes
UsePAM yes

In reply to rockxk:

>Local support have connected a screen but there is no video output unfortunately
Have they plugged in a keyboard and hit return a few times? Linux will blank the console, if there is no input. Remote hands chaps are not usually great at debugging stuff and sometimes have to be talked through even simple actions.

In reply to dread-i:

So I think that sounds as though root ssh connections are allowd, but the password match is failing (or am I misinterpreting that?).

Yes, I've restarted sshd now and still have the ssh session which is good.

'who' is returning root, and SELinux is disabled.

Local support would restart the box and probably reset the password, but that wouldn't guarantee a connection except by KVM terminal.

In reply to Nexonen:

I couldn't su to tester (?) but sudo worked. The password on root then failed.

[root@s1 ~]# su tester11
could not open session
[root@s1 ~]# sudo -i -u tester11
[tester11@s1 ~]$ su root
Password:
could not open session
[tester11@s1 ~]$ exit
logout
[root@s1 ~]#

In reply to dread-i:

>You can bind a shell to a port, even a root shell
I have a local linux box so will look into that. Thanks.

>Before you do that though, I'd have a check to see if sshd is running and I'd work at trying to ssh to a user account.
In your /etc/ssh/sshd_config file have a look for
PasswordAuthentication yes
UsePAM yes

Both these are there and sshd is running.

In reply to rockxk:

You can bypass the root passwd.
If you edit /etc/passwd
change
root:x:0:0:root:/root:/bin/bash
to
root::0:0:root:/root:/bin/bash

You will be able to ssh or su with no root passwd.
If that is the case then it is something in the way your passwd hashed or an error in /etc/shadow

You probably don't want to leave it that way for long...

In reply to dread-i:

Yup, this doesn't work (changed back now). I've also just tried pasting in a known working /etc/shadow line.

In reply to rockxk:

Still no joy. Thanks for the help all

In reply to rockxk:

have you checked top? If no joy with that, can you edit the grub on reboot?

In reply to herbe_rouge:

Checked top?

In reply to rockxk:

Right, well I think it's fixed. Cheers for the help folks - much appreciated

In reply to rockxk:


What was the solution?

In reply to krikoman:

Had to restore some files that had gone walkabouts.

In reply to rockxk:

Any hints to which files you had to restore?

In reply to rockxk:

Glad to hear it. It's good practise to report the solution since others may find it useful in future so it would be good to hear details of how you fixed it.

In reply to left_edge:


Yes, fair. Have just checked back in as have been a shade busy.

The entire /etc/secure folder was missing, and some scripts that were running also. Luckily there were fairly recent backups available, but it looks like the box was hacked :-/

Lesson learned: push security up the agenda.

Unplug your box, block remote root logins and run chkrootkit and rkhunter now...

In reply to rockxk:

Glad you've fixed it, however, if you've been hacked, the only way to be sure that the box is secure is to wipe the whole thing and restore from backups. Harsh, but it's the only way to know that there isn't some files on your box that will allow access again.

As others have said, do not allow root access via SSH. Also, I'd say don't use passwords either, SSH keys are the way to go. Then turn PasswordAuthentication off.

Cent OS 6 isn't ancient, but moving to Cent OS 7 would be a good idea.

Hacked servers means more spam and malicious traffic for the rest of us, so get it sorted ASAP.

In reply to duchessofmalfi:

If you can boot from a CD and do your root kit checking, even better.
You can't trust your box or anything on it.

If you're lucky, they may have trojan'd some binaries and you can compare the MD5 to know good versions. If they have installed a kernel based root kit, all the files will be ok or will return the correct MD5, but you'll still be owned. Also, you can't trust your backups of any binary files and any data should also be treated with suspicion as well. Any machines that connect to it or are on the same networks, treat as compromised as well, along with any routers and networked printers. Fun, fun, fun.

In reply to rockxk:
If your machine has been compromised, I would remove it from the network immediately. If possible keep the hard drive (for further investigations), get a new hard drive and do a fresh install on that.

How many other machines connect to this? They could easily be messed up too.

Treat your backups as suspicious at best.

Treat any system logs as possibly manipulated.

I mean this in the nicest possible way, try to get someone who knows a bit more about Linux to have a look at it offline. They may be able to run some forensics on the files you hope to recover.

Do some reading on passwordless ssh and keys. Disable ssh by password, be specific in usernames which CAN ssh into the machine (NOT ROOT!)

The above may all sound like a hassle, but it is the only way to prevent this happening again.