shell 中的ftp-ftp批处理

最近在用ftp从ncbi倒腾东西,东西很多,自动化好一些。

Sometimes I want to FTP a file from one machine to another. Usually, I can do the transfer interactively, but every so often, I would like to have a shell script do the file transfer. This task has eluded me in the past, but I finally figured it out. I've not seen this particular trick documented in the past, so I submit it for your approval.

The Problem

The problem I always encountered in scripting ftp transfers involved getting a password to the ftp server. Typical ftp client programs under Unix, Linux, Solaris and NetBSD all read the ftp password from /dev/tty.

Example (non-working) script

#!/bin/sh 
HOST
='ftp.users.qwest.net' 
USER
='yourid' 
PASSWD
='yourpw' 
FILE
='file.txt'  
ftp $HOST 
<<END_SCRIPT 
user $USER $PASSWD 
put $FILE 
quit 
END_SCRIPT 
exit 
0 
  

The above script will just hang if run in the foreground (in an xterm), or if run in the background (from a cron job), it will fail to perform the work of transferring file.txt.

/dev/tty names a strange, magic device. Each process (more strictly each process group) has a different /dev/tty, and you can not naively make ftp clients read the password from some non-magic, yet convenient source, like a "here document". When run in an xterm, the script above appears to hang because it reads the password from /dev/tty. The xterm constitutes the script's /dev/tty, so the script waits for keyboard input.

Example Working Script

#!/bin/sh 
HOST
='ftp.users.qwest.net' 
USER
='yourid' 
PASSWD
='yourpw' 
FILE
='file.txt'  
ftp 
-n $HOST <<END_SCRIPT 
quote USER $USER 
quote PASS $PASSWD 
put $FILE 
quit 
END_SCRIPT 
exit 
0 
The Tricks Getting the password to the ftp server without having the ftp client program read the password from /dev/tty requires two tricks: 
  1. Using the -n option on the ftp client program to prevent the ftp client from trying to log in immediately. That way, the ftp client does not ask for a user ID and password. No use of /dev/tty.
  2. Use the ftp client program command quote to send user ID and password to the ftp server.

Further Refinements

The above sh script will spew lots of ftp client output to standard output. Even if everything works perfectly, the user running the above script will see lots of incomprehensible text scrolling by quite rapidly. One refinement would send output to different places:
ftp -n $HOST > /tmp/ftp.worked 2> /tmp/ftp.failed <<END_SCRIPT 
One could further refine error handling by acting on the ftp client program's exit status:
ftp -n $HOST > /tmp/ftp.worked 2> /tmp/ftp.failed <<END_SCRIPT 
blah blah 
END_SCRIPT 
EXITSTATUS
=$?  
if [ $EXITSTATUS != "0" ] 
then     
# handle the error fi  
 

Except that the above doesn't always work - most FTP clients always exit with a status of 0. This leads to ugly "false negatives": the file transfer fails, but the script doesn't detect the problem.

One way to verify that a file transfer took place - transfer it back:

 
#!/bin/sh  ftp -<< END_SCRIPT 
open $
1 user $2 $3 
put $
4 
get $4 
retrieval.$$ 
bye 
END_SCRIPT  
if [ -f retrieval.$$ ] 
then     
echo 
"FTP of $4 to $1 worked"     
rm 
-f retrieval.$$ 
else     
echo 
"FTP of $4 did not work" 
fi

Regular FTPs there and back of large files can consume a lot of time.

Control of ftp by a shell script

One obvious improvement would have the ftp client program controlled by the shell script. I don't think that would comprise an impossible task, but I also don't think that it would have much value. Scripting ftp transfer using expect might cause you less pain.


Alternative #1

I saw a second way of doing this in a usenet article:

#!/bin/sh 
USER
=userid 
PASSWD
=userpw 
ftp 
-n f2dev <<SCRIPT 
user $USER $PASSWD 
binary 
get some.file 
quit 
SCRIPT 
 

It still uses the "-n" trick, but it sends user ID and password in the same "user" command.


Alternative #2

Use a .netrc file

Linux, Unix and BSD users have the alternative of using a .netrc file. The ftp man page documents the format of .netrc. To accomplish the task of using ftp in a shell script you would have to fill out a .netrc file something like this:

 machine something.else.com login myid password mypassword  

ftp demands that .netrc not have group or world read or write permissions:

 $ ls -l .netrc -rw-------    1 bediger  users          51 Dec 16 13:30 .netrc  

Using a .netrc file has a few problems that may or may not prevent you from using it.

  • A shell scripkt that does FTP using .netrc is no longer self-contained. You have to keep track of two files, which means that bugs can be less than obvious.
  • ftp reads it's user ID's .netrc. If you develop your script under a given user ID, then put it in production under a second user ID, you have to coordinate .netrc file contents between those two user IDs.

Alternative #3

Apparently, the Ckermit program from Columbia University understands FTP. You could use Ckermit to script FTP transfers. This looks to have advantages and disadvantages. On the "pro" side, it appears that Ckermit can exit on various errors, like unknown user IDs, or bad passwords. On the "con" side, you have to have Ckermit. I don't recall that it had a too onerous install, but it doesn't come with many Linux distros these days, and it probably doesn't come with any vendor Unix.


需要注意的一点是,如果在脚本中使用mget命令,需要在ftp时加上-i 选项,这样可以关闭 interactive prompt,不然悲剧.

阻止连接超时: 可以修改服务器端ssh配置文件;修改客户端ssh配置文件(/etc/ssh/ssh_config serveraliveinterval 和 clientalivecountmax);直接ssh -o ServerAliveInterval=x user@ip
这是找到的一个参考,转帖注明出处

posted on 2012-01-12 11:10 ewre 阅读(355) 评论(0)  编辑 收藏 引用 所属分类: linux shell


只有注册用户登录后才能发表评论。
网站导航: 博客园   IT新闻   BlogJava   知识库   博问   管理


导航

<2024年11月>
272829303112
3456789
10111213141516
17181920212223
24252627282930
1234567

留言簿(2)

文章分类

文章档案

最新评论

阅读排行榜