logo头像
Snippet 博客主题

Linux基础-wget与curl命令

本文于805天之前发表,文中内容可能已经过时

Linux系统下的下载工具

wget 是一个从网络上自动下载文件的自由工具,支持通过 HTTP,HTTPS,FTP 三个最常见的 TCP/IP协议 下载,并可以使用 HTTP 代理.”wget” 这个名称来源于 “World Wide Web” 与 “get” 的结合.
所谓自动下载,是指 wget 可以在用户退出系统的之后在继续后台执行,直到下载任务完成.

命令选项

  • -a<日志文件>:在指定的日志文件中记录资料的执行过程;
  • -A<后缀名>:指定要下载文件的后缀名,多个后缀名之间使用逗号进行分隔;
  • -b:进行后台的方式运行wget;
  • -B<连接地址>:设置参考的连接地址的基地地址;
  • -c:继续执行上次终端的任务,断点续传;
  • -C<标志>:设置服务器数据块功能标志on为激活,off为关闭,默认值为on;
  • -d:调试模式运行指令;
  • -D<域名列表>:设置顺着的域名列表,域名之间用”,”分隔;
  • -e<指令>:作为文件”.wgetrc”中的一部分执行指定的指令;
  • -h:显示指令帮助信息;
  • -i<文件>:从指定文件获取要下载的URL地址;
  • -l<目录列表>:设置顺着的目录列表,多个目录用”,”分隔;
  • -L:仅顺着关联的连接;
  • -O: 将要下载的文件进行重命名
  • -r:递归下载方式;
  • -nc:文件存在时,下载文件不覆盖原有文件;
  • -nv:下载时只显示更新和出错信息,不显示指令的详细执行过程;
  • -q:不显示指令执行过程;
  • -nh:不查询主机名称;
  • -v:显示详细执行过程;
  • -V:显示版本信息;
  • –passive-ftp:使用被动模式PASV连接FTP服务器;
  • –follow-ftp:从HTML文件中下载FTP连接文件.

    参考示例

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    1.下载网络资源
    [root@litong ~]# wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz #使用wget下载Python安装包
    --2018-05-22 16:57:49-- https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz
    Resolving www.python.org (www.python.org)... 151.101.8.223, 2a04:4e42:2::223
    Connecting to www.python.org (www.python.org)|151.101.8.223|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 22994617 (22M) [application/octet-stream]
    Saving to: ‘Python-3.6.5.tgz’

    100%[=============================================================================================================&gt;] 22,994,617 27.6MB/s in 0.8s

    2018-05-22 16:57:50 (27.6 MB/s) - ‘Python-3.6.5.tgz’ saved [22994617/22994617]

    [root@litong ~]# ll | grep Python
    -rw-r--r-- 1 root root 22994617 Mar 28 18:24 Python-3.6.5.tgz

2.断点续传

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@litong ~]# wget -c http://d1.music.126.net/dmusic/netease-cloud-music_1.1.0_amd64_ubuntu.deb  #下载Linux版本的网易云音乐
--2018-05-22 17:04:04-- http://d1.music.126.net/dmusic/netease-cloud-music_1.1.0_amd64_ubuntu.deb
Resolving d1.music.126.net (d1.music.126.net)... 157.185.168.24
Connecting to d1.music.126.net (d1.music.126.net)|157.185.168.24|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 68515076 (65M) [application/octet-stream]
Saving to: ‘netease-cloud-music_1.1.0_amd64_ubuntu.deb’
46% [====================================================================&gt; ] 31,560,638 15.2MB/s ^C #按下Ctrl+C终止下载
[root@litong ~]# wget -c http://d1.music.126.net/dmusic/netease-cloud-music_1.1.0_amd64_ubuntu.deb #端点续传,继续下载,不会重新下载
--2018-05-22 17:04:09-- http://d1.music.126.net/dmusic/netease-cloud-music_1.1.0_amd64_ubuntu.deb
Resolving d1.music.126.net (d1.music.126.net)... 157.185.168.24
Connecting to d1.music.126.net (d1.music.126.net)|157.185.168.24|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 68515076 (65M), 33805218 (32M) remaining [application/octet-stream]
Saving to: ‘netease-cloud-music_1.1.0_amd64_ubuntu.deb’
100%[+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++==========================================================================&gt;] 68,515,076 14.7MB/s in 2.2s
Last-modified header invalid -- time-stamp ignored.
2018-05-22 17:04:12 (14.7 MB/s) - ‘netease-cloud-music_1.1.0_amd64_ubuntu.deb’ saved [68515076/68515076]

3.后台下载

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@litong ~]# wget -b https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz && tail -50f wget-log  #后台下载并查看下载日志.下载并不阻塞Shell交互
Continuing in background, pid 301.
Output will be written to ‘wget-log’.
--2018-05-22 17:14:55-- https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz
Resolving www.python.org (www.python.org)... 151.101.8.223, 2a04:4e42:2::223
Connecting to www.python.org (www.python.org)|151.101.8.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22994617 (22M) [application/octet-stream]
Saving to: ‘Python-3.6.5.tgz’
0K .......... .......... .......... .......... .......... 0% 17.2M 1s
50K .......... .......... .......... .......... .......... 0% 23.7M 1s
100K .......... .......... .......... .......... .......... 0% 33.3M 1s
150K .......... .......... .......... .......... .......... 0% 22.4M 1s
200K .......... .......... .......... .......... .......... 1% 34.5M 1s
250K .......... .......... .......... .......... .......... 1% 32.9M 1s
.......... .......... .......... .......... ..........
22250K .......... .......... .......... .......... .......... 99% 13.2M 0s
22300K .......... .......... .......... .......... .......... 99% 10.7M 0s
22350K .......... .......... .......... .......... .......... 99% 12.4M 0s
22400K .......... .......... .......... .......... .......... 99% 14.3M 0s
22450K ..... 100% 129M=0.8s
2018-05-22 17:14:56 (27.7 MB/s) - ‘Python-3.6.5.tgz’ saved [22994617/22994617]

测试接口常用的curl

curl是利用URL语法在命令行方式下工作的开源文件传输工具.它支持文件的上传和下载,所以是综合传输工具.
curl支持包括HTTP,HTTPS,ftp等众多协议,还支持POST,cookies认证,从指定偏移处下载部分文件,用户代理字符串,限速,文件大小,进度条等特征.做网页处理流程和数据检索自动化,curl可以祝一臂之力.

参考示例

1.简单请求

1
2
3
4
[root@litong ~]# curl https://www.baidu.com  #简单GET请求请求百度
&lt;!DOCTYPE html&gt;
&lt;!--STATUS OK--&gt;&lt;html&gt; &lt;head&gt;&lt;meta http-equiv=content-type content=text/html;charset=utf-8&gt;&lt;meta http-equiv=X-UA-Compatible content=IE=Edge&gt;&lt;meta content=always name=referrer&gt;&lt;link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css&gt;&lt;title&gt;百度一下,你就知道&lt;/title&gt;&lt;/head&gt; &lt;body link=#0000cc&gt; &lt;div id=wrapper&gt; &lt;div id=head&gt; &lt;div class=head_wrapper&gt; &lt;div class=s_form&gt; &lt;div class=s_form_wrapper&gt; &lt;div id=lg&gt; &lt;img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129&gt; &lt;/div&gt; &lt;form id=form name=f action=//www.baidu.com/s class=fm&gt; &lt;input type=hidden name=bdorz_come value=1&gt; &lt;input type=hidden name=ie value=utf-8&gt; &lt;input type=hidden name=f value=8&gt; &lt;input type=hidden name=rsv_bp value=1&gt; &lt;input type=hidden name=rsv_idx value=1&gt; &lt;input type=hidden name=tn value=baidu&gt;&lt;span class="bg s_ipt_wr"&gt;&lt;input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus&gt;&lt;/span&gt;&lt;span class="bg s_btn_wr"&gt;&lt;input type=submit id=su value=百度一下 class="bg s_btn" autofocus&gt;&lt;/span&gt; &lt;/form&gt; &lt;/div&gt; &lt;/div&gt; &lt;div id=u1&gt; &lt;a href=http://news.baidu.com name=tj_trnews class=mnav&gt;新闻&lt;/a&gt; &lt;a href=https://www.hao123.com name=tj_trhao123 class=mnav&gt;hao123&lt;/a&gt; &lt;a href=http://map.baidu.com name=tj_trmap class=mnav&gt;地图&lt;/a&gt; &lt;a href=http://v.baidu.com name=tj_trvideo class=mnav&gt;视频&lt;/a&gt; &lt;a href=http://tieba.baidu.com name=tj_trtieba class=mnav&gt;贴吧&lt;/a&gt; &lt;noscript&gt; &lt;a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb&gt;登录&lt;/a&gt; &lt;/noscript&gt; &lt;script&gt;document.write('&lt;a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb"&gt;登录&lt;/a&gt;');
&lt;/script&gt; &lt;a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;"&gt;更多产品&lt;/a&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;div id=ftCon&gt; &lt;div id=ftConw&gt; &lt;p id=lh&gt; &lt;a href=http://home.baidu.com&gt;关于百度&lt;/a&gt; &lt;a href=http://ir.baidu.com&gt;About Baidu&lt;/a&gt; &lt;/p&gt; &lt;p id=cp&gt;&copy;2017&nbsp;Baidu&nbsp;&lt;a href=http://www.baidu.com/duty/&gt;使用百度前必读&lt;/a&gt;&nbsp; &lt;a href=http://jianyi.baidu.com/ class=cp-feedback&gt;意见反馈&lt;/a&gt;&nbsp;京ICP证030173号&nbsp; &lt;img src=//www.baidu.com/img/gs.gif&gt; &lt;/p&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;/body&gt; &lt;/html&gt;

2.缓存(下载)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@litong ~]# curl -o baidu.html https://www.baidu.com  #将百度首页保存成html本地文件
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2443 100 2443 0 0 10319 0 --:--:-- --:--:-- --:--:-- 10351
[root@litong ~]# cat baidu.html
&lt;!DOCTYPE html&gt;
&lt;!--STATUS OK--&gt;&lt;html&gt; &lt;head&gt;&lt;meta http-equiv=content-type content=text/html;charset=utf-8&gt;&lt;meta http-equiv=X-UA-Compatible content=IE=Edge&gt;&lt;meta content=always name=referrer&gt;&lt;link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css&gt;&lt;title&gt;百度一下,你就知道&lt;/title&gt;&lt;/head&gt; &lt;body link=#0000cc&gt; &lt;div id=wrapper&gt; &lt;div id=head&gt; &lt;div class=head_wrapper&gt; &lt;div class=s_form&gt; &lt;div class=s_form_wrapper&gt; &lt;div id=lg&gt; &lt;img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129&gt; &lt;/div&gt;
... #省略片段
[root@litong ~]# curl -o Python-3.5.5.tgz https://www.python.org/ftp/python/3.5.5/Python-3.5.5.tgz #替换wget命令下载Python,断点续传-C
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 19.8M 100 19.8M 0 0 29.0M 0 --:--:-- --:--:-- --:--:-- 29.0M
[root@litong ~]# ll | grep Python
-rw-r--r-- 1 root root 20766931 May 22 17:38 Python-3.5.5.tgz

3.请求方式设置

1
2
3
4
5
[root@litong ~]# curl -H "Content-Type:application/json" -X POST -d '{ "arg": 125086 }' http://47.67.44.121:9130/learning-path/tree/queryPhaseAndSubject  #以json的方式进行请求
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 231 0 214 100 17 214 17 0:00:01 --:--:-- 0:00:01 14437
{"message":null,"result":"success","data":[{"subjectId":"4267aa39-6ecb-40b7-8e10-7b309d022aec","subjectName":"语文","phaseId":"c7241629-1ec4-4035-b04c-ef17f7866688","phaseName":"高中"}],"code":0,"success":true}

4.设置referer

1
[root@litong ~]# curl --referer http://www.google.com http://www.baidu.com

5.设置User-Agent

1
[root@litong ~]# curl -A "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36" http://www.baidu.com

6.设置cookie

1
[root@litong ~]# curl  --cookie "user=litong" http://www.baidu.com

7.设置请求头

1
[root@litong ~]# curl  -H "accept-language:zh-cn" http://www.baidu.com

8.进行认证

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@litong ~]# curl  -u litong:0517108 http://www.gitlab.com/users/sign_in
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0&lt;! DOCTYPE html&gt;
&lt;html class="devise-layout-html"&gt;
&lt;head prefix="og: http://ogp.me/ns#"&gt;
&lt;meta charset="utf-8"&gt;
&lt;meta content="IE=edge" http-equiv="X-UA-Compatible"&gt;
&lt;meta content="object" property="og:type"&gt;
&lt;meta content="GitLab" property="og:site_name"&gt;
&lt;meta content="Sign in" property="og:title"&gt;
&lt;meta content="GitLab Community Edition" property="og:description"&gt;
...省略片段

9.上传文件

1
2
3
4
 [root@litong ~]# curl http://127.0.0.1/up -F "file=@logo.png"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4616 0 76 100 4540 76 4540 0:00:01 --:--:-- 0:00:01 8213{"hash":"FmP639IPMcunQpiXidRPNisi2lDp","key":"FmP639IPMcunQpiXidRPNisi2lDp"}

支付宝打赏 微信打赏

请作者喝杯咖啡吧