likes
comments
collection
share

HttpClient的使用与连接资源释放

作者站长头像
站长
· 阅读数 18

HttpClient的使用与连接资源释放

前情概要:

在代码检查中,我写一个关于http连接的功能,引出了一个问题:要不要释放httpGet.releaseConnection(),要不要复用连接以及如何复用的问题。由于之前没有做过相关的了解,只会基础的使用,所以深入研究了一下,整理了一些经验分享,欢迎大家积极评论,指正不足。

介绍

服务与服务之间的调用与交互通常会使用Http请求来处理,HttpClient是常用的框架,主要实现了以下功能:

(1)实现了所有的HTTP方法(GET、POST、PUT、DELETE等)

(2)支持自动转向

(3)支持HTTPS协议

(4)支持代理服务器

在进行更深的学习和分析之前,先简单介绍一下httpclientjdk内部提供HttpURLConnection,可以实现对于http的请求等使用,很多公司和组织都会对Http进行封装再开发,提供更加方便使用的工具类,例如org.apache.httpcomponentshttpclient包,com.squareup.okhttp3okhttps等等,本文介绍的是apachehttpClient

一、请求类型

Http请求的基类是HttpRequestBase继承了AbstractExecutionAwareRequest类,并且实现了HttpUriRequestConfigurable接口,可以进行配置。

/**get*/
HttpGet, 
/**post*/
HttpPost, 
/**put*/
HttpPut,
/**patch*/
HttpPatch,
/**delete*/
HttpDelete,

/**其他*/
HttpHead,
HttpOptions,
HttpRequestBase, 
HttpRequestWrapper, 
HttpTrace, 
RequestWrapper
HttpEntityEnclosingRequestBase, 
EntityEnclosingRequestWrapper,

二、使用依赖

pom依赖:单独使用的话,可以引入apache的包,内部有对Http进行封装后的一些类的使用

<!--httpClient-->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.2</version>
</dependency>

三、参考文档

参考文档:

四、使用

4.1 获取httpClient

使用依赖包中的HttpClients来实例化CloseableHttpClient,其中的HttpClientBuilderCloseableHttpClient的建造者(参考设计模式-建造者模式)。

@Immutable
public class HttpClients {

    private HttpClients() {
        super();
    }

    public static HttpClientBuilder custom() {
        return HttpClientBuilder.create();
    }

    public static CloseableHttpClient createDefault() {
        return HttpClientBuilder.create().build();
    }

    public static CloseableHttpClient createSystem() {
        return HttpClientBuilder.create().useSystemProperties().build();
    }

    public static CloseableHttpClient createMinimal() {
        return new MinimalHttpClient(new PoolingHttpClientConnectionManager());
    }

    public static CloseableHttpClient createMinimal(final HttpClientConnectionManager connManager) {
        return new MinimalHttpClient(connManager);
    }
}

使用

// 使用工厂类 HttpClients 进行创建
// 1、默认配置创建
CloseableHttpClient httpClient = HttpClients.createDefault();

// 2、使用 builder来创建,可以添加自定义配置
// 自定义 connectionManager 连接管理器
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(connectionManager)
                .build();

4.2 相关配置

无论是采用的那种工厂方法实例化的CloseableHttpClient,其中都会有很多的配置,主要的有两个HttpClientConnectionManagerRequestConfig

4.2.1 HttpClientConnectionManager

HTTP连接管理器。它负责新HTTP连接的创建、管理连接的生命周期还有保证一个HTTP连接在某一时刻只被一个线程使用。

  • 实现

    • BasicHttpClientConnectionManager:每次只管理一个connection。不过,虽然它是thread-safe的,但由于它只管理一个连接,所以只能被一个线程使用。它在管理连接的时候如果发现有相同route的请求,会复用之前已经创建的连接,如果新来的请求不能复用之前的连接,它会关闭现有的连接并重新打开它来响应新的请求。
    • PoolingHttpClientConnectionManager:它管理着一个连接池。它可以同时为多个线程服务。每次新来一个请求,如果在连接池中已经存在route相同并且可用的connection,连接池就会直接复用这个connection;当不存在route相同的connection,就新建一个connection为之服务;如果连接池已满,则请求会等待直到被服务或者超时。
  • HttpClients.createDefault():默认创建的是PoolingHttpClientConnectionManager

  • 默认配置

        public PoolingHttpClientConnectionManager(
            final HttpClientConnectionOperator httpClientConnectionOperator,
            final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
            final long timeToLive, final TimeUnit tunit) {
            super();
            this.configData = new ConfigData();
            this.pool = new CPool(new InternalConnectionFactory(
                    this.configData, connFactory), 2, 20, timeToLive, tunit);
            this.pool.setValidateAfterInactivity(2000);
            this.connectionOperator = Args.notNull(httpClientConnectionOperator, "HttpClientConnectionOperator");
            this.isShutDown = new AtomicBoolean(false);
        }
    
    

4.2.2 RequestConfig

封装请求配置项的类

  • HttpClient.defaultConfig:默认配置参数

            Builder() {
                super();
                // 确定是否要使用陈旧的连接检查。 陈旧的连接检查可能会导致每个请求最多 30 毫秒的开销,并且应该仅在适当的时候使用。
                this.staleConnectionCheckEnabled = false;
                // 确定是否应自动处理重定向
                this.redirectsEnabled = true;
                // 返回要遵循的最大重定向数。 重定向次数限制旨在防止无限循环
                this.maxRedirects = 50;
                // 确定是否应拒绝相对重定向。 HTTP 规范要求位置值是绝对 URI
                this.relativeRedirectsAllowed = true;
                // 确定是否应自动处理身份验证
                this.authenticationEnabled = true;
                // 返回从连接管理器请求连接时使用的超时时间(以毫秒为单位)。 默认值: -1,为无限超时。
                this.connectionRequestTimeout = -1;
                // 确定建立连接之前的超时时间(以毫秒为单位)。 默认值: -1,为无限超时。
                this.connectTimeout = -1;
                // 以毫秒为单位定义套接字超时,它是等待数据的超时,或者换句话说,两个连续数据包之间的最长不活动时间。默认值: -1,为无限超时。 
                this.socketTimeout = -1;
                // 确定是否请求目标服务器压缩内容
                this.contentCompressionEnabled = true;
            }
    
  • 注意:默认配置中有几个超时时间都是-1,这是无限超时的意思,为了更好的使用和管理,在使用的过程中需要对这几个参数进行设置,如果没有设置的话,请求会持续存在,也不会抛出异常,十分不方便处理

    • connectionRequestTimeout:返回从连接管理器请求连接时使用的超时时间

    • connectTimeout:连接超时

    • socketTimeout:读取数据超时

使用

  • 配置给HttpClient:所有该 httpClient执行的请求,如果没有指定配置,则都会采用该defaultRequestConfig

  • 配置给HttpRequest-methods:配置了requestConfig,在请求时使用该配置

        // 创建http 请求配置
        RequestConfig requestConfig = RequestConfig.custom()
            .setConnectTimeout(5 * 1000)
            .setConnectionRequestTimeout(5 * 1000)
            .setSocketTimeout(5 * 1000)
            .build();
    
         // 1.配置给CloseableHttpClient
         CloseableHttpClient  httpClient = HttpClients.custom()
                    .setConnectionManager(connectionManager)
                    .setDefaultRequestConfig(requestConfig)
                    .build();
    
        // 2.配置http GET请求
        HttpGet httpGet = new HttpGet(url);
        httpGet.setConfig(requestConfig);
    

4.3 使用示例:GET

在需要进行请求时,创建httpClient,然后创建HttpGet请求,配置路由、请求头、请求参数等,接收execute请求,获取结果并处理。

    public static void test(String url) {

        // 创建http client客户端
        CloseableHttpClient httpClient = HttpClients.createDefault();
        
     	// 创建http 请求配置
        RequestConfig requestConfig = RequestConfig.custom()
                .setConnectTimeout(5 * 1000)
                .setConnectionRequestTimeout(5 * 1000)
                .setSocketTimeout(5 * 1000)
                .build();
        
        // 创建http GET请求
        HttpGet httpGet = new HttpGet(url);
        httpGet.setConfig(requestConfig);
        // 设置请求头部编码
        httpGet.setHeader(new BasicHeader("Content-Type", "application/x-www-form-urlencoded; charset=utf-8"));
        // 设置返回编码
        httpGet.setHeader(new BasicHeader("Accept", "text/plain;charset=utf-8"));

        // 返回响应
        CloseableHttpResponse response = null;
        try {
            response = httpClient.execute(httpGet);
			// 判断响应码
            if (response.getStatusLine().getStatusCode() == 200) {
                HttpEntity entity = response.getEntity();
                // 使用工具类EntityUtils 从响应中读取内容
                String result = EntityUtils.toString(entity, "utf-8");
                System.out.println(result);
            }

        } catch (Exception e) {
            System.out.print("http GET 请求异常" + e);
        } finally {

            // 释放资源
            try {
                if (response != null) {
                    response.close();
                }
            } catch (IOException e) {
                System.out.print("关闭流异常" + e);
            }
            
            // 关闭客户端
            try {
                httpClient.close();
            } catch (IOException e) {
                System.out.print("关闭HttpClient异常" + e);
            }
        }
    }

五、问题探讨

示例中有多个Close,分别是关闭了什么呢,是否可以省略,又在什么时候调用呢?在使用过程中,有时也会涉及到releaseConnection(),这又是什么?有什么作用?是否是必要的呢?

5.1 关闭

  • response.close():官网的解释是,最底层的HTTP connection是由响应对象response持有的,如果没有完全的消费response content或者正确地关闭,对应的connection是不能被安全重用的,会被connection manager给关闭和丢弃。

  • httpClient.close():关闭客户端,会先关闭客户端中的所有连接,然后销毁客户端。

  • method.releaseConnection():释放连接到连接池。

5.2 不关闭

所有的资源都是有限的,如果持续消费资源而不释放资源,很快就会出现因为资源获取不到而导致进程阻塞,参考一个常见的问题就是**死锁问题。在开发过程中,很多时候都会因为没注意到这点导致程序出现问题(比如:流未关闭,资源就释放不了),一旦并发量**、数据量等上升,问题出现的几率和产生的影响可能成几何倍增长,所以一直强调要资源释放,就是这个问题。

HttpClient使用过程中也会出现这样的问题,下面我们来探讨一下,如果不关闭资源,会出现什么样的问题,不同的方式来关闭又会出现什么样的问题。

5.3 response

问题:消费不彻底

多次请求,对于response消费不彻底,没有进行关闭

// 相同的URL,多次请求
    public static void testNoCloseResponse(String url, int num) {
        // 创建http client客户端
        CloseableHttpClient httpClient = HttpClients.createDefault();
        // 创建http GET请求
        HttpGet httpGet = new HttpGet(url);
        // 设置请求头部编码
        httpGet.setHeader(new BasicHeader("Content-Type", "application/x-www-form-urlencoded; charset=utf-8"));
        // 设置返回编码
        httpGet.setHeader(new BasicHeader("Accept", "text/plain;charset=utf-8"));

        for (int i = 0; i < num; i++) {
            // 返回响应
            CloseableHttpResponse response = null;
            try {
                response = httpClient.execute(httpGet);
//                if (response.getStatusLine().getStatusCode() == 200) {
//                    HttpEntity entity = response.getEntity();
//                    // 使用工具类EntityUtils 从响应中读取内容
//                    String result = EntityUtils.toString(entity, "utf-8");
//                    System.out.println(result);
//                }
            } catch (Exception e) {
                System.out.print("http GET 请求异常" + e);
            }finally{
//                // 释放资源
//                try {
//                    if (response != null) {
//                        response.close();
//                    }
//                } catch (IOException e) {
//                    e.printStackTrace();
//                }
            }
        }
    }

第一次连接:连接无法被复用,kept alive 0,同时占用了一个route

16:37:15.048 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
16:37:15.119 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]

// 连接完成后
httpClient.connManager.pool = [leased: [[id:0][route:{}->http://172.23.22.58:8081][state:null]]][available: []][pending: []]

第二次连接:连接无法被复用,kept alive 0,相同的IP和请求路由,又占用了一个route

16:41:57.223 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
16:41:57.224 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 20]

// 连接完成后
httpClient.connManager.pool = [leased: [[id:1][route:{}->http://172.23.22.58:8081][state:null], [id:0][route:{}->http://172.23.22.58:8081][state:null]]][available: []][pending: []]

第三次连接:相同的IP和请求路由,由于默认配置中:httpClient.connManager.pool.defaultMaxPerRoute = 2(相同的请求路径最多可以同时存在2个),没有可用的route此时就会一直等待原连接的释放,获取到route之后才可以进行连接。

问题:消费彻底

使用工具类消费能够更加彻底地消费response,可以达到释放资源,复用的效果,但是如果关闭response,仍然无法复用

            // 返回响应
            CloseableHttpResponse response = null;
            try {
                response = httpClient.execute(httpGet);
//                if (response.getStatusLine().getStatusCode() == 200) {
//                    HttpEntity entity = response.getEntity();
//                    // 使用工具类EntityUtils 从响应中读取内容
//                    String result = EntityUtils.toString(entity, "utf-8");
//                    System.out.println(result);
//                }
            } catch (Exception e) {
                System.out.print("http GET 请求异常" + e);
            }

第一次连接:total kept alive: 1,连接可以被复用

16:56:23.188 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
16:56:23.188 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]

第二次连接:虽然有一个连接可以复用,但是在尝试复用的时候,发现该通道对应的流并没有关闭,无法使用,所以在关闭了该连接后,重新生成了一个

16:57:32.922 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
16:57:32.922 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "end of stream"
16:57:32.922 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-0: Close connection
16:57:32.922 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
16:57:32.922 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Opening connection {}->http://172.23.22.58:8081

第三次连接:与第二次相同

17:05:07.966 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
17:05:07.966 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "end of stream"
17:05:07.966 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-1: Close connection
17:05:07.966 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 2][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
17:05:07.966 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Opening connection {}->http://172.23.22.58:8081

问题:关闭资源

但是与response消费不彻底相比,并没有阻塞第三次的请求,这是能够让资源重复使用的一个提高点。

在每次请求后关闭response,则效果如下:

// 释放资源或者使用 try-resources可以自动关闭
try {
    if (response != null) {
        response.close();
    }
} catch (IOException e) {
    e.printStackTrace();
}

第一次连接:total kept alive: 1,连接可以被复用

17:09:10.296 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
17:09:10.309 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]

17:09:10.341 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
17:09:10.341 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]

第二、三次连接:可以复用之前的连接,也不会增加新的routeallocated(出现了http-outgoing-0 << "[read] I/O error: Read timed out"的报错,没有仔细研究,原因可能是和http版本的协议相关,感兴趣的可以深入了解)

17:09:18.121 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
17:09:18.135 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[read] I/O error: Read timed out"
17:09:18.135 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]

17:09:18.139 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
17:09:18.139 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]

5.4 httpClient

在日常使用中,我们通常会使用HttpClients.createDefault()方式来获取客户端,这种方式采用了默认配置。在系统中,资源是有限的,而应用服务需要处理的请求和操作是无限的,如何提高Http连接的使用效率那就要考虑到及时回收资源,合理分配资源。

如果每次在使用的时候都新生成一个,不关闭HttpClient,很显然,无限制的生成那么必然会导致资源的浪费,这是一种不可取的方式。

5.4.1 httpClient.close

  • 在结束使用的时候httpClient.close(),在close()的时候,会对内部的pool进行shutdowm(),关闭所有的可用连接、正在进行的连接,释放所有的资源。

       public void shutdown() throws IOException {
            if (this.isShutDown) {
                return ;
            }
            this.isShutDown = true;
            this.lock.lock();
            try {
                for (final E entry: this.available) {
                    entry.close();
                }
                for (final E entry: this.leased) {
                    entry.close();
                }
                for (final RouteSpecificPool<T, C, E> pool: this.routeToPool.values()) {
                    pool.shutdown();
                }
                this.routeToPool.clear();
                this.leased.clear();
                this.available.clear();
            } finally {
                this.lock.unlock();
            }
        }
    

使用11个不同的请求URL,流程示例:

  • 关闭前

    • 在使用的连接数:0
    • 可用连接数:11
    • 等待请求数:0
    httpClient.connManager.pool
    [leased: []]
    [available: [
    	[id:10][route:{}->http://py.qianlong.com:80][state:null], 
    	[id:9][route:{}->http://www.bnia.cn:80][state:null], 
    	[id:8][route:{s}->https://m.you.163.com:443][state:null], 
    	[id:7][route:{}->http://www.wenming.cn:80][state:null], 
    	[id:6][route:{}->http://jubao.aq.163.com:80][state:null], 
    	[id:5][route:{s}->https://www.12377.cn:443][state:null], 
    	[id:4][route:{}->http://www.12377.cn:80][state:null], 
    	[id:3][route:{}->http://www.bjjubao.org:80][state:null], 
    	[id:2][route:{}->http://cimg.163.com:80][state:null], 
    	[id:1][route:{s}->https://static.ws.126.net:443][state:null], 
    	[id:0][route:{}->http://www.baidu.com:80][state:null]]
    	]
    [pending: []]
    
  • 关闭

    18:36:51.900 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager is shutting down
    18:36:51.900 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-10: Close connection
    18:36:51.901 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-9: Close connection
    18:36:51.901 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-8: Close connection
    18:36:51.903 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-7: Close connection
    18:36:51.903 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-6: Close connection
    18:36:51.903 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-5: Close connection
    18:36:51.904 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-4: Close connection
    18:36:51.904 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-3: Close connection
    18:36:51.905 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-2: Close connection
    18:36:51.905 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-1: Close connection
    18:36:51.906 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-0: Close connection
    18:36:51.906 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager shut down
    
  • 关闭后

    • 在使用的连接数:0
    • 可用连接数:0
    • 等待请求数:0
    httpClient.connManager.pool
    [leased: []]
    [available: []]
    [pending: []]
    

5.4.2 如何高效获取和使用httpClient

获取HttpClient主要有以下三种方式:

  1. 使用时生成

  2. 连接池获取

  3. 全局共享


1. 使用时生成

使用时生成,即每次在请求时,初始化生成一个HttpClient,接着在生成连接对象(例如:httpPost / httpGet ),进行连接,然后从返回结果取出entity,保存成一个字符串,最后显式关闭responsehttpClient

通过httpClient.close()的源码和示例,可以知道在这个过程中,反复创建HttpClient、创建TCP连接的开销,使用完成后再销毁的开销,对于高频次的请求,那么很显然消耗会很大,考虑可以通过实现连接的 复用 ,从而降低开销,提高效率。

2.连接池获取

上面使用时生成中提到,可以通过复用来提高效率,一是对Httpclient的使用,二是对连接的使用。

显然,先想到的就是使用连接池,通过创建连接池的方式,每次需要请求的时候,从连接池中获取,接着进行请求的相关操作。

  • 池形式地获取HttpClient

    这种方式是提高了效率,只不过提高的是获取client的效率,每次建立连接的开销并没有降低。不过可以通过共享连接池,使得多个HttpClient可以共享一个连接管理器。

  • 池形式地获取Connection

    以连接为最小元,连接池的方式来获取。

HttpClient本身就实现了连接池式的管理器。

3.全局共享

HttpClient是线程安全的类,没有必要每次使用时创建,我们可以全局共享同一个,同时apache提供的HttpClient中就有连接池的存在,用于管理connectionconnManager(PoolingHttpClientConnectionManager),可以实现连接的复用。

  • requestConfig:用于配置请求的参数

    • setConnectionRequestTimeout:返回从连接管理器请求连接时使用的超时时间(以毫秒为单位)
    • setConnectTimeout:确定建立连接之前的超时时间(以毫秒为单位)
    • setSocketTimeout:以毫秒为单位定义套接字超时,它是等待数据的超时,或者换句话说,两个连续数据包之间的最长不活动时间
  • connectionManager:用于配置HttpClients中的连接池

    • setMaxTotal:设置连接池的最大连接数
    • setDefaultMaxPerRoute:设置每个路由上的默认连接个数
    • setMaxPerRoute:则单独为某个站点设置最大连接个数
  • 示例

        /**
         * 请求配置
         */
        private static RequestConfig requestConfig;
        /**
         * Http客户端
         */
        private static CloseableHttpClient httpClient;
    
        static {
    
            // 配置请求参数,请求时长,连接时长,读取数据时长
            requestConfig = RequestConfig.custom()
                    .setConnectTimeout(5*1000)
                    .setConnectionRequestTimeout(5*1000)
                    .setSocketTimeout(5*1000)
                    .build();
    
            // 配置连接池关联
            PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
            connectionManager.setMaxTotal(100);
            connectionManager.setDefaultMaxPerRoute(10);
            // 初始化客户端
            httpClient = HttpClients.custom()
                    .setConnectionManager(connectionManager)
                    .setConnectionTimeToLive(1, TimeUnit.MINUTES)
                    .build();
        }
    
  • 使用示例

    在使用的时候可以全局获取httpClient,使用requestConfig对请求进行配置,每次使用完成后,也不用对httpClient进行关闭。

        public String doGet(String url) {
            // 创建http GET请求
            HttpGet httpGet = new HttpGet(url);
            // 设置请求头部编码
            httpGet.setHeader(new BasicHeader("Content-Type", "application/x-www-form-urlencoded; charset=utf-8"));
            // 设置返回编码
            httpGet.setHeader(new BasicHeader("Accept", "text/plain;charset=utf-8"));
    
            // 配置连接,如果没有对httpClient设置默认配置
            // httpGet.setConfig(requestConfig);
            
            try (CloseableHttpResponse response = httpClient.execute(httpGet)) {
    
                // 判断返回状态是否为200
                if (response.getStatusLine().getStatusCode() == CODE_SUCCESS) {
                    return EntityUtils.toString(response.getEntity(), ENCODING);
                }
            } catch (Exception e) {
                logger.error("http 无参  GET 请求异常", e);
            }
            return null;
        }
    

5.5 releaseConnection

connection的释放,在使用过程中很少会涉及,也没有很清楚的说明是否要执行releaseConnection(),那么connection要不要释放呢?如果不用,那为什么?如果用,那要怎么释放?

首先要知道,releaseConnection做的是什么事情。

5.5.1 基于请求methodsreleaseConnection

Http请求连接底层类 是HttpRequestBase ,其中有一个方法是releaseConnection

public abstract class HttpRequestBase extends AbstractExecutionAwareRequest
    implements HttpUriRequest, Configurable {
    /**
     * A convenience method to simplify migration from HttpClient 3.1 API. This method is
     * equivalent to {@link #reset()}.
     *
     * @since 4.2
     */
    public void releaseConnection() {
        reset();
    }
}

其中的reset()是继承AbstractExecutionAwareRequest类中的方法,重置了内部的状态,使得该请求可以重用。

public abstract class AbstractExecutionAwareRequest extends AbstractHttpMessage implements
        HttpExecutionAware, AbortableHttpRequest, Cloneable, HttpRequest { 
    
    private final AtomicBoolean aborted;
    
	/**
     * Resets internal state of the request making it reusable.
     *
     * @since 4.2
     */
    public void reset() {
        final Cancellable cancellable = this.cancellableRef.getAndSet(null);
        if (cancellable != null) {
            cancellable.cancel();
        }
        this.aborted.set(false);
    }
}

5.5.2 基于请求ConnectionRequestreleaseConnection

在请求中,最底层使用的对象接口是ConnectionRequest,通过HttpClientConnectionManager进行管理。HttpClientConnectionManager,HTTP 连接管理器的目的是作为新 HTTP 连接的工厂,管理持久连接并同步对持久连接的访问,确保一次只有一个执行线程可以访问一个连接。 因为此接口的方法可以从多个线程执行,对共享数据的访问必须同步,此接口的实现必须是线程安全的。

/**
 * 持久客户端连接的管理器
 * @since 4.3
 */
public interface HttpClientConnectionManager {

	/**
	 * 返回一个新的ConnectionRequest ,从中可以获得一个HttpClientConnection或者可以中止请求
	 */
    ConnectionRequest requestConnection(HttpRoute route, Object state);

    /**
	 * 释放与管理器的连接,使其有可能被其他消费者重用
	 * 可以使用validDuration和timeUnit参数定义管理器应保持连接活动的validDuration timeUnit 
	 * conn – 要释放的托管连接
	 * validDuration – 此连接可重复使用的持续时间
	 * timeUnit – 时间单位
	 */
    void releaseConnection(
            HttpClientConnection conn, Object newState, long validDuration, TimeUnit timeUnit);
}

PoolingHttpClientConnectionManager:连接池对于connection进行释放,keepalive连接的可用持续时长(存活时间)也会影响到连接池对连接的处理。

    @Override
    public void releaseConnection(
            final HttpClientConnection managedConn,
            final Object state,
            final long keepalive, final TimeUnit tunit) {
        Args.notNull(managedConn, "Managed connection");
        // synchronized:线程安全的方式获取连接,对连接进行操作
        synchronized (managedConn) {
            final CPoolEntry entry = CPoolProxy.detach(managedConn);
            if (entry == null) {
                return;
            }
            final ManagedHttpClientConnection conn = entry.getConnection();
            try {
                if (conn.isOpen()) {
                    final TimeUnit effectiveUnit = tunit != null ? tunit : TimeUnit.MILLISECONDS;
                    entry.setState(state);
                    entry.updateExpiry(keepalive, effectiveUnit);
                    if (this.log.isDebugEnabled()) {
                        final String s;
                        // keepalive,连接可重复使用的持续时间
                        if (keepalive > 0) {
                            s = "for " + (double) effectiveUnit.toMillis(keepalive) / 1000 + " seconds";
                        } else {
                            s = "indefinitely";
                        }
                        this.log.debug("Connection " + format(entry) + " can be kept alive " + s);
                    }
                }
            } finally {
                // 连接池进行 release,不同keepalive也会影响到连接池对连接的释放操作
                this.pool.release(entry, conn.isOpen() && entry.isRouteComplete());
                if (this.log.isDebugEnabled()) {
                    this.log.debug("Connection released: " + format(entry) + formatStats(entry.getRoute()));
                }
            }
        }
    }

5.5.3 不释放连接测试

5.5.3.1 少量固定请求

模拟请求重复连接

  • 前置:10个线程连续请求一个相同的url

  • 自定义配置:最大总连接数20,相同route最多为2个连接

  /**
     * 连接复用:会复用连接池中的已有连接
     *
     * @param num 次数
     */
    public static void testMultithreading3(int num) {
        String url = "http://www.baidu.com";

        CloseableHttpClient httpClient = HttpClients.createDefault();

        HttpGet get = new HttpGet(url);

        MultiHttpClientConnThread[] threads = new MultiHttpClientConnThread[num];
        for (int i = 0; i < num; i++) {
            threads[i] = new MultiHttpClientConnThread(httpClient, get);
        }

        try {
            for (int i = 0; i < num; i++) {
                threads[i].start();
            }

            for (int i = 0; i < num; i++) {
                threads[i].join();
            }
        } catch (InterruptedException e) {
            System.out.println("线程执行异常" + e);
        }
    }

// 执行请求的线程
public class MultiHttpClientConnThread extends Thread {

    private CloseableHttpClient client;

    private HttpGet get;

    public MultiHttpClientConnThread(CloseableHttpClient httpClient, HttpGet get) {
        this.client = httpClient;
        this.get = get;
    }

    @Override
    public void run() {
        CloseableHttpResponse response = null;
        try {
            response = client.execute(get);
            EntityUtils.consume(response.getEntity());

        } catch (ClientProtocolException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (response != null) {
                IOUtils.closeQuietly(response);
            }
        }
    }
}

截取的控制台输出:

  • 阶段一:10个线程开始获取连接请求,Connection request
  • 阶段二:有2个线程优先获得了请求连接的资源,Connection leased
  • 阶段三:有线程持有的请求连接完成请求,由于route总数限制为2,连接池管理释放连接,Connection released
  • 阶段四:重复阶段三,直到多有线程都获取到资源,完成了请求连接
  • 阶段五:进程结束,释放所有连接资源
23:28:06.718 [Thread-6] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.718 [Thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.718 [Thread-7] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.719 [Thread-5] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.718 [Thread-8] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.719 [Thread-0] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.719 [Thread-4] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.719 [Thread-9] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.720 [Thread-2] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 10]
23:28:06.735 [Thread-9] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 10]
23:28:06.735 [Thread-2] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.735 [Thread-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 10]
23:28:06.766 [Thread-2] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 1][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.766 [Thread-9] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.766 [Thread-9] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 2; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.766 [Thread-8] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.767 [Thread-2] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.767 [Thread-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.769 [Thread-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.769 [Thread-3] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.769 [Thread-8] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 1][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.769 [Thread-8] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.769 [Thread-4] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.769 [Thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.771 [Thread-4] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.771 [Thread-4] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.771 [Thread-0] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.771 [Thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 1][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.771 [Thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.771 [Thread-7] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.773 [Thread-7] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 1][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.774 [Thread-0] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.774 [Thread-0] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 1; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.774 [Thread-5] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.774 [Thread-6] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 0; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.776 [Thread-6] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.776 [Thread-5] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 1][route: {}->http://172.23.22.58:8081] can be kept alive indefinitely
23:28:06.776 [Thread-5] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 1][route: {}->http://172.23.22.58:8081][total kept alive: 2; route allocated: 2 of 2; total allocated: 2 of 10]
23:28:06.776 [Thread-6] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.23.22.58:8081][total kept alive: 2; route allocated: 2 of 2; total allocated: 2 of 10]

执行结果:上示示例有 10 个线程,并发执行 10 个请求,由于连接池的限制,同一个route最多2个请求。在执行过程中,虽然线程会等待,但是10个线程在请求过程中一直用的都是相同的2 个连接,实现了复用。

Connection [id: 1][route: {}->http://172.23.22.58:8081]
Connection [id: 0][route: {}->http://172.23.22.58:8081]
5.5.3.2 大量不固定请求

模拟请求超出最大连接数

  • 前置:一共有11个不同的url,进行请求

  • 自定义配置:最大总连接数10,相同route最多为2个连接

  • 结果:

    • id来看,第11次请求的时候,由于连接数最大为10已满,将最久未使用的http-outgoing-1:connection [id:1]给关闭了,同时创建了新的连接 Connection leased: [id: 11]
    • 关闭client时,共关闭了10个连接,没有[id:11] 取代了 [id:11]
    10:16:25.231 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://py.qianlong.com:80][total kept alive: 10; route allocated: 0 of 2; total allocated: 10 of 10]
    10:16:25.231 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-1: Close connection
    10:16:25.231 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 11][route: {}->http://py.qianlong.com:80][total kept alive: 9; route allocated: 1 of 2; total allocated: 10 of 10]
    10:16:25.231 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Opening connection {}->http://py.qianlong.com:80
    
    10:16:25.362 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 11][route: {}->http://py.qianlong.com:80] can be kept alive indefinitely
    10:16:25.362 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 11][route: {}->http://py.qianlong.com:80][total kept alive: 10; route allocated: 1 of 2; total allocated: 10 of 10]
    10:16:25.362 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager is shutting down
    10:16:25.362 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-11: Close connection
    10:16:25.362 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-10: Close connection
    10:16:25.363 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-9: Close connection
    10:16:25.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-8: Close connection
    10:16:25.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-7: Close connection
    10:16:25.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-6: Close connection
    10:16:25.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-5: Close connection
    10:16:25.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-4: Close connection
    10:16:25.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-3: Close connection
    10:16:25.365 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-2: Close connection
    
5.5.3.3 结果

对于PoolingHttpClientConnectionManager

  • 从上述两个测试来说,使用连接池来管理连接的时候,可以对连接进行复用,在连接达到连接池最大数时,也会采用相应的策略来对连接进行关闭,从而释放出资源,创建新的请求连接。

对于BasicHttpClientConnectionManager

  • 单个连接的连接管理器,由于每次只允许一个线程进行一个连接,所以显示的releaseconnection也就没有必要,管理器内部就会去执行。

对于使用时创建的HttpClient

  • 在结束使用的时候httpClient.close(),在close()的时候,会对内部的pool进行shutdowm(),关闭所有的可用连接、正在进行的连接。

最终,releaseConnection是否是必须的呢?答案是不必须,不过在使用不同方式的HttpClient的时候和在请求的时候还是要注意对资源的释放的,毕竟服务器资源就那么多,要合理利用。

5.5.3.4 附加:用时测试
  • 释放和不释放用时统计(单线程 + 相同连接)
  • 结果:
    • 请求次数在1000以内的话,少量相同的连接不释放的速度更快
    • 在超过10000的时候,反而释放更快了(表中未列出)
  • 注:本次测试为单机,同一台机器作为服务和请求双方;服务端处理较为简单;
connection(ms)110501002005001000
释放第一次763717531678110651314
释放第二次7831962023607721245
释放第三次7531941923018271575
不释放第一次7330861652747061365
不释放第二次7929871302325341452
不释放第三次7530851492917951232

六、场景及策略

场景用已经资源的关闭等当面都做了介绍,下面用分享一下不同场景下,使用的不同策略。

6.1 请求数量少,间隔时间长

  • 场景:单线程(主线程调用)或多线程少量;很长时间才请求一次的话,对于请求的响应等要求不高;不与用户操作相关联(不用考虑及时反馈)
  • 策略:可以不用考虑请求资源的复用
    • 可以用使用时生成,结束的时候,关闭httpClient,回收所有资源,等待下次使用。

6.2 请求数量多,间隔事件短

  • 场景:多线程请求;请求时间间隔短,对于请求的响应要求高;与用户操作相关联(需要及时反馈)
  • 策略:考虑资源的复用与回收
    • 全局共享方式来使用HttpClient,降低创建、销毁连接的开销

七、全局共享HttpClient的使用方法

基于单例的形式,全局共享HtppClient,通过PoolingHttpClientConnectionManager连接池管理器的方式来实现连接的高效获取和复用,下面是整理的一套使用的具体代码示例。

7.1 依赖配置pom

    <!--httpClient-->
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.2</version>
    </dependency>

7.2 配置类HttpClientConfig

/**
 * HttpClientProperties
 *
 * @author xuzhou
 * @version v1.0.0
 * @create 2021/7/22 16:35
 */
@Component
@ConfigurationProperties(prefix = "http.client")
public class HttpClientConfig {
    /**
     * 返回从连接管理器请求连接时使用的超时时间(以毫秒为单位)。
     * 默认值: -1,为无限超时。
     */
    private int connectionRequestTimeout = 5000;

    /**
     * 连接超时:连接一个url的连接等待时间
     * 确定建立连接之前的超时时间(以毫秒为单位)。
     * 默认值: -1,为无限超时。
     */
    private int connectTimeout = 5000;

    /**
     * 读取数据超时:连上url,获取response的返回等待时间
     * 以毫秒为单位定义套接字超时,它是等待数据的超时,或者换句话说,两个连续数据包之间的最长不活动时间。
     * 默认值: -1,为无限超时。
     */
    private int socketTimeout = 5000;

    /**
     * 客户端总并行链接最大数
     */
    private int maxTotal = 50;

    /**
     * 客户端每个路由最高链接最大数
     */
    private int maxPreRoute = 4;

    /**
     * 连接存活时长:秒
     */
    private long connectionTimeToLive = 60;

    /**
     * 重试尝试最大次数
     * 默认为3
     */
    private int retryCount = 3;

    /**
     * 非幂等请求是否可以重试
     * 默认不开启
     */
    private boolean requestSentRetryEnabled = false;

    public int getConnectionRequestTimeout() {
        return connectionRequestTimeout;
    }

    public void setConnectionRequestTimeout(int connectionRequestTimeout) {
        this.connectionRequestTimeout = connectionRequestTimeout;
    }

    public int getConnectTimeout() {
        return connectTimeout;
    }

    public void setConnectTimeout(int connectTimeout) {
        this.connectTimeout = connectTimeout;
    }

    public int getSocketTimeout() {
        return socketTimeout;
    }

    public void setSocketTimeout(int socketTimeout) {
        this.socketTimeout = socketTimeout;
    }

    public int getMaxTotal() {
        return maxTotal;
    }

    public void setMaxTotal(int maxTotal) {
        this.maxTotal = maxTotal;
    }

    public int getMaxPreRoute() {
        return maxPreRoute;
    }

    public void setMaxPreRoute(int maxPreRoute) {
        this.maxPreRoute = maxPreRoute;
    }

    public long getConnectionTimeToLive() {
        return connectionTimeToLive;
    }

    public void setConnectionTimeToLive(long connectionTimeToLive) {
        this.connectionTimeToLive = connectionTimeToLive;
    }

    public int getRetryCount() {
        return retryCount;
    }

    public void setRetryCount(int retryCount) {
        this.retryCount = retryCount;
    }

    public boolean isRequestSentRetryEnabled() {
        return requestSentRetryEnabled;
    }

    public void setRequestSentRetryEnabled(boolean requestSentRetryEnabled) {
        this.requestSentRetryEnabled = requestSentRetryEnabled;
    }
}

7.3 结果类HttpResult

/**
 * http请求返回对象
 *
 * @author xuzhou
 * @version 1.0.0
 */
public class HttpResult {

    /**
     * 状态码
     */
    private Integer status;
    /**
     * 返回数据
     */
    private String stringEntity;

    public HttpResult() {
    }

    /**
     * http请求返回对象
     *
     * @param status       返回状态
     * @param stringEntity 返回数据
     */
    public HttpResult(Integer status, String stringEntity) {
        this.status = status;
        this.stringEntity = stringEntity;
    }

    public Integer getStatus() {
        return status;
    }

    public void setStatus(Integer status) {
        this.status = status;
    }

    public String getStringEntity() {
        return stringEntity;
    }

    public void setStringEntity(String stringEntity) {
        this.stringEntity = stringEntity;
    }

    @Override
    public String toString() {
        return "HttpResult{" +
                "status=" + status +
                ", stringEntity='" + stringEntity + '\'' +
                '}';
    }
}

7.4 工具类HttpClientUtils

/**
 * http连接工具
 * 不会对请求的结果做处理,用户可以访问 {@link HttpResult}
 * 通过{@linkplain HttpResult#getStatus()}判断响应码code
 * 通过{@linkplain HttpResult#getStringEntity()}获取响应实体字符串
 *
 * @author xuzhou
 * @version v1.0.0
 */
public class HttpClientUtils {

    /**
     * Http客户端
     */
    public static final CloseableHttpClient httpClient;

    /**
     * 配置类
     */
    private static final HttpClientConfig HTTP_CLIENT_CONFIG;

    /**
     * 编码方式
     */
    private static final String ENCODING = "utf-8";

    /**
     * 日志对象
     */
    private static final Logger log = LoggerFactory.getLogger(HttpClientUtils.class);

    /**
     * 请求配置
     */
    private static final RequestConfig request_config;

    static {
        // 配置类
        HTTP_CLIENT_CONFIG = new HttpClientConfig();

        // 配置请求参数,请求时常,连接市场,读取数据时长
        request_config = RequestConfig.custom()
                .setConnectTimeout(HTTP_CLIENT_CONFIG.getConnectTimeout())
                .setConnectionRequestTimeout(HTTP_CLIENT_CONFIG.getConnectionRequestTimeout())
                .setSocketTimeout(HTTP_CLIENT_CONFIG.getSocketTimeout())
                .build();

        // 配置连接池关联
        PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
        connectionManager.setMaxTotal(HTTP_CLIENT_CONFIG.getMaxTotal());
        connectionManager.setDefaultMaxPerRoute(HTTP_CLIENT_CONFIG.getMaxPreRoute());

        // 初始化客户端
        httpClient = HttpClients.custom()
                .setConnectionManager(connectionManager)
                .setDefaultRequestConfig(request_config)
                // 重试机制
                .setRetryHandler(new DefaultHttpRequestRetryHandler(HTTP_CLIENT_CONFIG.getRetryCount(), HTTP_CLIENT_CONFIG.isRequestSentRetryEnabled()))
                // 开启后台线程清除过期的连接
                .evictExpiredConnections()
                // 开启后台线程清除闲置的连接
                .evictIdleConnections(HTTP_CLIENT_CONFIG.getConnectionTimeToLive(), TimeUnit.SECONDS)
                .build();
    }

    private HttpClientUtils() {

    }


    /**
     * GET请求
     * 1.支持不带参数的请求
     * 2.支持参数拼接在URl中的请求
     *
     * @param url 请求地址
     * @return 返回值
     */
    public static HttpResult doGet(String url) {
        return doGet(url, null, null);
    }

    /**
     * 带有参数的GET请求
     *
     * @param url    请求地址
     * @param params 请求参数
     * @return 返回值
     */
    public static HttpResult doGet(String url, Map<String, Object> params) {
        return doGet(url, params, null);
    }

    /**
     * Get 请求:指定请求头,请求参数
     *
     * @param url     请求地址
     * @param headers 请求头参数
     * @param params  请求参数
     * @return HttpResult
     */
    public static HttpResult doGet(String url, Map<String, Object> params, Map<String, String> headers) {

        log.info("Http GET 请求URL:{}", url);
        log.info("Http GET 请求参数:{}", JSONObject.toJSONString(params));

        try {
            // 创建访问对象地址
            URIBuilder uriBuilder = new URIBuilder(url);
            if (params != null && !params.isEmpty()) {
                // 构建在URL中的请求参数
                Set<? extends Entry<?, ?>> entrySet = params.entrySet();
                for (Entry<?, ?> entry : entrySet) {
                    uriBuilder.addParameter((String) entry.getKey(), String.valueOf(entry.getValue()));
                }
            }

            HttpGet httpGet = new HttpGet(uriBuilder.build().toString());

            // 封装请求头
            packageHeader(headers, httpGet);

            return execute(httpGet);
        } catch (URISyntaxException e) {
            log.error("Get请求构建URL失败", e);
        }
        return null;
    }

    /**
     * 执行POST请求
     *
     * @param url 请求地址
     * @return 返回值
     */
    public static HttpResult doPost(String url) {
        return doPost(url, null, null);
    }

    /**
     * 执行POST请求:有参数
     *
     * @param url    请求地址
     * @param params 请求参数
     * @return 返回值
     */
    public static HttpResult doPost(String url, Map<String, Object> params) {
        return doPost(url, params, null);
    }

    /**
     * 执行POST请求
     *
     * @param url     请求地址
     * @param headers 请求头
     * @param params  请求参数
     * @return 返回值
     */
    public static HttpResult doPost(String url, Map<String, Object> params, Map<String, String> headers) {

        log.info("Http POST 请求URL:{}", url);
        log.info("Http POST 请求参数:{}", JSONObject.toJSONString(params));

        // 创建http POST请求
        HttpPost httpPost = new HttpPost(url);

        try {
            // 封装请求头
            packageHeader(headers, httpPost);

            // 封装请求参数
            packageParam(params, httpPost);
            return execute(httpPost);

        } catch (UnsupportedEncodingException e) {
            log.error("POST请求参数编码异常", e);
        }

        return null;
    }

    /**
     * http post json数据
     *
     * @param url  请求地址
     * @param json 请求参数
     * @return 返回值
     */
    public static HttpResult doPostJson(String url, String json) {
        return doPostJson(url, json, null);
    }


    /**
     * http post json数据
     *
     * @param url     请求地址
     * @param json    请求参数
     * @param headers 请求头
     * @return 返回值
     */
    public static HttpResult doPostJson(String url, String json, Map<String, String> headers) {
        log.info("Http post json请求URL:{}", url);
        log.info("Http post json请求参数:{}", json);
        // 创建http POST请求
        HttpPost httpPost = new HttpPost(url);

        httpPost.setHeader(new BasicHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString()));
        httpPost.setHeader(new BasicHeader(HttpHeaders.ACCEPT, "text/plain;charset=utf-8"));

        // 封装请求头
        packageHeader(headers, httpPost);
        if (json != null) {
            // 构造一个JSON请求的实体
            StringEntity stringEntity = new StringEntity(json, ContentType.APPLICATION_JSON);
            // 将请求实体设置到httpPost对象中
            httpPost.setEntity(stringEntity);
        }
        return execute(httpPost);
    }

    /**
     * http post stream请求
     *
     * @param url 请求地址
     * @param in  输入流
     * @return 返回数据
     */
    public static HttpResult doPostInputStream(String url, InputStream in) {
        // 创建http POST请求
        HttpPost httpPost = new HttpPost(url);
        httpPost.setHeader(new BasicHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_FORM_URLENCODED.toString()));
        httpPost.setHeader(new BasicHeader(HttpHeaders.ACCEPT, "text/plain;charset=utf-8"));

        if (in != null) {
            httpPost.setEntity(new InputStreamEntity(in));
        }

        return execute(httpPost);
    }

    /**
     * http post text请求
     *
     * @param url  请求地址
     * @param text 文本内容
     * @return 返回数据
     */
    public static HttpResult doPostWrite(String url, String text) {
        // 创建http POST请求
        HttpPost httpPost = new HttpPost(url);
        httpPost.setHeader(new BasicHeader(HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_FORM_URLENCODED.toString()));
        httpPost.setHeader(new BasicHeader(HttpHeaders.ACCEPT, "text/plain;charset=utf-8"));

        if (StringUtils.isNotBlank(text)) {
            StringEntity stringEntity = new StringEntity(text, ContentType.TEXT_PLAIN);
            httpPost.setEntity(stringEntity);
        }
        return execute(httpPost);
    }

    /**
     * 执行HTTP请求
     *
     * @param request {@link HttpRequestBase} 请求
     * @return {@link HttpResult} 请求结果
     */
    public static HttpResult execute(HttpRequestBase request) {
        // 执行http请求
        try (CloseableHttpResponse response = httpClient.execute(request)) {
            // 构建返回实体
            return new HttpResult(response.getStatusLine().getStatusCode(),
                    EntityUtils.toString(response.getEntity()));
        } catch (Exception e) {
            log.error("http 请求异常", e);
        }

        return null;
    }

    /**
     * 将请求参数处理为 NameValuePair
     *
     * @param params 请求参数Map
     * @return List<NameValuePair>
     */
    public static List<NameValuePair> convertParams2NVPS(Map<String, Object> params) {
        if (!params.isEmpty()) {
            List<NameValuePair> parameters = new ArrayList<>();
            params.forEach((key, value) -> parameters.add(new BasicNameValuePair(key, String.valueOf(value))));
            return parameters;
        }
        return Collections.emptyList();
    }

    /**
     * 封装请求头
     *
     * @param headers    请求头参数列表
     * @param httpMethod 请求方式
     */
    public static void packageHeader(Map<String, String> headers, HttpRequestBase httpMethod) {
        if (MapUtils.isNotEmpty(headers)) {
            Set<Entry<String, String>> entrySet = headers.entrySet();
            for (Entry<String, String> entry : entrySet) {
                // 设置请求头到 HttpRequestBase 对象中
                httpMethod.setHeader(entry.getKey(), entry.getValue());
            }
        }
    }

    /**
     * 封装请求参数
     *
     * @param params     请求参数
     * @param httpMethod 请求方式
     * @throws UnsupportedEncodingException 不支持字符编码异常
     */
    private static void packageParam(Map<String, Object> params, HttpEntityEnclosingRequest httpMethod)
            throws UnsupportedEncodingException {

        if (MapUtils.isNotEmpty(params)) {
            List<NameValuePair> nameValuePairs = convertParams2NVPS(params);
            httpMethod.setEntity(new UrlEncodedFormEntity(nameValuePairs, ENCODING));
        }
    }
}

7.5 使用示例

使用HttpClientUtils,不对所有的返回结果进操作,只封装请求相关,后续逻辑操作另外定义完成

   public static void testGet(String url) {
        
        HashMap<String, String> headers = new HashMap<>(2);

        // 设置请求头部编码
        headers.put("Content-Type", "application/x-www-form-urlencoded; charset=utf-8");
        // 设置返回编码
        headers.put("Accept", "text/plain;charset=utf-8");

        HttpResult httpResult = HttpClientUtils.doGet(url, headers, null);
        if (Objects.nonNull(httpResult)){
            if(httpResult.getStatus() == 200){
                System.out.println(httpResult.getStringEntity());
            }
        }
    }


7.6 自定义请求

支持自定义请求,包中全局共享一个HttpClient,使用者可以通过HttpClientUtils.httpClient获取到连接客户端,自定义实现请求。也可以自定义HttpRequestBase请求,通过HttpClientUtils.execute()执行。

// 方式一
CloseableHttpClient httpClient = HttpClientUtils.httpClient;
HttpGet httpGet = new HttpGet(url);
httpGet.setHeader("key","value");
try(CloseableHttpResponse response = httpClient.execute(httpGet);){
    // handle response
}catch (ClientProtocolException e) {
    e.printStackTrace();
} catch (IOException e) {
    e.printStackTrace();
}
// 方式二
HttpGet httpGet = new HttpGet(url);
httpGet.setHeader("key","value");
// 调用工具类的 execute
HttpResult result = HttpClientUtils.execute(httpGet);
// 获取请求状态嘛(为response的原生响应码)
if(result.getStatus() == 200){
    // 自定义的接口返回结果在stringEntity中
    System.out.println(result.getStringEntity());
}
转载自:https://juejin.cn/post/7078658461407379463
评论
请登录