将网站内容读取为字符串

url java networking io

2022-09-03 00:34:37

目前，我正在开发一个可用于读取由url指定的网站内容的类。我刚刚开始我的冒险，所以我需要咨询我的设计。java.iojava.net

用法：

TextURL url = new TextURL(urlString);
String contents = url.read();

我的代码：

package pl.maciejziarko.util;

import java.io.*;
import java.net.*;

public final class TextURL
{
    private static final int BUFFER_SIZE = 1024 * 10;
    private static final int ZERO = 0;
    private final byte[] dataBuffer = new byte[BUFFER_SIZE];
    private final URL urlObject;

    public TextURL(String urlString) throws MalformedURLException
    {
        this.urlObject = new URL(urlString);
    }

    public String read() 
    {
        final StringBuilder sb = new StringBuilder();

        try
        {
            final BufferedInputStream in =
                    new BufferedInputStream(urlObject.openStream());

            int bytesRead = ZERO;

            while ((bytesRead = in.read(dataBuffer, ZERO, BUFFER_SIZE)) >= ZERO)
            {
                sb.append(new String(dataBuffer, ZERO, bytesRead));
            }
        }
        catch (UnknownHostException e)
        {
            return null;
        }
        catch (IOException e)
        {
            return null;
        }

        return sb.toString();
    }

    //Usage:
    public static void main(String[] args)
    {
        try
        {
            TextURL url = new TextURL("http://www.flickr.com/explore/interesting/7days/");
            String contents = url.read();

            if (contents != null)
                System.out.println(contents);
            else
                System.out.println("ERROR!");
        }
        catch (MalformedURLException e)
        {
            System.out.println("Check you the url!");
        }
    }
}

我的问题是：这是实现我想要的东西的好方法吗？有没有更好的解决方案？

我特别不喜欢，但我无法以不同的方式表达它。每次迭代都创建一个新的字符串好吗？我想不是。sb.append(new String(dataBuffer, ZERO, bytesRead));

还有其他弱点吗？

提前致谢！

答案 1

请考虑改用 URLConnection。此外，您可能希望利用Apache Commons IO中的IOUtils来使字符串读取更容易。例如：

URL url = new URL("http://www.example.com/");
URLConnection con = url.openConnection();
InputStream in = con.getInputStream();
String encoding = con.getContentEncoding();  // ** WRONG: should use "con.getContentType()" instead but it returns something like "text/html; charset=UTF-8" so this value must be parsed to extract the actual encoding
encoding = encoding == null ? "UTF-8" : encoding;
String body = IOUtils.toString(in, encoding);
System.out.println(body);

如果你不想使用，我可能会重写上面的那一行，比如：IOUtils

ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[8192];
int len = 0;
while ((len = in.read(buf)) != -1) {
    baos.write(buf, 0, len);
}
String body = new String(baos.toByteArray(), encoding);

答案 2

我强烈建议使用专用库，如HtmlParser：

Parser parser = new Parser (url);
NodeList list = parser.parse (null);
System.out.println (list.toHtml ());

编写自己的html解析器是如此宽松的时间。这是它的 maven 依赖项。看看它的JavaDoc来挖掘它的功能。

看看下面的示例应该令人信服：

Parser parser = new Parser(url);
NodeList movies = parser.extractAllNodesThatMatch(
    new AndFilter(new TagNameFilter("div"),
    new HasAttributeFilter("class", "movie")));