信息和文档 - WARC文件格式

[中文适用范围]： 本文件规定了WARC文件格式： ——存储来自主流互联网应用层协议（如HTTP、DNS和FTP）的负载内容和控制信息； ——存储与其他存储数据链接的任意元数据（例如主题分类器、发现的语言、编码）； ——支持数据压缩并保持数据记录的完整性； ——存储来自收集协议的所有控制信息（例如请求标头），而不仅仅是响应信息； ——存储链接到其他存储数据的数据转换结果； ——存储与其他存储数据链接的重复检测事件（以减少存在相同或基本相似资源的情况下的存储）； ——在不中断现有功能的情况下进行扩展； ——在需要时支持通过截断或分段来处理过长的记录。 [外文原描述]： ISO 28500:2017 specifies the WARC file format: - to store both the payload content and control information from mainstream Internet application layer protocols, such as the HTTP, DNS, and FTP; - to store arbitrary metadata linked to other stored data (e.g. subject classifier, discovered language, encoding); - to support data compression and maintain data record integrity; - to store all control information from the harvesting protocol (e.g. request headers), not just response information; - to store the results of data transformations linked to other stored data; - to store a duplicate detection event linked to other stored data (to reduce storage in the presence of identical or substantially similar resources); - to be extended without disruption to existing functionality; - to support handling of overly long records by truncation or segmentation, where desired.