LLM Document Extraction Benchmark 2026 - ChatGPT, Claude and Gemini on a 20-Page Bill of Lading

Parseur LLM Extraction Benchmark 2026. Tested in October 2026. Four tools, seven bills of lading, one 20-page document, 77 fields.

In the Parseur LLM Extraction Benchmark 2026, we gave ChatGPT, Claude, Gemini and Parseur the same 20-page bill of lading and asked for the same 11 fields. ChatGPT and Gemini returned none of the 47 container numbers, answering null as if the document had no containers, while every other field they returned was correct. Claude and Parseur extracted all 47 containers and correctly excluded a crossed-out container. Parseur also returned all 105 cargo lines. Across all seven test documents, Parseur scored 78 (77 fields plus one bonus point), Claude 77, ChatGPT 76 and Gemini 74. We checked every value against the printed document.

Key Takeaways

• Two of the most widely used AI chatbots silently dropped every container on a long document. On a 20-page bill of lading, ChatGPT (GPT-6) and Gemini (3.6 Flash) returned null for container numbers. The document lists 47.
• Short documents don't show the problem. On six one- or two-page bills of lading, ChatGPT and Claude extracted all 66 fields correctly, matching Parseur. The gap only appeared on the 20-page document.
• Totals were easy, detail was hard. All four tools found the shipment totals on the summary page (836 pallets, 544,564.1 kg). Only Claude and Parseur read the 16 rider pages that list the containers.
• A silent miss is worse than an error message. Neither ChatGPT nor Gemini signals in its output that data is missing. Every other field is correct, so the result looks complete.
• Only Parseur returned the cargo detail as structured data. With the field setup used for all seven documents, Parseur returned 105 cargo lines with container, seal, description, Harmonized System (HS) code, packages and weight, each matching the document.
• Chatbot output changes shape from one tool to the next. In the chat apps, the same prompt produced four different JSON (JavaScript Object Notation) structures, so each would need its own mapping before the data reaches a business system.

The 20-page bill of lading test, by the numbers

Why are long documents hard for AI chatbots?

Long documents are hard for AI chatbots because the data a business needs is often spread across many pages, while the summary sits in one place. A chatbot can answer correctly from the summary page and never read the details.

Bills of lading show this well. A consolidated ocean shipment can carry dozens of containers. The first page holds the parties, ports, vessel and voyage, and says the containers are listed on attached rider sheets. Those rider sheets run for many pages, with running subtotals at the bottom of each one. Real documents also carry corrections, such as a container that was shut out at the port and rolled to the next sailing.

For a logistics team, the container list is the document's purpose. It is what gets matched against bookings, customs entries and the transportation management system (TMS). A tool that returns the totals but not the containers has not done the job. For background on the document itself, see what a bill of lading is.

Which tool extracted the 20-page bill of lading most accurately?

In our October 2026 test, Claude and Parseur were the most accurate on the 20-page bill of lading, with all 11 fields correct and all 47 containers listed. ChatGPT and Gemini scored 10 of 11 because both returned no container numbers.

Source: Parseur LLM Extraction Benchmark 2026, tested in October 2026. The bonus point is explained under "How we tested".

All four tools got the bill of lading number, carrier, shipper, consignee, ports, vessel, voyage and both totals right. The only field that separated them was the one that required reading all 20 pages.

What went wrong on the 20-page document?

In our test, the errors on the 20-page bill of lading fell into three patterns: a silent null for data that exists, reading only the summary, and a crossed-out container that had to be left out.

1. Which chatbots returned no containers at all?

ChatGPT (GPT-6) and Gemini (3.6 Flash) both returned "container_number": null on the 20-page bill of lading. In the prompt, null was defined as "the value does not appear in the document". The document lists 47 shipped containers across 16 pages.

The same prompt worked on shorter documents. On six one- or two-page bills of lading, ChatGPT returned a list of container numbers whenever a document had more than one. Only the document length changed.

Every other field in both answers is correct, which makes the miss easy to overlook. Someone checking the output would see a complete-looking record with the right totals.

2. Did the chatbots read all 20 pages?

Gemini's answer cited only pages 1 and 18: the main bill of lading page and the cargo summary. Those two pages hold every field it got right. The 16 rider pages with the container list are not cited.

ChatGPT didn't cite pages, so we can't say which pages it used. Its result, correct totals and no containers, matches Gemini's.

3. Which tools handled the crossed-out container?

Claude and Parseur both excluded the shut-out container. On page 12, container CSLU3455388 is crossed out and marked "shut out, not loaded", with a note that it was rolled to the next sailing. Including it would add a container that never sailed.

Claude explained its reasoning and checked its totals against the running subtotals. Parseur returned 47 containers in document order, with the shut-out container left out of both the list and the totals.

What did Parseur return beyond the 11 fields?

Parseur returned all 105 cargo lines on the 20-page bill of lading, each with container number, seal number, container type, description of goods, HS code, number of packages and gross weight. We added up the lines for each container and all 47 matched the container subtotals printed on the rider sheets.

This used the same field setup as all seven documents. Parseur's setup includes a table for container and cargo lines, configured once before the test. The chatbots were asked for the 11 summary fields only.

How did the tools do on shorter bills of lading?

On six bills of lading of one or two pages, ChatGPT, Claude and Parseur extracted all 66 fields correctly. Gemini made two errors.

Source: Parseur LLM Extraction Benchmark 2026, tested in October 2026.

Gemini's two errors were both silent:

• BOL-03: it read the bill of lading number MDCG2026GOA03371 as "NDCG2026GOA03371". One wrong letter breaks matching against bookings and customs records.
• BOL-04: it returned the carrier's name together with the shipper's ("PACIFIC CREST SHIPPING CO., LTD. / SHENZHEN BRIGHTWAY ELECTRONICS…"), mixing two parties into one field.

Overall ranking: seven bills of lading

Across all seven documents and 77 fields, Parseur ranked first with 78 points (77 fields plus one bonus point for the cargo lines), followed by Claude with 77, ChatGPT with 76 and Gemini with 74.

Source: Parseur LLM Extraction Benchmark 2026, tested in October 2026.

On the 11 fields alone, Parseur and Claude tied at 77/77. The bonus point separates them, rewarding structured cargo detail rather than a better read of the 11 fields.

Which output can go straight into a business system?

Only Parseur returned the same structure for every document: the same field names, numbers as plain numbers, weight units normalized to KG or LB, dates in a standard format, and one row per cargo line. Each chatbot returned a different structure for the same prompt. Chatbots accessed through APIs can use a fixed output schema, but in the chat apps we tested, each model chose its structure for each answer.

Source: Parseur LLM Extraction Benchmark 2026, tested in October 2026.

For a one-off question, any of these formats is readable. For documents that arrive every day and feed a TMS, an ERP or a spreadsheet, a structure that changes between answers means rebuilding the mapping every time.

When is a chatbot enough, and when do you need intelligent document processing?

A chatbot is enough to extract data from a single short document that someone will check by hand. Our results show that ChatGPT and Claude are accurate on one- or two-page bills of lading. For more on where LLMs hold up and where they don't, see our guide to LLM document processing.

An intelligent document processing (IDP) platform is the better fit when:

• Documents arrive continuously from many shippers, carriers and forwarders, each with its own layout.
• Documents are long, with detail spread across rider sheets, attachments or continuation pages.
• Data must land in a system such as a TMS, an ERP or a spreadsheet, in the same structure every time.
• Nobody can check every field by hand, so a silent null would go unnoticed.

Parseur is an intelligent document processing platform for operations teams that receive documents from many senders and need the data in their own systems. You define the fields once, then every document sent by email, upload or API is processed with the same setup and delivered through integrations, webhooks or the document extraction API. It has run in production since 2016, has processed more than 100 million documents, is hosted in the EU and is SOC 2 Type II compliant. For bills of lading specifically, see bill of lading OCR and logistics automation.

Tool-by-tool results

Parseur: 78 (77 of 77 fields, plus 1 bonus point). All fields correct on all seven documents. On the 20-page bill of lading, it returned all 47 containers in document order, excluded the shut-out container, and returned 105 cargo lines that matched the document's subtotals. It converted European number formats to plain numbers and labeled pounds and kilograms separately. Processing the 20-page document took about 23 seconds.

Claude (Sonnet 5.5): 77 of 77. All fields correct on all seven documents, including all 47 containers on the 20-page document. It explained its choices, such as excluding the shut-out container and checking totals against the running subtotals. It used its own JSON structure, with value and unit objects and an "as printed" copy of each number.

ChatGPT (GPT-6): 76 of 77. All 66 fields correct on the six short documents, including container lists on multi-container documents. On the 20-page document, it returned null for container numbers, with every other field correct, and returned package count and weight as bare numbers without units.

Gemini (3.6 Flash): 74 of 77. Two silent errors on the short documents: one wrong letter in a bill of lading number, and the carrier mixed into the shipper field. On the 20-page document, it returned null for container numbers and cited only pages 1 and 18. It returned values as printed text, such as "38.240,500 KGS", rather than as numbers.

How we tested

We tested four tools on the same seven bills of lading between 13 and 15 October 2026, using one fixed list of 11 fields, and checked every returned value against an answer key built from the printed documents.

Which documents did we use?

We used seven synthetic bills of lading, not real customer documents. We built them to reproduce real layouts from ocean carriers, US trucking and European forwarders, including the traps that cause errors in practice: multiple containers, continuation pages, distractor parties, local number formats, a phone photo and a handwritten value. Synthetic documents mean every value is known in advance, and the documents can be shared without exposing customer data.

The documents use three languages: English, Italian with English labels, and German with English labels. All container numbers follow the ISO 6346 standard with valid check digits, and every subtotal and total on the documents adds up.

The pages below show four of the harder cases.

Page 1 of BOL-07 holds the header fields. The cargo box gives no container numbers. It refers to the rider sheets on pages 2 to 17 and the summary on page 18.

Page 12 of BOL-07 is one of 16 rider sheets. Container CSLU3455388 is crossed out as shut out, and the carried-forward subtotal at the bottom excludes it.

BOL-04 is a phone photo of a printed bill of lading, with a handwritten voyage number (118N) and stamps over the form.

BOL-03 uses Italian and English labels, European number formats and day/month dates, with the totals for two containers on one line.

How did we run the test?

1. Fixed the fields. We chose 11 fields that logistics teams use from every bill of lading: bill of lading number, carrier, shipper, consignee, port of loading, port of discharge, container number, vessel name, voyage number, package count and gross weight.
2. Built the answer key from the printed values on each document before running any tool.
3. Ran the chatbots in a new chat each, with the same prompt and the same files, and saved every answer as text and screenshots. We uploaded the six short documents together and the 20-page document on its own.
4. Ran Parseur by uploading all seven documents to one mailbox with one field setup, and exported the results.
5. Scored every field against the answer key using the rules below, and checked each error against the original document and the tool's screenshot. Each document was scored on the same 11 fields: 66 fields across the six short documents and 11 on the 20-page document, for 77 total.

Tools and setup

• ChatGPT (GPT-6), Claude (Sonnet 5.5) and Gemini (3.6 Flash) in their chat apps on their free plans, each document set in a new chat, one run per tool.
• The same prompt for every chatbot, asking for 11 fields as JSON. The prompt said to return null for any value that does not appear in the document and not to infer missing information. The 20-page document was uploaded on its own.
• Parseur with one mailbox and one field setup, configured before the final run and used unchanged for all seven documents. The setup covers the 11 data points, plus a table for container and cargo lines and fields for dates and weight units.

Scoring rules

1. The ground truth is the value printed on the document.
2. Formatting differences were not errors: "22 PLT" and "22 pallets", "Genova" and "Genoa", or a weight with or without its unit.
3. A field not on the document had to come back empty. Filling in a value where the document has none counted as an error.
4. Container numbers were scored as one field: correct only if every shipped container was listed and the shut-out container was left out.
5. We took Parseur's package count as the sum of its container rows, which matched the printed totals on every document.
6. On the truck bill of lading, Parseur's container field was set up to capture the trailer number (53-11872). We counted it as correct because the field was defined that way. Scored strictly against the chatbot prompt, Parseur's total would be one point lower.
7. Bonus point: Parseur received one bonus point on the 20-page document for returning all 105 cargo lines, each verified against the document. The chatbots weren't asked for cargo lines, so we show the bonus separately from the 77-field score.

Limitations

This is a small hands-on test, not a statistical benchmark, and Parseur ran it as one of the tools tested. Each chatbot was run once per document set, and results from chat apps can vary between runs and model updates. The documents are synthetic, built to reproduce real layouts.

How to test an LLM or extraction tool on your own documents

The container miss in this test only showed up because we checked the output against the document. Before relying on a tool, run your own documents through it and check:

1. Your longest documents, not just one-page samples. Include rider sheets, attachments and continuation pages.
2. Every repeated item, such as containers, line items or shipments. Count them against the document.
3. That an empty or null value really means absent. A missing value that exists in the document is the hardest error to spot.
4. Corrections on the document, such as crossed-out or shut-out items, which should be left out.
5. That the output parses and keeps the same structure across documents, so your integration doesn't break.
6. That parties aren't mixed up: shipper, consignee, notify party, forwarder and carrier.

For a related test on invoices in four languages, see the Multilingual Invoice Extraction Benchmark 2026. For dedicated bill of lading tools, see our comparison of the best bill of lading processing software.

Test it on your own bills of lading

Seven documents show where tools break, but your shipping documents are the real test. Run your longest bills of lading through Parseur and check the container list against the original.

Last updated on October 20th, 2026