本页尚未翻译

本页内容仍为英文原文，源自官方文档 integrations/ai/synthetic-data-kit。中文翻译正在分批进行中。

Meta Llama Synthetic Data Kit

Use Meta Llama's Synthetic Data Kit with LanceDB to generate high-quality synthetic datasets for LLM fine-tuning and training.

Synthetic Data Kit is a tool from Meta LLAMA that helps you generate high-quality synthetic datasets for fine-tuning large language models (LLMs). It simplifies the process of preparing data for fine-tuning by providing a command-line interface (CLI) with a modular four-command flow.

One of the key features of the synthetic-data-kit is its use of the Lance format for storing and ingesting datasets. This allows for efficient storage and retrieval of data, which is crucial when working with large datasets.

Key Features:

• Data Ingestion: The toolkit can ingest various file formats, including PDF, HTML, YouTube transcripts, DOCX, PPT, and TXT.
• Fine-tuning Format Creation: It can create different fine-tuning formats, such as question-answer (QA) pairs, QA pairs with Chain-of-Thought (CoT), and summarization formats.
• Data Curation: The tool uses Llama as a judge to curate high-quality examples, ensuring the quality of the generated dataset.
• Flexible Saving Options: You can save the generated datasets in various formats compatible with your fine-tuning workflow, including HuggingFace, JSONL, and JSON.

How it Works:

The synthetic-data-kit follows a simple four-step process:

1. Ingest: Import your input files into the toolkit. The data is stored in the Lance format for efficient processing.
2. Create: Generate diverse fine-tuning datasets, such as reasoning, summarization, and QA pairs, from the ingested documents.
3. Curate: Use Llama to filter and select high-quality examples from the generated dataset.
4. Save-as: Export the curated dataset in your preferred format.

Usage

The synthetic-data-kit uses Lance format to store and manage the data that you ingest. The workflow is a series of commands that build on each other, starting with the ingest command.

Here is an example of the end-to-end workflow:

1. Ingest Data into a LanceDB dataset

   This command reads a source document and creates a Lance dataset from it.

   synthetic-data-kit ingest docs/report.pdf --multimodal
   # This will create a Lance dataset at data/parsed/report.lance
   # with 'text' and 'image' columns.

2. Create fine-tuning data

   This command uses the Lance dataset created in the previous step to generate multimodal question-answer pairs.

   synthetic-data-kit create data/parsed/report.lance --type multimodal-qa
   # This writes data/generated/report.json

3. Curate the data

   This step uses a language model to curate the generated data and ensure its quality.

   synthetic-data-kit curate data/generated/report.json
   # This writes data/curated/report_cleaned.json

4. Save the final dataset

   Finally, save the curated data in the format you need: jsonl, alpaca, ft or chatml.

   synthetic-data-kit save-as data/curated/report_cleaned.json -f jsonl -o ./my_finetuning_data.jsonl

This workflow allows you to go from a collection of documents to a high-quality, fine-tuning dataset with just a few commands. The use of LanceDB in the background makes the process efficient and scalable.

Getting Started:

To get started with the synthetic-data-kit, you can clone the GitHub Repository and install the necessary dependencies.

Note: You will also need access to a Llama model, either running locally or via a hosted API.