Palimpzest (PZ)

Learn How to Use PZ

Our full documentation is the definitive resource for learning how to use PZ. It contains all of the installation and quickstart materials on this page, as well as user guides, full API documentation, and much more.

Getting started

You can find a stable version of the PZ package on PyPI here. To install the package, run:

$ pip install palimpzest

Alternatively, to install the latest version of the package from this repository, you can clone this repository and run the following commands:

$ git clone git@github.com:mitdbg/palimpzest.git
$ cd palimpzest
$ pip install .

Join the PZ Community

We are actively hacking on PZ and would love to have you join our community

Our Discord server is the best place to:

• Get help with your PZ program(s)
• Give feedback to the maintainers
• Discuss the future direction(s) of the project
• Discuss anything related to data processing with LLMs!

We are eager to learn more about your workloads and use cases, and will take them into consideration in planning our future roadmap.

Quick Start

The easiest way to get started with Palimpzest is to run the quickstart.ipynb jupyter notebook. We demonstrate the full workflow of working with PZ, including registering a dataset, composing and executing a pipeline, and accessing the results. To run the notebook, you can use the following command:

$ jupyter notebook

And then access the notebook from the jupyter interface in your browser at localhost:8888.

Even Quicker Start

For eager readers, the code in the notebook can be found in the following condensed snippet. However, we do suggest reading the notebook as it contains more insight into each element of the program.

import palimpzest as pz

# define the fields we wish to compute
email_cols = [
    {"name": "sender", "type": str, "desc": "The email address of the sender"},
    {"name": "subject", "type": str, "desc": "The subject of the email"},
    {"name": "date", "type": str, "desc": "The date the email was sent"},
]

# lazily construct the computation to get emails about holidays sent in July
dataset = pz.Dataset("testdata/enron-tiny/")
dataset = dataset.sem_add_columns(email_cols)
dataset = dataset.sem_filter("The email was sent in July")
dataset = dataset.sem_filter("The email is about holidays")

# execute the computation w/the MinCost policy
config = pz.QueryProcessorConfig(policy=pz.MinCost(), verbose=True)
output = dataset.run(config)

# display output (if using Jupyter, otherwise use print(output_df))
output_df = output.to_df(cols=["date", "sender", "subject"])
display(output_df)

Python Demos

Below are simple instructions to run PZ on a test data set of enron emails that is included with the system.

Downloading test data

To run the provided demos, you will need to download the test data. Due to the size of the data, we are unable to include it in the repository. You can download the test data by running the following command from a unix terminal (requires wget and tar):

chmod +x testdata/download-testdata.sh
./testdata/download-testdata.sh

Running the Demos

Set your OpenAI (or Together.ai) api key at the command line:

# set one (or both) of the following:
export OPENAI_API_KEY=<your-api-key>
export TOGETHER_API_KEY=<your-api-key>

Now you can run the simple test program with:

$ python demos/simple-demo.py --task enron --dataset testdata/enron-eval-tiny --verbose