Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 0.6.2 #593

Merged
merged 7 commits into from
Aug 21, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Update README.md (#579)
According to new docs, which I also linked from the sidebar for increased discoverability.
  • Loading branch information
philipkiely-baseten authored Aug 17, 2023
commit 073ef84a9170262056a987a8f47b3833e2805806
82 changes: 60 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -13,10 +13,9 @@

See Trusses for popular models including:

* 🦅 [Falcon 40B](https://github.com/basetenlabs/falcon-40b-truss)
* 🧙 [WizardLM](https://github.com/basetenlabs/wizardlm-truss)
* 🎨 [Stable Diffusion](https://github.com/basetenlabs/stable-diffusion-truss)
* 🗣 [Whisper](https://github.com/basetenlabs/whisper-truss)
* 🦙 [Llama 2 7B](https://github.com/basetenlabs/truss-examples/tree/main/llama-2-7b-chat) ([13B](https://github.com/basetenlabs/truss-examples/tree/main/llama-2-13b-chat)) ([70B](https://github.com/basetenlabs/truss-examples/tree/main/llama-2-70b-chat))
* 🎨 [Stable Diffusion XL](https://github.com/basetenlabs/truss-examples/tree/main/stable-diffusion-xl-1.0)
* 🗣 [Whisper](https://github.com/basetenlabs/truss-examples/tree/main/whisper-truss)

and [dozens more examples](examples/).

@@ -36,61 +35,100 @@ As a quick example, we'll package a [text classification pipeline](https://huggi

To get started, create a Truss with the following terminal command:

```
```sh
truss init text-classification
```

This will create an empty Truss at `./text-classification`.
When prompted, give your Truss a name like `Text classification`.

Then, navigate to the newly created directory:

```sh
cd text-classification
```

### Implement the model

The model serving code goes in `./text-classification/model/model.py` in your newly created Truss.
One of the two essential files in a Truss is `model/model.py`. In this file, you write a `Model` class: an interface between the ML model that you're packaging and the model server that you're running it on.

There are two member functions that you must implement in the `Model` class:

* `load()` loads the model onto the model server. It runs exactly once when the model server is spun up or patched.
* `predict()` handles model inference. It runs every time the model server is called.

Here's the complete `model/model.py` for the text classification model:

```python
from typing import List
from transformers import pipeline


class Model:
def __init__(self, **kwargs) -> None:
def __init__(self, **kwargs):
self._model = None

def load(self):
self._model = pipeline("text-classification")

def predict(self, model_input: str) -> List:
def predict(self, model_input):
return self._model(model_input)
```

There are two functions to implement:

* `load()` runs once when the model is spun up and is responsible for initializing `self._model`
* `predict()` runs each time the model is invoked and handles the inference. It can use any JSON-serializable type as input and output.

### Add model dependencies

The pipeline model relies on Transformers and PyTorch. These dependencies must be specified in the Truss config.
The other essential file in a Truss is `config.yaml`, which configures the model serving environment. For a complete list of the config options, see [the config reference](https://truss.baseten.co/reference/config).

In `./text-classification/config.yaml`, find the line `requirements`. Replace the empty list with:
The pipeline model relies on [Transformers](https://huggingface.co/docs/transformers/index) and [PyTorch](https://pytorch.org/). These dependencies must be specified in the Truss config.

In `config.yaml`, find the line `requirements`. Replace the empty list with:

```yaml
requirements:
- torch==2.0.1
- transformers==4.30.0
```

No other configuration needs to be changed.
No other configuration is needed.

## Deployment

You can deploy a Truss to your [Baseten](https://baseten.co) account with:
Truss is maintained by [Baseten](https://baseten.co), which provides infrastructure for running ML models in production. We'll use Baseten as the remote host for your model.

```
cd ./text-classification
Other remotes are coming soon, starting with AWS SageMaker.

### Get an API key

To set up the Baseten remote, you'll need a [Baseten API key](https://app.baseten.co/settings/account/api_keys). If you don't have a Baseten account, no worries, just [sign up for an account](https://app.baseten.co/signup/) and you'll be issued plenty of free credits to get you started.

### Run `truss push`

With your Baseten API key ready to paste when prompted, you can deploy your model:

```sh
truss push
```

Truss will support other remotes soon, starting with AWS SageMaker.
You can monitor your model deployment from [your model dashboard on Baseten](https://app.baseten.co/models/).

### Invoke the model

After the model has finished deploying, you can invoke it from the terminal.

**Invocation**

```sh
truss predict -d '"Truss is awesome!"'
```

**Response**

```json
[
{
"label": "POSITIVE",
"score": 0.999873161315918
}
]
```

## Truss contributors