llama.cpp inference

[s0569591]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

Please provide a detailed written description of what you were trying to do, and what you expected llama.cpp to do as an enhancement.

I want to inference llama.cpp (model: 7B-chat) locally with a dataset from huggingface (e.g. wmt16 Link: https://huggingface.co/datasets/wmt16).
How can i accomplish this?
I could not find any special answers on the internet.

Motivation

Please provide a detailed written description of reasons why this feature is necessary and how it is useful to llama.cpp users.

I needed for a university project, where i have to measure the power consumption of Llama2 on a local machine.

Possible Implementation

I am currently trying to learn langchain to create an inference file.

If you have an idea as to how it can be implemented, please write a detailed description. Feel free to give links to external sources or share visuals that might be helpful to understand the details better.

[y10ab1]

You can run llama.cpp in server mode in the background and perform inference on your dataset by calling the API using a Python script. I believe this is the easiest way to accomplish the task.