Skip to content

Commit f93001d

Browse files
baberabbbaberabb
authored
delete neuralmagic models (#3112)
1 parent e69ca5e commit f93001d

File tree

7 files changed

+6
-502
lines changed

7 files changed

+6
-502
lines changed

‎.github/workflows/unit_tests.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ jobs:
6868
pip install hf_xet
6969
7070
- name: Test with pytest
71-
run: python -m pytest --showlocals -s -vv -n=auto --ignore=tests/models/test_neuralmagic.py --ignore=tests/models/test_openvino.py --ignore=tests/models/test_hf_steered.py
71+
run: python -m pytest --showlocals -s -vv -n=auto --ignore=tests/models/test_openvino.py --ignore=tests/models/test_hf_steered.py
7272
continue-on-error: true # Continue workflow even if tests fail
7373

7474
# Save test artifacts
@@ -106,7 +106,7 @@ jobs:
106106
# - name: Install dependencies
107107
# run: |
108108
# python -m pip install --upgrade pip
109-
# pip install -e '.[dev,optimum,deepsparse,sparseml,api]' --extra-index-url https://download.pytorch.org/whl/cpu
109+
# pip install -e '.[dev,optimum,api]' --extra-index-url https://download.pytorch.org/whl/cpu
110110
# pip install -U transformers peft accelerate
111111
#
112112
# - name: Test with pytest

‎README.md‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -364,7 +364,7 @@ lm_eval --model local-completions --tasks gsm8k --model_args model=facebook/opt-
364364
Note that for externally hosted models, configs such as `--device` which relate to where to place a local model should not be used and do not function. Just like you can use `--model_args` to pass arbitrary arguments to the model constructor for local models, you can use it to pass arbitrary arguments to the model API for hosted models. See the documentation of the hosting service for information on what arguments they support.
365365

366366
| API or Inference Server | Implemented? | `--model <xxx>` name | Models supported: | Request Types: |
367-
| --------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
367+
|---------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
368368
| OpenAI Completions | :heavy_check_mark: | `openai-completions`, `local-completions` | All OpenAI Completions API models | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
369369
| OpenAI ChatCompletions | :heavy_check_mark: | `openai-chat-completions`, `local-chat-completions` | [All ChatCompletions API models](https://platform.openai.com/docs/guides/gpt) | `generate_until` (no logprobs) |
370370
| Anthropic | :heavy_check_mark: | `anthropic` | [Supported Anthropic Engines](https://docs.anthropic.com/claude/reference/selecting-a-model) | `generate_until` (no logprobs) |
@@ -377,8 +377,6 @@ Note that for externally hosted models, configs such as `--device` which relate
377377
| Huggingface Optimum (Causal LMs) | :heavy_check_mark: | `openvino` | Any decoder-only AutoModelForCausalLM converted with Huggingface Optimum into OpenVINO™ Intermediate Representation (IR) format | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
378378
| Huggingface Optimum-intel IPEX (Causal LMs) | :heavy_check_mark: | `ipex` | Any decoder-only AutoModelForCausalLM | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
379379
| Neuron via AWS Inf2 (Causal LMs) | :heavy_check_mark: | `neuronx` | Any decoder-only AutoModelForCausalLM supported to run on [huggingface-ami image for inferentia2](https://aws.amazon.com/marketplace/pp/prodview-gr3e6yiscria2) | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
380-
| [Neural Magic DeepSparse](https://github.com/neuralmagic/deepsparse) | :heavy_check_mark: | `deepsparse` | Any LM from [SparseZoo](https://sparsezoo.neuralmagic.com/) or on [HF Hub with the "deepsparse" tag](https://huggingface.co/models?other=deepsparse) | `generate_until`, `loglikelihood` |
381-
| [Neural Magic SparseML](https://github.com/neuralmagic/sparseml) | :heavy_check_mark: | `sparseml` | Any decoder-only AutoModelForCausalLM from [SparseZoo](https://sparsezoo.neuralmagic.com/) or on [HF Hub](https://huggingface.co/neuralmagic). Especially useful for models with quantization like [`zoo:llama2-7b-gsm8k_llama2_pretrain-pruned60_quantized`](https://sparsezoo.neuralmagic.com/models/llama2-7b-gsm8k_llama2_pretrain-pruned60_quantized) | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
382380
| NVIDIA NeMo | :heavy_check_mark: | `nemo_lm` | [All supported models](https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/core/core.html#nemo-models) | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
383381
| Watsonx.ai | :heavy_check_mark: | `watsonx_llm` | [Supported Watsonx.ai Engines](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx) | `generate_until` `loglikelihood` |
384382
| [Your local inference server!](docs/API_guide.md) | :heavy_check_mark: | `local-completions` or `local-chat-completions` | Support for OpenAI API-compatible servers, with easy customization for other APIs. | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
@@ -613,7 +611,7 @@ Extras dependencies can be installed via `pip install -e ".[NAME]"`
613611
|----------------------|--------------------------------|----------------|---------------------------------------|
614612
| tasks | All task-specific dependencies | api | API models (Anthropic, OpenAI, local) |
615613
| acpbench | ACP Bench tasks | audiolm_qwen | Qwen2 audio models |
616-
| ifeval | IFEval task | deepsparse | DeepSparse models (CPU) |
614+
| ifeval | IFEval task | | |
617615
| japanese_leaderboard | Japanese LLM tasks | gptq | AutoGPTQ models |
618616
| longbench | LongBench tasks | gptqmodel | GPTQModel models |
619617
| math | Math answer checking | hf_transfer | Speed up HF downloads |
@@ -624,7 +622,7 @@ Extras dependencies can be installed via `pip install -e ".[NAME]"`
624622
| promptsource | PromptSource prompts | neuronx | AWS inf2 instances |
625623
| sentencepiece | Sentencepiece tokenizer | optimum | Intel OpenVINO models |
626624
| testing | Run test suite | sae_lens | SAELens model steering |
627-
| unitxt | Run unitxt tasks | sparseml | SparseML models (CPU) |
625+
| unitxt | Run unitxt tasks | | |
628626
| wandb | Weights & Biases | sparsify | Sparsify model steering |
629627
| zeno | Result visualization | vllm | vLLM models |
630628

‎docs/CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ in order to ensure linters and other checks will be run upon committing.
3030
We use [pytest](https://docs.pytest.org/en/latest/) for running unit tests. All library unit tests can be run via:
3131

3232
```bash
33-
python -m pytest --showlocals -s -vv -n=auto --ignore=tests/models/test_neuralmagic.py --ignore=tests/models/test_openvino.py
33+
python -m pytest --showlocals -s -vv -n=auto --ignore=tests/models/test_openvino.py
3434
```
3535

3636
## Contributor License Agreement

‎lm_eval/models/__init__.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010
ibm_watsonx_ai,
1111
mamba_lm,
1212
nemo_lm,
13-
neuralmagic,
1413
neuron_optimum,
1514
openai_completions,
1615
optimum_ipex,

0 commit comments

Comments
 (0)