Skip to content

Commit f7f062f

Browse files
vsochvsoch
committed
hello world: the project not named fractale
This is a rough draft to create a subsystem database and query it. Everything needs to be improved, but it is a good start. Signed-off-by: vsoch <[email protected]>
0 parents  commit f7f062f

28 files changed

+1512
-0
lines changed

‎.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
fractale.egg-info
2+
build
3+
dist
4+
env
5+
.eggs
6+
__pycache__

‎.pre-commit-config.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
exclude: "examples"
2+
repos:
3+
- repo: https://github.com/pre-commit/pre-commit-hooks
4+
rev: v4.3.0
5+
hooks:
6+
- id: check-added-large-files
7+
args: ["--maxkb=2000"]
8+
- id: check-case-conflict
9+
- id: check-docstring-first
10+
- id: end-of-file-fixer
11+
- id: trailing-whitespace
12+
- id: mixed-line-ending
13+
14+
- repo: local
15+
hooks:
16+
- id: black
17+
name: black
18+
language: python
19+
types: [python]
20+
entry: black
21+
22+
- id: isort
23+
name: isort
24+
args: [--filter-files]
25+
language: python
26+
types: [python]
27+
entry: isort

‎COPYRIGHT

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Intellectual Property Notice
2+
----------------------------
3+
4+
HPCIC DevTools is licensed under the MIT license (LICENSE).
5+
6+
Copyrights and patents in this project are retained by
7+
contributors. No copyright assignment is required to contribute to
8+
HPCIC DevTools.
9+
10+
SPDX usage
11+
------------
12+
13+
Individual files contain SPDX tags instead of the full license text.
14+
This enables machine processing of license information based on the SPDX
15+
License Identifiers that are available here: https://spdx.org/licenses/

‎LICENSE

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2022-2023 LLNS, LLC and other HPCIC DevTools Developers.
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎MANIFEST.in

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
recursive-include fractale *
2+
prune env*
3+
global-exclude .env
4+
global-exclude *.py[co]
5+
recursive-exclude .git *
6+
global-exclude __pycache__

‎NOTICE

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
This work was produced under the auspices of the U.S. Department of
2+
Energy by Lawrence Livermore National Laboratory under Contract
3+
DE-AC52-07NA27344.
4+
5+
This work was prepared as an account of work sponsored by an agency of
6+
the United States Government. Neither the United States Government nor
7+
Lawrence Livermore National Security, LLC, nor any of their employees
8+
makes any warranty, expressed or implied, or assumes any legal liability
9+
or responsibility for the accuracy, completeness, or usefulness of any
10+
information, apparatus, product, or process disclosed, or represents that
11+
its use would not infringe privately owned rights.
12+
13+
Reference herein to any specific commercial product, process, or service
14+
by trade name, trademark, manufacturer, or otherwise does not necessarily
15+
constitute or imply its endorsement, recommendation, or favoring by the
16+
United States Government or Lawrence Livermore National Security, LLC.
17+
18+
The views and opinions of authors expressed herein do not necessarily
19+
state or reflect those of the United States Government or Lawrence
20+
Livermore National Security, LLC, and shall not be used for advertising
21+
or product endorsement purposes.

‎README.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# fractale
2+
3+
> Translation layer for a jobspec specification to cluster execution
4+
5+
[![PyPI version](https://badge.fury.io/py/fractale.svg)](https://badge.fury.io/py/fractale)
6+
[![DOI](https://zenodo.org/badge/773568660.svg)](https://zenodo.org/doi/10.5281/zenodo.13787066)
7+
8+
This library is primarily being used for development for the descriptive thrust of the Fractale project. It is called fractale, but also not called fractale. You can't be sure of the name until you open the box.
9+
10+
## Design
11+
12+
We want to:
13+
14+
1. Generate software graphs for some cluster (fluxion JGF) (this is done with [compspec](https://github.com/compspec/compspec)
15+
2. Register N clusters to a tool (should be written as a python module)
16+
3. Tool would have ability to select clusters from resources known, return
17+
4. Need graphical reprsentation (json) of each cluster - this will be used with the LLM inference
18+
19+
See [examples/fractale](examples/fractale) for a detailed walk-through of the above.
20+
21+
<!-- ⭐️ [Documentation](https://compspec.github.io/fractale) ⭐️ -->
22+
23+
## License
24+
25+
HPCIC DevTools is distributed under the terms of the MIT license.
26+
All new contributions must be made under this license.
27+
28+
See [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),
29+
[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and
30+
[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.
31+
32+
SPDX-License-Identifier: (MIT)
33+
34+
LLNL-CODE- 842614

‎examples/fractale/README.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Fractale Workflow
2+
3+
This example will walk through the steps to generate local compatibility graphs to describe a user-space, and then make a request to generate a jobspec and match.
4+
5+
## Install Fractale
6+
7+
You can pip install, or install from GitHub.
8+
9+
```bash
10+
pip install fractale
11+
12+
# or
13+
git clone https://github.com/compspec/fractale
14+
cd fractale
15+
pip install -e .
16+
```
17+
18+
## Local Subsystems
19+
20+
A local subsystem is typically a user-space install of software or other metadata that is associated with a cluster. Local subsystems can be defined for one or more clusters, and all provided to fractale to determine with of the clusters can support the work. For each cluster, the user will use fractale and (via the [compspec](https://github.com/compspec/compspec) library and plugins) generate one or more subsystem graphs associated with different clusters. Let's start with generating metadata for clusters A and B:
21+
22+
```bash
23+
fractale generate --cluster A spack /home/vanessa/Desktop/Code/spack
24+
```
25+
26+
## Satisfy Request
27+
28+
This is the step where we want to say "Run gromacs on 2-4 nodes with these requirements." Since we haven't formalized a way to do that, I'm going to start with a flux jobspec, and then add attributes that can be used to search our subsystems. For example, I generated [software-gromacs.json](software-gromacs.json) with:
29+
30+
```bash
31+
flux submit --dry-run --setattr=requires.software=spack:gromacs gmx | jq
32+
```
33+
34+
Note that the important part of that is:
35+
36+
```json
37+
"attributes": {
38+
"system": {
39+
"requires": {
40+
"software": "gromacs"
41+
}
42+
}
43+
},
44+
```
45+
46+
This is going to say "search subsystems that are of type software looking for gromacs." In practice, this looks for root nodes where the type is "software," which would be the case for spack or environment modules. This file can be json or yaml. Then we ask to satisfy. Either of these will work:
47+
48+
```bash
49+
fractale satisfy ./examples/fractale/software-gromacs.yaml
50+
fractale satisfy ./examples/fractale/software-gromacs.json
51+
```
52+
53+
By default, the above assumes subsystems located in the fractale home. If you want to adjust that, set `fractale --config-dir=<path> satisfy...` to adjust that (and note you will need to have generated the tree here. What we basically do with satisfy is build a database with tables for:
54+
55+
- clusters
56+
- subsystems
57+
- nodes
58+
- attributes
59+
60+
And right now the search is just over attributes to find matching clusters. E.g., here are attributes for a spack package:
61+
62+
```json
63+
"attributes": {
64+
"name": "py-ply",
65+
"version": "3.11",
66+
"platform": "linux",
67+
"target": "zen4",
68+
"os": "ubuntu24.04",
69+
"vendor": "AuthenticAMD"
70+
}
71+
```
72+
73+
We likely want to have a more structured query syntax that can handle AND, OR, and other specifics. The actual search should remain general to support any generic key/value pair of attributes. My database structure and queries are also bad.

‎examples/fractale/software-curl.json

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{
2+
"resources": [
3+
{
4+
"type": "slot",
5+
"count": 1,
6+
"with": [
7+
{
8+
"type": "core",
9+
"count": 1
10+
}
11+
],
12+
"label": "task"
13+
}
14+
],
15+
"tasks": [
16+
{
17+
"command": [
18+
"gmx"
19+
],
20+
"slot": "task",
21+
"count": {
22+
"per_slot": 1
23+
}
24+
}
25+
],
26+
"attributes": {
27+
"system": {
28+
"duration": 0,
29+
"requires": {
30+
"software": [
31+
{
32+
"name": "curl",
33+
"type": "binary"
34+
}
35+
]
36+
}
37+
}
38+
},
39+
"version": 1
40+
}

‎examples/fractale/software-curl.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
version: 1
2+
resources:
3+
- type: slot
4+
count: 1
5+
with:
6+
- type: core
7+
count: 1
8+
label: task
9+
tasks:
10+
- command:
11+
- gmx
12+
slot: task
13+
count:
14+
per_slot: 1
15+
attributes:
16+
system:
17+
duration: 0
18+
requires:
19+
software:
20+
- name: curl
21+
type: binary

0 commit comments

Comments
 (0)