Skip to content

Commit 224f08f

Browse files
authored
Merge pull request #28 from xedinseu/teccod-update
new update
2 parents 9254dea + 7a240b6 commit 224f08f

File tree

116 files changed

+1062
-246
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

116 files changed

+1062
-246
lines changed

README.md

Lines changed: 121 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -3,56 +3,158 @@
33
[![Gitter](https://img.shields.io/badge/chat-on%20telegram-blue.svg)](https://t.me/joinchat/FoZ4M0KF5NakAJPwyg1Saw)
44

55
InterSystems Developer Community analytics.
6-
Project made with InterSystems Analytics (DeepSee) and various BI systems to visualize and analyze members, articles, questions, answers, views and other pieces of content and activity on [InterSystems Developer Community](community.intersystems.com). This project contains pre-configured IRIS and Atscale deployment in Docker containers and project files for BI systems .
6+
Project made with InterSystems IRIS BI (DeepSee), Power BI and Logi Report Designer to visualize and analyze members, articles, questions, answers, views and other pieces of content and activity on [InterSystems Developer Community](community.intersystems.com). This project contains pre-configured IRIS and Atscale deployment in Docker containers and project files for BI systems . This project also deployed online and you can check it here:
77

8-
[See Live DC analytics](we'll provide the link to online demo as soon as we made it)
8+
### See Live DC analytics
99

10-
### 1. To start IRIS
11-
Open the terminal in this directory and run:
10+
[PowerBI Interactive Dashboard with connection to the Adaptive Analytics Instance](https://app.powerbi.com/view?r=eyJrIjoiMjU0NjgyNjgtYjA0NS00M2Q3LTg3NzUtN2EwNjVjMGQ0Yzk5IiwidCI6IjRlMzI1ZjgzLWE4YWYtNGY2Yy05ODY3LTBkYTA4YmU1MGZjMCIsImMiOjEwfQ%3D%3D&pageName=ReportSection&language=en-us)
11+
12+
[Instance of Adaptive Analytics powered by AtScale](http://atscale-public.demo.community.intersystems.com:10500/login)
13+
14+
Login: user
15+
Password: userp@$$
16+
17+
### How to run services local:
18+
First, you need to clone this repository to your local machine. Below in the text we will use relative paths to files when describing what is where. We assume that you are already in the project's root directory.
19+
20+
#### Authenticating to the ICR
21+
To log into the ICR, take the following steps:
22+
23+
Load https://containers.intersystems.com/ in your browser and log in with your InterSystems/WRC credentials.
24+
25+
Retrieve your Docker login token, or the full login command.
26+
27+
In your Docker interface (for example, your PowerShell window or Linux command line), authenticate to the ICR using the provided credentials. You can do this by copying and pasting the full docker login command displayed, for example:
28+
29+
```
30+
docker login -u="your_username" -p="provided_password" containers.intersystems.com
31+
```
32+
33+
For security reasons, however, you may want to instead enter the command docker login containers.intersystems.com, then enter your username at the Username prompt and paste your password into the Password: prompt.
34+
35+
Note: If you are logged into another Docker registry, the docker login command may result in an error; log out of the other registry before logging into containers.intersystems.com.
36+
37+
### 1. Starting IRIS
38+
InterSystem IRIS stores all the data in this sample project/ So we need InterSystems IRIS up and running and persists data. Below we will take the vanila image of InterSystems IRIS and load Community analytics data. Open the terminal in the project directory and run:
1239
```
1340
$ docker-compose up -d iris
1441
```
15-
Check the availability of the service:
16-
http://localhost:52773/dsw/index.html#/DCANALYTICS
17-
42+
43+
This procedure will take 10-15 minutes.
44+
45+
After build will complete, you may check the IRIS BI dashboards here:
46+
http://localhost:52773/dsw/index.html#/DCANALYTICS
47+
48+
It will looks like that:
49+
50+
<img width="437" alt="image" src="https://user-images.githubusercontent.com/41373877/184346201-61a3cc68-8cdc-4fcc-a27d-664a877728eb.png">
51+
1852
Standard login and password:
1953
_system/SYS
2054

21-
### 2. To start the Atscale server:
55+
One of basic dashboards:
56+
<img width="1188" alt="image" src="https://user-images.githubusercontent.com/41373877/184346339-7e896fcd-629b-4225-85c2-950d9fe63723.png">
57+
58+
#### Option: UDAF optimization
59+
If you have a commercial license on IRIS you may have LOGIN and PASSWORD for use in private ZPM registry.
60+
In that case you may install isc-udaf package from there.
61+
Installation command in IRIS looks like this:
62+
```
63+
zpm "repo -n registry -r -url https://pm.intersystems.com/ -user LOGIN -pass PASSWORD"
64+
zpm "install isc-udaf"
65+
```
66+
Next, you have to swith UDAF on in AtScale Data Warehouse settings:
67+
Go to http://[your-atscale-server]:10500/org/default/settings/data-warehouses and set Custom Function Installation Mode to Custom Managed value.
68+
![image](https://user-images.githubusercontent.com/41373877/184353600-f0032b36-cf35-418b-94bf-d79c5e1a9a53.png)
69+
70+
UDAF gives AtScale 2 main advantages:
71+
- the ability to store query cash (they call it Aggregate Tables), so that the next query, using aggregation on data, takes from the database already pre calculated results.
72+
- the ability to use additional functions (actually User-Defined Aggregate Functions) and data processing algorithms that AtScale is forced to store in the data source.
73+
They are stored in the database in a separate table and Adaptive Analytics can call them by name in auto generated queries. When AtScale can use these functions, the speed of queries increases.
2274

23-
A license must be entered to run Atscale server. To do this, you need to put the json file with the license in the folder "atscale-dataset/src/license".
75+
AtScale has an internal logic for updating aggregate tables, but it is much more convenient to control this process yourself.
76+
You can configure updates on a per-cube basis in the web interface of AtScale and then use scripts from "iris/src/aggregate tables update shedule scripts" to export schedules and import to another instance, or use the exported schedule file as a backup. You will also find a script to set all cubes to the same update schedule if you do not want to configure each one individually.
2477

25-
If you need to change hostname parameter, open directory "atscale-dataset", file "atscale.yaml" and edit (ip or dns name):
78+
79+
### 2. To start an Atscale server:
80+
81+
A license must be provided to run Atscale server. To do this you need to put the json file with the license in the folder "atscale-dataset/src/license".
82+
#ToDo Add how to give a link by docker-compose having one license for several projects.
83+
84+
In case you plan to deploy this project to some server you might want to alter hostname parameter: open directory "atscale-dataset", file "atscale.yaml" and edit (ip or dns name):
2685
loadbalancer_dns_name: "127.0.0.1"
2786

2887
Run docker container with the command:
2988
```
3089
docker-compose up -d atscale
3190
```
32-
Wait 2-3 minutes while server is starting up and check the availability of the service:
91+
This will take a few minutes to pull and build the container.
92+
93+
After successful build wait 2-3 minutes while server is starting up and check the availability of the service:
94+
3395
http://localhost:10500/login
3496

3597
Standard login and password:
3698
admin/admin
3799

38-
If you want to access the web interface, at the initialization step, you need to leave the default connection port, leave the login and password admin / admin and select an Embedded directory for storing data. These connection settings are stored in the BI projects settings.
100+
This web interface lets you create and edin AtScale BI cubes. You need to follow the initial AtScale setup wizard and choose the following options.
101+
102+
- you need to leave the default connection port
103+
- leave the login and password admin / admin
104+
- select an Embedded directory for storing data
105+
106+
These connection settings are stored in the BI projects settings.
107+
108+
If you want to modify AtScale cubes you can find them in the projects tab. When you finish editing cubes and want to save changes go to the export tab of the project and download the xml representation of your cube. Then you need to put this file in the “/atscale-server/src/cubes/” folder of the repository (file name doesn't matter) and delete the existing xml file of the project. If you don't delete the original file AtScale wouldn't publish the updated project with the same name and id. At the next build a new version of the project will be automatically passed to AtScale and will be ready for use as a default project.
109+
110+
AtScale doesn't provide public documentation. In order to obtain the documentation open http://localhost:10500/org/default/support and go to the Documentation tab. You will be redirected to https://documentation.atscale.com/ and will be able to view it. We recommend examining [this](https://documentation.atscale.com/2022.2.0/creating-and-sharing-cubes/creating-cubes) chapter to work with existing projects.
111+
112+
#### Saving changes:
113+
By default, all changes to the IRIS or Atscale projects are saved inside these systems.
114+
To cancel all changes and return to the default settings, from the current directory, run the command:
115+
```
116+
docker-compose down -v
117+
```
118+
**Warning!** This will remove all created Iris and Atscale containers and any changes saved in them.
39119

40-
### 3. To use BI projects
120+
121+
### 3. Using BI projects
122+
123+
#### Using demo IRIS and AtScale.
124+
125+
In "analytic dashboards" folder you can find BI projects that are configured for both local data source and deployed online demo.
126+
127+
The AtScale online demo located at http://atscale-public.demo.community.intersystems.com:10500/login. Login and password for it: user/userp@$$
41128

42129
#### Power BI
130+
The option of creation and editing of Power BI dashboards is available only for Windows OS.
131+
132+
You may simply run the .pbix file from "analytic dashboards\Power BI" in the [Power BI desktop](https://powerbi.microsoft.com/en-gb/desktop/). Power BI stores BI data in this file, so when you open it for the first time it will show BI dashboards with data that was loaded by developers of this project. Click the refresh button to load the actual data into Power BI. Use the same credentials as in the Adaptive Analytics (AtScale) with Simple Auth method (admin/admin).
133+
134+
If you want to modify the Power BI dashboards you may check [official beginner's guide](https://docs.microsoft.com/en-us/power-bi/fundamentals/desktop-getting-started) to learn the basics. We already prepare all the data, so you can start from [build report](https://docs.microsoft.com/en-us/power-bi/fundamentals/desktop-getting-started#build-reports) step.
43135

44-
You may simply run the .pbix file from "analytic dashboards\Power BI" in Power BI desktop. Power BI stored data in this file, so to have actual data you firstly need to update the report by the Refresh button.
136+
After you finish modifying the Power BI project and save it, the changes will be automatically stored in the Power BI file in the repository.
45137

46138
#### Logi
47-
You need a licensed copy of the Logi report Designer to run the reports.
139+
You need a licensed copy of the Logi Report Designer to edit and run the reports. Also, you will need drivers for jdbc connection. You can find them here: https://github.com/teccod/Logi-JDBC-drivers. To use them you need to place [lib](https://github.com/teccod/Logi-JDBC-drivers/tree/main/lib) folder in the root folder of Logi Report Designer's installation directory.
140+
141+
#ToDo Describe how to obtain your copy of Logi Designer or Logi Server. From WRC or similar.
142+
143+
On the Start page of Logi Report Designer you need to click the Browse button in the Open category and select the path to the .cat file, then open a .cls file.
144+
145+
In "analytic dashboards\logi report\En report" folder you may find the Сommunity monthly overview report - x pages report.
146+
147+
In "analytic dashboards\logi report\members by community" folder you may find a simple report regarding the distribution of members by different language communities.
148+
149+
If you want to modify Logi reports you may be interested in [official beginner's guide](https://devnet.logianalytics.com/hc/en-us/articles/4405661292567-Report-Types) to understand this BI system. If you only want quickly examine some data, you may be interested in explanation how to make a [chart](https://devnet.logianalytics.com/hc/en-us/articles/4405664328087-Lesson-4-Creating-a-Chart-Report) or a [table](https://devnet.logianalytics.com/hc/en-us/articles/4405664328983-Lesson-5-Creating-a-Table-Report#Task1). The beginner's guide describes the process of creating a widget in a new report file. You can insert a chart or table into an existing report. You can do it from the Insert tab or Component window.
48150

49-
In the folder "analytic dashboards\logi report\En report" you can find the report for the EN community.
151+
![image](https://user-images.githubusercontent.com/91419671/181744373-dcbe9939-e6cf-46e4-8ef2-a60f0ed70e21.png)
50152

51-
In folder "analytic dashboards\logi report\members by community" you can find the report about distribution of members by different language communities.
153+
When you finish editing reports, you need to save the changes in the cls file, which is responsible for the visual part and in the cat file both, which is responsible for the structure and data processing.
52154

53155
# Collaboration
54156

55-
If you want to use this project on your inw IRIS installation, you may use the following instructions.
157+
If you want to use this project on your onw IRIS installation, you may use the following instructions.
56158

57159
## Installation
58160
### Basic
@@ -68,7 +170,7 @@ DCANALYTICS> do ##class(Community.Utils).setup("path/to/global/archive.gz file")
68170
7. Open in your browser `<server:port>/dsw/index.html?ns=DCANALYTICS`.
69171
8. Done
70172
### With iKnow
71-
1. Enable iKnow in namespace web-application settings. Be sure, that everything listed above is installed.
173+
1. Enable iKnow in namespace web-application settings. Be sure that everything listed above is installed.
72174
2. Download from release `DSW.config.and.iKnow.files.zip` and move files `sets.txt` and `backlist.txt` from archive to `<your_instance>/Mgr/DCANALYTICS/`.
73175
3. Run in terminal:
74176
```
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

atscale-server/entrypoint.sh

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
#!/bin/sh
2+
3+
if [ -f ./first-time.set ]
4+
then
5+
# JQ package used later in backend response parsing
6+
apt update
7+
apt install -y jq
8+
9+
cp ./atscale.yaml /opt/atscale/conf
10+
chown atscale:atscale /opt/atscale/conf/atscale.yaml
11+
rm ./first-time.set
12+
su - atscale -c "/opt/atscale/versions/2021.3.0.3934/bin/configurator.sh --activate --automatic-install"
13+
14+
RESPONSE="1"
15+
TIME=1
16+
echo "Waiting for engine ......."
17+
18+
while [ "$RESPONSE" != "200 OK" ]
19+
do
20+
for license in /root/license/*
21+
do
22+
RESPONSE=$(curl --location --request PUT 'http://127.0.0.1:10502/license' --header 'Content-Type: application/json' --data-binary "@$license" | jq -r '.status.message')
23+
sleep 1s
24+
done
25+
echo "$TIME seconds waiting"
26+
TIME=$(($TIME + 1))
27+
done
28+
echo "Engine started"
29+
30+
TOKEN=$(curl -u admin:admin --location --request GET 'http://localhost:10500/default/auth')
31+
32+
for cube in /root/cubes/*
33+
do
34+
sleep 1s
35+
curl --location --request POST "http://localhost:10500/api/1.0/org/default/project" --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/xml" --data-binary "@$cube"
36+
sleep 5s
37+
done
38+
39+
for connection in /root/connections/*
40+
do
41+
sleep 1s
42+
curl --location --request POST "http://localhost:10502/connection-groups/orgId/default" --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" --data-binary "@$connection"
43+
sleep 5s
44+
done
45+
46+
PROJECTS_ID=$(curl --location --request GET "http://localhost:10500/api/1.0/org/default/projects" --header "Authorization: Bearer $TOKEN" | jq -r '.response[].id')
47+
for project in $PROJECTS_ID
48+
do
49+
curl --location --request POST "http://localhost:10500/api/1.0/org/default/project/$project/publish" --header "Authorization: Bearer $TOKEN"
50+
sleep 1s
51+
done
52+
53+
54+
sleep 1s
55+
curl --location --request POST "http://localhost:10502/organizations/orgId/default" --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" --data-raw '{"hiveServer2Port": 11111}'
56+
sleep 1s
57+
58+
for settings_list in /root/settings/*
59+
do
60+
curl --location --request PATCH 'http://localhost:10502/settings' --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" --data-binary "@$settings_list"
61+
done
62+
63+
curl --location --request POST 'http://localhost:10500/api/1.0/referrerOrg/default/support/service-control/engine/restart'
64+
else
65+
su - atscale -c "/opt/atscale/bin/atscale_start"
66+
fi
67+
68+
exec "$@"

dev.md

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ s s=##class(%Stream.FileBinaryGzip).%New() do s.LinkToFile(fn) do $System.OBJ.Ex
99

1010
## build container with no cache
1111
```
12-
docker-compose build --no-cache
12+
docker-compose build --no-cache --progress=plain
1313
```
1414
## open terminal to docker
1515
```
@@ -24,4 +24,22 @@ docker-compose exec iris iris session iris -U IRISAPP
2424
## service account key base64 command
2525
```
2626
$ base64 iris-community-demos-09899421cfc4.json | tr -d '\n'
27-
```
27+
```
28+
29+
## alternative atscale image
30+
```
31+
ghcr.io/intersystems-community/atscale-server:2021.3.0.3934
32+
```
33+
34+
To use this alternative image you need to authorize docker to download images from [ghcr.io](https://ghcr.io).
35+
Login requires your GitHub account credentials, run the command in the terminal:
36+
```
37+
$ docker login ghcr.io
38+
> Username: your_username
39+
> Password: your_github_token
40+
```
41+
42+
When entering a password, you need to use the GitHub access token, you can read how to get it at the [link](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-token).
43+
In the "GitHub access token" settings, should be opened access to read packages from the GitHub package registry:
44+
<img src="https://user-images.githubusercontent.com/49229973/157600567-f80a8f9f-15e2-41c6-bf18-2c3ef6e1639d.jpg" width="600" />
45+

docker-compose.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ services:
1414
- ~/iris.key:/usr/irissys/mgr/iris.key
1515
- ./:/irisdev/app
1616
atscale:
17-
image: ghcr.io/intersystems-community/atscale-server:2021.3.0.3934
17+
image: containers.intersystems.com/intersystems/adaptive-analytics:2021.3.0.3934
1818
restart: always
1919
tty: true
2020
hostname: atscale
@@ -23,6 +23,7 @@ services:
2323
- conf:/opt/atscale/conf
2424
- log:/opt/atscale/log
2525
- ./atscale-server/atscale.yaml:/root/atscale.yaml
26+
- ./atscale-server/entrypoint.sh:/root/entrypoint.sh
2627
- ./atscale-server/src/connections:/root/connections
2728
- ./atscale-server/src/cubes:/root/cubes
2829
- ./atscale-server/src/license:/root/license

iris/Installer.cls

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,8 @@ XData setup
1212

1313
<Configuration>
1414
<Database Name="${Namespace}" Dir="/opt/${app}/data" Create="yes" Resource="%DB_${Namespace}"/>
15-
16-
<Import File="${SourceDir}" Flags="ck" Recurse="1"/>
1715
</Configuration>
18-
<CSPApplication Url="/csp/${app}" Directory="${cspdir}${app}" ServeFiles="1" Recurse="1" MatchRoles=":%DB_${Namespace}" AuthenticationMethods="32"
19-
20-
/>
16+
<CSPApplication Url="/csp/${app}" Directory="${cspdir}${app}" ServeFiles="1" Recurse="1" MatchRoles=":%DB_${Namespace}" AuthenticationMethods="32"/>
2117
</Namespace>
2218

2319
</Manifest>

0 commit comments

Comments
 (0)