[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[taler-grid5k] 72/141: configure postgres
From: |
gnunet |
Subject: |
[taler-grid5k] 72/141: configure postgres |
Date: |
Thu, 18 Nov 2021 14:50:13 +0100 |
This is an automated email from the git hooks/post-receive script.
marco-boss pushed a commit to branch master
in repository grid5k.
commit 047946516b5e410a018b40558c35821506d58434
Author: Boss Marco <bossm8@bfh.ch>
AuthorDate: Mon Oct 25 22:36:53 2021 +0200
configure postgres
---
configs/etc/monitor/postgres-exporter.yaml | 2 +-
experiment/scripts/database.sh | 22 ++++++++++---
experiment/scripts/exchange.sh | 2 +-
experiment/scripts/proxy.sh | 2 +-
experiment/scripts/test.sh | 4 ++-
experiment/taler-perf-any.rspec | 28 +++++++++++++----
screenshot.sh | 50 ++++++++++++++++++++++++++++++
7 files changed, 96 insertions(+), 14 deletions(-)
diff --git a/configs/etc/monitor/postgres-exporter.yaml
b/configs/etc/monitor/postgres-exporter.yaml
index d1a20a1..e473b6a 100644
--- a/configs/etc/monitor/postgres-exporter.yaml
+++ b/configs/etc/monitor/postgres-exporter.yaml
@@ -159,7 +159,7 @@ pg_database:
description: "Disk space used by the database"
pg_stat_statements:
- query: "SELECT t2.rolname, t3.datname, queryid, calls, total_time / 1000 as
total_time_seconds, min_time / 1000 as min_time_seconds, max_time / 1000 as
max_time_seconds, mean_time / 1000 as mean_time_seconds, stddev_time / 1000 as
stddev_time_seconds, rows, shared_blks_hit, shared_blks_read,
shared_blks_dirtied, shared_blks_written, local_blks_hit, local_blks_read,
local_blks_dirtied, local_blks_written, temp_blks_read, temp_blks_written,
blk_read_time / 1000 as blk_read_time_seconds, [...]
+ query: "SELECT t2.rolname, t3.datname, queryid, calls, total_exec_time /
1000 as total_time_seconds, min_exec_time / 1000 as min_time_seconds,
max_exec_time / 1000 as max_time_seconds, mean_exec_time / 1000 as
mean_time_seconds, stddev_exec_time / 1000 as stddev_time_seconds, rows,
shared_blks_hit, shared_blks_read, shared_blks_dirtied, shared_blks_written,
local_blks_hit, local_blks_read, local_blks_dirtied, local_blks_written,
temp_blks_read, temp_blks_written, blk_read_time / 1000 a [...]
master: true
metrics:
- rolname:
diff --git a/experiment/scripts/database.sh b/experiment/scripts/database.sh
index 80ebe9d..9926774 100755
--- a/experiment/scripts/database.sh
+++ b/experiment/scripts/database.sh
@@ -1,17 +1,28 @@
#!/bin/bash
set -eux
+source ~/scripts/helpers.sh
+
# move to tmp to prevent change dir errors
cd /tmp
sed -i "s\<DB_URL_HERE>\postgresql:///${DB_NAME}\g" \
/etc/taler/secrets/exchange-db.secret.conf
-echo "listen_addresses='*'" >> /etc/postgresql/13/main/postgresql.conf
+echo "
+listen_addresses='*'
+log_destination=syslog
+syslog_ident='taler-database'
+log_min_duration_statement=1000
+shared_preload_libraries='pg_stat_statements,auto_explain'
+" >> /etc/postgresql/13/main/postgresql.conf
+
# Enable password for taler since this is commonly the case
-echo "host all ${DB_USER} 172.16.0.0/12 md5" >>
/etc/postgresql/13/main/pg_hba.conf
# For the postgres user do not enable authentication (used in metrics)
-echo "host all postgres 172.16.0.0/12 trust" >>
/etc/postgresql/13/main/pg_hba.conf
+echo "
+host all ${DB_USER} 172.16.0.0/12 md5
+host all postgres 172.16.0.0/12 trust
+" >> /etc/postgresql/13/main/pg_hba.conf
systemctl restart postgresql
@@ -24,7 +35,8 @@ sudo -u taler-exchange-httpd taler-exchange-dbinit
su postgres << EOF
psql
-create user "${DB_USER}" with encrypted password '${DB_PASSWORD}';
+CREATE USER "${DB_USER}" with encrypted password '${DB_PASSWORD}';
+CREATE EXTENSION pg_stat_statements;
EOF
su taler-exchange-httpd -s /bin/bash << EOF
@@ -33,4 +45,6 @@ GRANT SELECT,INSERT,UPDATE ON ALL TABLES IN SCHEMA public TO
"${DB_USER}";
GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO "${DB_USER}";
EOF
+restart_rsyslog
+
exit 0
diff --git a/experiment/scripts/exchange.sh b/experiment/scripts/exchange.sh
index 4a7d6b6..6aac170 100755
--- a/experiment/scripts/exchange.sh
+++ b/experiment/scripts/exchange.sh
@@ -26,7 +26,7 @@ systemctl restart taler-exchange.target
for (( i=1; i < ${NUM_EXCHANGE_PROCESSES}; i++ ))
do
- systemctl restart taler-exchange-httpd@808"${i}".socket
taler-exchange-httpd@808"${i}".service
+ systemctl restart taler-exchange-httpd@100"${i}".socket
taler-exchange-httpd@100"${i}".service
done
wait_for_keys "${EXCHANGE_DOMAIN}/management"
diff --git a/experiment/scripts/proxy.sh b/experiment/scripts/proxy.sh
index 22514b9..bbe49d4 100755
--- a/experiment/scripts/proxy.sh
+++ b/experiment/scripts/proxy.sh
@@ -10,7 +10,7 @@ sed -e "/<SERVERS_HERE>/a \ \ server ${EXCHANGE_DOMAIN}:80;" \
# We want n processes, one is already enabled on port 80
for (( i=1; i < ${NUM_EXCHANGE_PROCESSES}; i++ ))
do
- sed -i "/<SERVERS_HERE>/a \ \ server ${EXCHANGE_DOMAIN}:808${i};" \
+ sed -i "/<SERVERS_HERE>/a \ \ server ${EXCHANGE_DOMAIN}:100${i};" \
/etc/nginx/sites-enabled/proxy
done
diff --git a/experiment/scripts/test.sh b/experiment/scripts/test.sh
index 12717c3..c8930ed 100755
--- a/experiment/scripts/test.sh
+++ b/experiment/scripts/test.sh
@@ -3,12 +3,14 @@
FAKE_PID=$1
DB_FILE=~/walletdb/wallet."${FAKE_PID}".json
+IP=$(hostname -I)
+
function log() {
logger -s --tcp \
--port 1515 \
--server ${MONITOR_DOMAIN} \
--tag taler-wallet \
- "ts=$(date -Ins) id=${FAKE_PID} host=${HOSTNAME} level=${2:-INFO}
${1}"
+ "ts=$(date -Ins) id=${FAKE_PID} host=${IP} level=${2:-INFO} ${1}"
}
for i in {0..10000};
diff --git a/experiment/taler-perf-any.rspec b/experiment/taler-perf-any.rspec
index 0306508..020edcb 100644
--- a/experiment/taler-perf-any.rspec
+++ b/experiment/taler-perf-any.rspec
@@ -1,5 +1,5 @@
<?xml version='1.0'?>
-<rspec xmlns="http://www.geni.net/resources/rspec/3" type="request"
generated_by="jFed RSpec Editor" generated="2021-10-21T20:59:36.138+02:00"
xmlns:emulab="http://www.protogeni.net/resources/rspec/ext/emulab/1"
xmlns:delay="http://www.protogeni.net/resources/rspec/ext/delay/1"
xmlns:jfed-command="http://jfed.iminds.be/rspec/ext/jfed-command/1"
xmlns:client="http://www.protogeni.net/resources/rspec/ext/client/1"
xmlns:jfed-ssh-keys="http://jfed.iminds.be/rspec/ext/jfed-ssh-keys/1" xmlns:
[...]
+<rspec xmlns="http://www.geni.net/resources/rspec/3" type="request"
generated_by="jFed RSpec Editor" generated="2021-10-25T19:40:23.568+02:00"
xmlns:emulab="http://www.protogeni.net/resources/rspec/ext/emulab/1"
xmlns:delay="http://www.protogeni.net/resources/rspec/ext/delay/1"
xmlns:jfed-command="http://jfed.iminds.be/rspec/ext/jfed-command/1"
xmlns:client="http://www.protogeni.net/resources/rspec/ext/client/1"
xmlns:jfed-ssh-keys="http://jfed.iminds.be/rspec/ext/jfed-ssh-keys/1" xmlns:
[...]
<node client_id="DB" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
<sliver_type name="raw-pc">
<disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
@@ -10,13 +10,14 @@
<sliver_type name="raw-pc">
<disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
</sliver_type>
+ <hardware_type name="grisou-nancy"/>
<location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="767.0"
y="292.5"/>
</node>
<node client_id="Wallet-0" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
<sliver_type name="raw-pc">
<disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
</sliver_type>
- <hardware_type name="nova-lyon"/>
+ <hardware_type name="grvingt-nancy"/>
<location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="633.5"
y="449.5"/>
</node>
<node client_id="Bank" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
@@ -29,7 +30,8 @@
<sliver_type name="raw-pc">
<disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
</sliver_type>
- <location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="769.5"
y="337.5"/>
+ <hardware_type name="grisou-nancy"/>
+ <location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="769.5"
y="334.5"/>
</node>
<node client_id="Monitor" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
<sliver_type name="raw-pc">
@@ -41,15 +43,15 @@
<sliver_type name="raw-pc">
<disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
</sliver_type>
- <hardware_type name="nova-lyon"/>
+ <hardware_type name="grvingt-nancy"/>
<location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="766.5"
y="450.5"/>
</node>
<node client_id="Wallet-2" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
<sliver_type name="raw-pc">
<disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
</sliver_type>
- <hardware_type name="nova-lyon"/>
- <location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="894.5"
y="448.5"/>
+ <hardware_type name="grvingt-nancy"/>
+ <location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="908.5"
y="451.5"/>
</node>
<node client_id="Merchant" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
<sliver_type name="raw-pc">
@@ -57,4 +59,18 @@
</sliver_type>
<location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="972.5"
y="250.5"/>
</node>
+ <node client_id="Wallet-3" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
+ <sliver_type name="raw-pc">
+ <disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
+ </sliver_type>
+ <hardware_type name="grvingt-nancy"/>
+ <location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="685.5"
y="492.5"/>
+ </node>
+ <node client_id="Wallet-4" exclusive="true"
component_manager_id="urn:publicid:IDN+am.grid5000.fr+authority+am">
+ <sliver_type name="raw-pc">
+ <disk_image
name="http://public.lille.grid5000.fr/~bfhch01/taler-debian11.dsc"/>
+ </sliver_type>
+ <hardware_type name="grvingt-nancy"/>
+ <location xmlns="http://jfed.iminds.be/rspec/ext/jfed/1" x="840.0"
y="491.5"/>
+ </node>
</rspec>
\ No newline at end of file
diff --git a/screenshot.sh b/screenshot.sh
new file mode 100755
index 0000000..d02369b
--- /dev/null
+++ b/screenshot.sh
@@ -0,0 +1,50 @@
+#!/bin/bash
+set -x
+
+API_KEY=eyJrIjoidkJsRzZzb2I3R2VWdTVBWWRiUURmeFdGUGpqS240YTgiLCJuIjoidGFsZXItcGVyZm9ybWFuY2UtYWRtaW4iLCJpZCI6MX0=
+
+AUTH_HEADER="Authorization: Bearer ${API_KEY}"
+
+BASE_URL="https://147.87.255.221:3000"
+
+RENDER=/render/d-solo
+
+DASHBOARDS=$(\
+ curl -X GET \
+ -k \
+ -H "${AUTH_HEADER}" \
+ -H "Content-Type: application/json" \
+ -H "Accept: application/json" \
+ "${BASE_URL}/api/search?tag=performance" \
+ | jq -r '.[] | .url'
+)
+
+for DASH in $DASHBOARDS; do
+
+ DUID=$(echo "$DASH" | cut -d "/" -f 3)
+
+ readarray -t PANELS < <(\
+ curl -X GET \
+ -k \
+ -H "${AUTH_HEADER}" \
+ -H "Accept: application/json" \
+ "${BASE_URL}/api/dashboards/uid/${DUID}" \
+ | jq | tee /dev/tty | jq -c '.dashboard.panels[]'
+ )
+
+ ENDPOINT=$(echo "$DASH" | sed "s\/d/\/d-solo/\g")
+ DASH_NAME=$(echo "$DASH" | cut -d "/" -f 4)
+
+ for PANEL in "${PANELS[@]}"; do
+ NAME=$(jq -r '.title' <<< "${PANEL}" | tr 'A-Z' 'a-z' | tr -cs 'a-z0-9\n'
'-')
+ ID=$(jq '.id' <<< "${PANEL}")
+ curl -X GET \
+ -k \
+ -H "${AUTH_HEADER}" \
+ -H "Accept: application/json" \
+ --output "${DASH_NAME}-${NAME}.png" \
+ "${BASE_URL}/render${ENDPOINT}?orgId=1&panelId=${ID}&theme=light"
+ done
+
+done
+
--
To stop receiving notification emails like this one, please contact
gnunet@gnunet.org.
- [taler-grid5k] 31/141: update scripts, (continued)
- [taler-grid5k] 31/141: update scripts, gnunet, 2021/11/18
- [taler-grid5k] 57/141: configure nginx logging to syslog, gnunet, 2021/11/18
- [taler-grid5k] 47/141: fix promtail, add helper function script, gnunet, 2021/11/18
- [taler-grid5k] 45/141: add rsyslog config for better performance with loki, gnunet, 2021/11/18
- [taler-grid5k] 48/141: fix script imports, add net packages, gnunet, 2021/11/18
- [taler-grid5k] 46/141: mulitple exch processes possible - syslog configuredwq, gnunet, 2021/11/18
- [taler-grid5k] 62/141: fix rsyslog and wallet log messages, gnunet, 2021/11/18
- [taler-grid5k] 19/141: add nameserver to /etc/resolv only if not there yet, gnunet, 2021/11/18
- [taler-grid5k] 41/141: fix prometheus, gnunet, 2021/11/18
- [taler-grid5k] 64/141: update promtail - issues with too many open files when too much labels are assigned, gnunet, 2021/11/18
- [taler-grid5k] 72/141: configure postgres,
gnunet <=
- [taler-grid5k] 40/141: update configurations, gnunet, 2021/11/18
- [taler-grid5k] 32/141: add taler default env, gnunet, 2021/11/18
- [taler-grid5k] 67/141: add merchant, some comments in scripts, gnunet, 2021/11/18
- [taler-grid5k] 54/141: add step to build newest wallet from source, gnunet, 2021/11/18
- [taler-grid5k] 56/141: add debug to every taler service, gnunet, 2021/11/18
- [taler-grid5k] 29/141: run dns on taler.net, gnunet, 2021/11/18
- [taler-grid5k] 53/141: clear dns updated, gnunet, 2021/11/18
- [taler-grid5k] 71/141: remove trhottle, gnunet, 2021/11/18
- [taler-grid5k] 33/141: update path, gnunet, 2021/11/18
- [taler-grid5k] 38/141: update image - add prometheus + loki, gnunet, 2021/11/18