malhuda/alice-lg
1
0
Fork 0
Code Issues 39 Pull Requests 11 Actions Packages Projects Releases 2 Wiki Activity

Initial Prometheus work #169

Merged
nicko170 merged 9 commits from prometheus into develop 2024-11-20 21:08:13 +08:00
Conversation 4 Commits 9 Files Changed 8 +234 -23
nicko170 commented 2024-10-17 08:45:11 +08:00 (Migrated from github.com)
Copy Link

I am sick of all the hacky Python scrips that I am using to gather data and send it to Prometheus / Grafana, so I thought may as well integrate it into the core codebase so anyone else that does something similar can have at it!

Let me know your thoughts - this is a very crude implementation as it calls the APIs internally, to save on duplicating cache collection code.

I'd prefer to have the cache update the data any time it changes - but the cache isn't reliably used and the API request entry points were the ones that made the decision to get from the cache or call the API in the background.

Happy to rework if required, let me know thoughts on this and what else is needed to merge it in, if it's something you'd consider :)

The exported metrics have the following shape, picking on AS10084 (IAA Content) on NSW-IX Route Server 1.

peer_state{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1
peer_state{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1
peer_uptime{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1.5738686327321e+07
peer_uptime{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1.5738688328803e+07
routes_accepted{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 13
routes_accepted{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 19
routes_filtered{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1
routes_filtered{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 0
routes_preferred{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 10
routes_preferred{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 17
routes_received{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 14
routes_received{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 19
I am sick of all the hacky Python scrips that I am using to gather data and send it to Prometheus / Grafana, so I thought may as well integrate it into the core codebase so anyone else that does something similar can have at it! Let me know your thoughts - this is a very crude implementation as it calls the APIs internally, to save on duplicating cache collection code. I'd prefer to have the cache update the data any time it changes - but the cache isn't reliably used and the API request entry points were the ones that made the decision to get from the cache or call the API in the background. Happy to rework if required, let me know thoughts on this and what else is needed to merge it in, if it's something you'd consider :) The exported metrics have the following shape, picking on AS10084 (IAA Content) on NSW-IX Route Server 1. ``` peer_state{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1 peer_state{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1 peer_uptime{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1.5738686327321e+07 peer_uptime{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1.5738688328803e+07 routes_accepted{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 13 routes_accepted{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 19 routes_filtered{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 1 routes_filtered{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 0 routes_preferred{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 10 routes_preferred{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 17 routes_received{peer_address="2001:7fa:11:4:0:2764:0:1",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_2",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 14 routes_received{peer_address="218.100.52.200",peer_asn="10084",peer_description="Internet Association of Australia",peer_id="AS10084_1",route_server_group="NSW-IX",route_server_id="rs1-nsw-v4",route_server_name="rs1.nsw.ix.asn.au"} 19 ```
annikahannig commented 2024-10-18 14:37:15 +08:00 (Migrated from github.com)
Copy Link

I'll need to review this a bit more in depth - but this is looking good. :)
Thank you!

I'll need to review this a bit more in depth - but this is looking good. :) Thank you!
nicko170 commented 2024-10-25 08:14:19 +08:00 (Migrated from github.com)
Copy Link

Thanks @annikahannig - happy to keep workshopping this one.

Some logical changes that I can see are moving the cache logic away from the API endpoints, so metrics can be processed there, and then serving the cache from the endpoints

It looks like there are also a few conflicts with the dev branch, let me merge that in locally - i'll revert one of the commits and clean it up a little, IntelliJ decided it would be a nice idea to reformat that markdown in the README..

Thanks @annikahannig - happy to keep workshopping this one. Some logical changes that I can see are moving the cache logic away from the API endpoints, so metrics can be processed there, and then serving the cache from the endpoints It looks like there are also a few conflicts with the dev branch, let me merge that in locally - i'll revert one of the commits and clean it up a little, IntelliJ decided it would be a nice idea to reformat that markdown in the README..
annikahannig commented 2024-11-20 21:05:35 +08:00 (Migrated from github.com)
Copy Link

I think this is looking good.

I've read a bit in the prometheus documentation and they state in their guide that metrics should have suffixes with clarification on the unit and type...

https://prometheus.io/docs/practices/naming/

So in this case here if this would be: peer_uptime_seconds_total and maybe changing peer state to peer_info and having the state as a label.

Rest of the metrics and labels look good I think.

I think this is looking good. I've read a bit in the prometheus documentation and they state in their guide that metrics should have suffixes with clarification on the unit and type... https://prometheus.io/docs/practices/naming/ So in this case here if this would be: `peer_uptime_seconds_total` and maybe changing peer state to `peer_info` and having the state as a label. Rest of the metrics and labels look good I think.
annikahannig commented 2024-11-20 21:08:09 +08:00 (Migrated from github.com)
Copy Link

I'll merge this but I'll likely change the config flag to just enable_metrics .
Also I think I'm going to rewrite this a bit using a custom metrics collector.

I'll merge this but I'll likely change the config flag to just `enable_metrics` . Also I think I'm going to rewrite this a bit using a custom metrics collector.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
bug
Something isn't working
dependencies
Pull requests that update a dependency file
duplicate
This issue or pull request already exists
enhancement
New feature or request
go
Pull requests that update Go code
good first issue
Good for newcomers
help wanted
Extra attention is needed
invalid
This doesn't seem right
javascript
Pull requests that update Javascript code
priority
question
Further information is requested
wontfix
This will not be worked on
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: malhuda/alice-lg#169
No description provided.
Delete Branch "prometheus"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?