AdGuardDNS/doc/development.md

292 lines
11 KiB
Markdown
Raw Permalink Normal View History

2024-10-14 17:44:24 +03:00
# AdGuard DNS Development Setup
2022-08-26 14:18:35 +03:00
2024-10-14 17:44:24 +03:00
## Contents
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
- [Initial setup](#init)
- [Common Makefile macros and targets](#makefile)
- [How to run AdGuard DNS](#run)
- [Testing](#testing)
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
## <a href="#init" id="init" name="init">Initial setup</a>
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
Development is supported on Linux and macOS (aka Darwin) systems.
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
1. Install Go 1.23 or later.
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
2. Call `make init` to set up the Git pre-commit hook.
2022-08-26 14:18:35 +03:00
2024-10-14 17:44:24 +03:00
3. Call `make go-tools` to install analyzers and other tools into the `bin` directory.
2022-08-26 14:18:35 +03:00
2024-10-14 17:44:24 +03:00
## <a href="#makefile" id="makefile" name="makefile">Common Makefile macros and targets</a>
Most development tasks are done through the use of our Makefile. Please keep the Makefile POSIX-compliant and portable.
### <a href="#makefile-macros" id="makefile-macros" name="makefile-macros">Macros</a>
This is not an extensive list. See `../Makefile` and the scripts in the `../scripts/make/` directory.
- `OUT`: The name of the binary to build. Default: `./AdGuardDNS`.
- `RACE`: Set to `1` to enable the race detector. The race detector is always enabled for `make go-test`.
- `VERBOSE`: Set to `1` to enable verbose mode. Default: `0`.
### <a href="#makefile-targets" id="makefile-targets" name="makefile-targets">Targets</a>
This is not an extensive list. See `../Makefile`.
- `make init`: Set up the pre-commit hook that runs checks, linters, and tests.
- `make go-build`: Build the binary. See also the `OUT` and `RACE` macros.
- `make go-gen`: Regenerate the automatically generated Go files that need to be periodically updated. Those generated files are:
- `../internal/backendpb/dns.pb.go`
- `../internal/backendpb/dns_grpc.pb.go`
- `../internal/ecscache/ecsblockilist_generate.go`
- `../internal/geoip/asntops_generate.go`
- `../internal/geoip/country_generate.go`
- `../internal/profiledb/internal/filecachepb/filecache.pb.go`
You'll need to [install `protoc`][protoc] for the last one.
- `make go-lint`: Run Go checkers and static analysis.
- `make go-test`: Run Go tests.
- `make go-bench`: Run Go benchmarks.
- `make go-tools`: Install the Go static analysis tools locally.
- `make test`: Currently does the same thing as `make go-test` but is defined both because it's a common target and also in case code in another language appears in the future.
2022-08-26 14:18:35 +03:00
2024-10-14 17:44:24 +03:00
- `make txt-lint`: Run plain text checkers.
2022-08-26 14:18:35 +03:00
2024-10-14 17:44:24 +03:00
[protoc]: https://protobuf.dev/getting-started/gotutorial/#compiling-protocol-buffers
2022-08-26 14:18:35 +03:00
2024-10-14 17:44:24 +03:00
## <a href="#run" id="run" name="run">How to run AdGuard DNS</a>
This is an example on how to run AdGuard DNS locally.
### <a href="#run-1" id="run-1" name="run-1">Step 1: prepare the TLS certificate and the key</a>
2022-08-26 14:18:35 +03:00
Keeping the test files in the `test` directory since it's added to `.gitignore`:
```sh
mkdir test
cd test
```
Generate the TLS certificate and the key:
```sh
openssl req -nodes -new -x509 -keyout cert.key -out cert.crt
```
Also, generate TLS session tickets:
```sh
openssl rand 32 > ./tls_key_1
openssl rand 32 > ./tls_key_2
```
2024-10-14 17:44:24 +03:00
### <a href="#run-2" id="run-2" name="run-2">Step 2: prepare the DNSCrypt configuration</a>
2022-08-26 14:18:35 +03:00
Install the [`dnscrypt`][dnsc] tool:
2024-10-14 17:44:24 +03:00
- On macOS, install from Brew:
2022-08-26 14:18:35 +03:00
```sh
brew install ameshkov/tap/dnscrypt
```
2024-10-14 17:44:24 +03:00
- On other unixes, such as Linux, [download][dnscdl] and install the latest release manually.
2022-08-26 14:18:35 +03:00
Then, generate the configuration:
```sh
dnscrypt generate -p testdns -o ./dnscrypt.yml
```
2024-10-14 17:44:24 +03:00
### <a href="#run-3" id="run-3" name="run-3">Step 3: prepare the configuration file</a>
2022-08-26 14:18:35 +03:00
```sh
cd ../
2023-06-11 12:58:40 +03:00
cp -f config.dist.yaml config.yaml
2022-08-26 14:18:35 +03:00
```
2024-10-14 17:44:24 +03:00
### <a href="#run-4" id="run-4" name="run-4">Step 4: prepare the test data</a>
2022-08-26 14:18:35 +03:00
```sh
2024-07-10 19:49:07 +03:00
echo '<html><body>General content ahead</body></html>' > ./test/block_page_general.html
2022-08-26 14:18:35 +03:00
echo '<html><body>Dangerous content ahead</body></html>' > ./test/block_page_sb.html
echo '<html><body>Adult content ahead</body></html>' > ./test/block_page_adult.html
echo '<html><body>Error 404</body></html>' > ./test/error_404.html
echo '<html><body>Error 500</body></html>' > ./test/error_500.html
```
2024-10-14 17:44:24 +03:00
### <a href="#run-5" id="run-5" name="run-5">Step 5: compile AdGuard DNS</a>
2022-08-26 14:18:35 +03:00
```sh
make build
```
2024-10-14 17:44:24 +03:00
### <a href="#run-6" id="run-6" name="run-6">Step 6: prepare cache data and GeoIP</a>
2022-08-26 14:18:35 +03:00
We'll use the test versions of the GeoIP databases here.
```sh
rm -f -r ./test/cache/
mkdir ./test/cache
curl 'https://raw.githubusercontent.com/maxmind/MaxMind-DB/main/test-data/GeoIP2-Country-Test.mmdb' -o ./test/GeoIP2-Country-Test.mmdb
2023-06-11 12:58:40 +03:00
curl 'https://raw.githubusercontent.com/maxmind/MaxMind-DB/main/test-data/GeoIP2-City-Test.mmdb' -o ./test/GeoIP2-City-Test.mmdb
2024-01-04 19:22:32 +03:00
curl 'https://raw.githubusercontent.com/maxmind/MaxMind-DB/main/test-data/GeoIP2-ISP-Test.mmdb' -o ./test/GeoIP2-ISP-Test.mmdb
2022-08-26 14:18:35 +03:00
```
2024-10-14 17:44:24 +03:00
### <a href="#run-7" id="run-7" name="run-7">Step 7: run AdGuard DNS</a>
2022-08-26 14:18:35 +03:00
You'll need to supply the following:
2024-10-14 17:44:24 +03:00
- [`ADULT_BLOCKING_URL`][env-ADULT_BLOCKING_URL]
- [`BILLSTAT_URL`][env-BILLSTAT_URL]
- [`CONSUL_ALLOWLIST_URL`][env-CONSUL_ALLOWLIST_URL]
- [`GENERAL_SAFE_SEARCH_URL`][env-GENERAL_SAFE_SEARCH_URL]
- [`LINKED_IP_TARGET_URL`][env-LINKED_IP_TARGET_URL]
- [`NEW_REG_DOMAINS_URL`][env-NEW_REG_DOMAINS_URL]
- [`PROFILES_URL`][env-PROFILES_URL]
- [`SAFE_BROWSING_URL`][env-SAFE_BROWSING_URL]
- [`YOUTUBE_SAFE_SEARCH_URL`][env-YOUTUBE_SAFE_SEARCH_URL]
2022-08-26 14:18:35 +03:00
See the [external HTTP API documentation][externalhttp].
2024-11-08 16:26:22 +03:00
You may use `go run ./scripts/backend` to start mock GRPC server for `BACKEND_PROFILES_URL`, `BILLSTAT_URL`, `DNSCHECK_REMOTEKV_URL`, and `PROFILES_URL` endpoints.
2024-01-04 19:22:32 +03:00
2024-10-14 17:44:24 +03:00
You may need to change the listen ports in `config.yaml` which are less than 1024 to some other ports. Otherwise, `sudo` or `doas` is required to run `AdGuardDNS`.
2022-08-26 14:18:35 +03:00
Examples below are for the configuration with the following changes:
2024-10-14 17:44:24 +03:00
- Plain DNS: `53``5354`
- DoT: `853``8853`
- DoH: `443``8443`
- DoQ: `853``8853`
2022-08-26 14:18:35 +03:00
2022-11-07 10:21:24 +03:00
You may also need to remove `probe_ipv6` if your network does not support IPv6.
2023-09-06 08:22:07 +03:00
If you're using an OS different from Linux, you also need to make these changes:
2024-10-14 17:44:24 +03:00
- Remove the `interface_listeners` section.
- Remove `bind_interfaces` from the `default_dns` server configuration and replace it with `bind_addresses`.
2023-09-06 08:22:07 +03:00
2022-08-26 14:18:35 +03:00
```sh
env \
2023-09-06 08:22:07 +03:00
ADULT_BLOCKING_URL='https://raw.githubusercontent.com/ameshkov/stuff/master/DNS/adult_blocking.txt' \
2024-01-04 19:22:32 +03:00
BILLSTAT_URL='grpc://localhost:6062' \
2023-09-06 08:22:07 +03:00
BLOCKED_SERVICE_INDEX_URL='https://adguardteam.github.io/HostlistsRegistry/assets/services.json' \
CONSUL_ALLOWLIST_URL='https://raw.githubusercontent.com/ameshkov/stuff/master/DNS/consul_allowlist.json' \
2023-06-11 12:58:40 +03:00
CONFIG_PATH='./config.yaml' \
2023-08-08 18:31:48 +03:00
FILTER_INDEX_URL='https://adguardteam.github.io/HostlistsRegistry/assets/filters.json' \
2022-08-26 14:18:35 +03:00
FILTER_CACHE_PATH='./test/cache' \
2023-09-06 08:22:07 +03:00
NEW_REG_DOMAINS_URL='https://raw.githubusercontent.com/ameshkov/stuff/master/DNS/nrd.txt' \
PROFILES_CACHE_PATH='./test/profilecache.pb' \
2024-01-04 19:22:32 +03:00
PROFILES_URL='grpc://localhost:6062' \
2023-09-06 08:22:07 +03:00
SAFE_BROWSING_URL='https://raw.githubusercontent.com/ameshkov/stuff/master/DNS/safe_browsing.txt' \
2022-08-26 14:18:35 +03:00
GENERAL_SAFE_SEARCH_URL='https://adguardteam.github.io/HostlistsRegistry/assets/engines_safe_search.txt' \
2024-01-04 19:22:32 +03:00
GEOIP_ASN_PATH='./test/GeoIP2-ISP-Test.mmdb' \
2023-06-11 12:58:40 +03:00
GEOIP_COUNTRY_PATH='./test/GeoIP2-City-Test.mmdb' \
2022-08-26 14:18:35 +03:00
QUERYLOG_PATH='./test/cache/querylog.jsonl' \
2023-09-06 08:22:07 +03:00
LINKED_IP_TARGET_URL='https://httpbin.agrd.workers.dev/anything' \
2022-08-26 14:18:35 +03:00
LISTEN_ADDR='127.0.0.1' \
LISTEN_PORT='8081' \
2023-09-06 08:22:07 +03:00
RULESTAT_URL='https://httpbin.agrd.workers.dev/post' \
2022-08-26 14:18:35 +03:00
SENTRY_DSN='https://1:1@localhost/1' \
VERBOSE='1' \
YOUTUBE_SAFE_SEARCH_URL='https://adguardteam.github.io/HostlistsRegistry/assets/youtube_safe_search.txt' \
./AdGuardDNS
```
2024-06-07 14:27:46 +03:00
[env-ADULT_BLOCKING_URL]: environment.md#ADULT_BLOCKING_URL
[env-BILLSTAT_URL]: environment.md#BILLSTAT_URL
[env-CONSUL_ALLOWLIST_URL]: environment.md#CONSUL_ALLOWLIST_URL
[env-GENERAL_SAFE_SEARCH_URL]: environment.md#GENERAL_SAFE_SEARCH_URL
[env-LINKED_IP_TARGET_URL]: environment.md#LINKED_IP_TARGET_URL
[env-NEW_REG_DOMAINS_URL]: environment.md#NEW_REG_DOMAINS_URL
[env-PROFILES_URL]: environment.md#PROFILES_URL
[env-SAFE_BROWSING_URL]: environment.md#SAFE_BROWSING_URL
[env-YOUTUBE_SAFE_SEARCH_URL]: environment.md#YOUTUBE_SAFE_SEARCH_URL
2022-08-26 14:18:35 +03:00
[externalhttp]: externalhttp.md
2024-10-14 17:44:24 +03:00
### <a href="#run-8" id="run-8" name="run-8">Step 8: test your instance</a>
2022-08-26 14:18:35 +03:00
Plain DNS:
```sh
dnslookup example.org 127.0.0.1:5354
```
DoT:
```sh
VERIFY=0 dnslookup example.org tls://127.0.0.1:8853
```
DoH:
```sh
VERIFY=0 dnslookup example.org https://127.0.0.1:8443/dns-query
```
DoQ:
```sh
VERIFY=0 dnslookup example.org quic://127.0.0.1:8853
```
Open `http://127.0.0.1:8081/metrics` to see the server's metrics.
2024-10-14 17:44:24 +03:00
DNSCrypt is a bit trickier. You need to open `dnscrypt.yml` and use values from there to generate an SDNS stamp on <https://dnscrypt.info/stamps>.
2022-08-26 14:18:35 +03:00
2024-10-14 17:44:24 +03:00
> [!NOTE]
> The example below is for a test configuration that won't work for you.
2022-08-26 14:18:35 +03:00
```sh
dnslookup example.org sdns://AQcAAAAAAAAADjEyNy4wLjAuMTo1NDQzIAbKgP3dmXybr1DaKIFgKjsc8zSFX4rgT_hFgymSq6w1FzIuZG5zY3J5cHQtY2VydC50ZXN0ZG5z
```
[dnsc]: https://github.com/ameshkov/dnscrypt
[dnscdl]: https://github.com/ameshkov/dnscrypt/releases
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
## <a href="#testing" id="testing" name="testing">Testing</a>
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
The `go-bench` and `go-test` targets [described earlier](#makefile-targets) should generally be enough, but there are cases where additional testing setup is required. One such case is package `bindtodevice`.
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
### <a href="#testing-bindtodevice" id="testing-bindtodevice" name="testing-bindtodevice">Testing `SO_BINDTODEVICE` features</a>
2023-03-18 17:11:10 +03:00
2024-10-14 17:44:24 +03:00
The `SO_BINDTODEVICE` features require a Linux machine with a particular IP routing set up. In order to test these features on architectures other than Linux, this repository has a Dockerfile and a convenient script to use it, see `scripts/test/bindtodevice.sh`.
2023-03-18 17:11:10 +03:00
A simple example:
2024-10-14 17:44:24 +03:00
- If your Docker is installed in a way that doesn't require `sudo` to use it:
2023-03-18 17:11:10 +03:00
```sh
sh ./scripts/test/bindtodevice.sh
```
2024-10-14 17:44:24 +03:00
- Otherwise:
2023-03-18 17:11:10 +03:00
```sh
env USE_SUDO=1 sh ./scripts/test/bindtodevice.sh
```
2024-10-14 17:44:24 +03:00
This will build the image and open a shell within the container. The container environment is defined by `scripts/test/bindtodevice.docker`, and has all utilities required to build the `AdGuardDNS` binary and test it. The working directory is also shared with the container through the `/test` directory inside it. The container also routes all IP connections to any address in the `172.17.0.0/16` subnet to the `eth0` network interface. So, calling `make go-test` or a similar command from within the container will actually test the `SO_BINDTODEVICE` features:
2023-03-18 17:11:10 +03:00
```sh
go test --cover -v ./internal/bindtodevice/
```
2024-10-14 17:44:24 +03:00
If you want to open an additional terminal (for example to launch `AdGuardDNS` in one and `dig` it in the other), use `docker exec` like this (you may need `sudo` for that):
2023-03-18 17:11:10 +03:00
```sh
docker exec -i -t agdns_bindtodevice_test /bin/sh
```