Skip to content

Add more clear logs when oap-cluster-internal data format is inconcsistent #13059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wu-sheng merged 2 commits into from
Feb 21, 2025
+25 −15
Merged

Add more clear logs when oap-cluster-internal data format is inconcsistent #13059

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
660d834
Add more clear logs when oap-cluster-internal data(metrics/traffic) f…
wu-sheng Feb 21, 2025
9fbf77e
Fix
wu-sheng Feb 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/en/changes/changes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
* Add `Get Alarm Runtime Status` API.
* Add `lock` when query the Alarm metrics window values.
* Add a fail-safe mechanism to prevent traffic metrics inconsistent between in-memory and database server.
* Add more clear logs when oap-cluster-internal data(metrics/traffic) format is inconsistent.

#### UI

Expand Down
39 changes: 24 additions & 15 deletions ...core/src/main/java/org/apache/skywalking/oap/server/core/remote/RemoteServiceHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,24 +117,33 @@ public void onNext(RemoteMessage message) {
String nextWorkerName = message.getNextWorkerName();
RemoteData remoteData = message.getRemoteData();

try {
Copy link
Copy Markdown
Member Author

@wu-sheng wu-sheng Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't change much. I just replaced this try/catch by several others.

RemoteHandleWorker handleWorker = workerInstanceGetter.get(nextWorkerName);
if (handleWorker != null) {
AbstractWorker nextWorker = handleWorker.getWorker();
StreamData streamData = handleWorker.getStreamDataClass().newInstance();
RemoteHandleWorker handleWorker = workerInstanceGetter.get(nextWorkerName);
if (handleWorker != null) {
AbstractWorker nextWorker = handleWorker.getWorker();
StreamData streamData;
try {
streamData = handleWorker.getStreamDataClass().newInstance();
} catch (Throwable t) {
remoteInErrorCounter.inc();
LOGGER.error(t.getMessage(), t);
return;
}
try {
streamData.deserialize(remoteData);
nextWorker.in(streamData);
} else {
remoteInTargetNotFoundCounter.inc();
LOGGER.warn(
"Work name [{}] not found. Check OAL script, make sure they are same in the whole cluster.",
nextWorkerName
);
} catch (Throwable t) {
remoteInErrorCounter.inc();
LOGGER.error("Can't deserialize data {}, this data is discarded.", message, t);
return;
}
} catch (Throwable t) {
remoteInErrorCounter.inc();
LOGGER.error(t.getMessage(), t);
nextWorker.in(streamData);
} else {
remoteInTargetNotFoundCounter.inc();
LOGGER.warn(
"Data is discarded due to worker not found. Check OAL/MAL script, make sure they are aligned in the whole cluster. The data is {}",
message
Copy link
Copy Markdown
Member

@kezhenxu94 kezhenxu94 Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is unhelpful. Message doesn't have a readable toString representation

Copy link
Copy Markdown
Member Author

@wu-sheng wu-sheng Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a proto object. Isn't it output the data? I read this from decompiling codes.

  @Override
  public final String toString() {
    return TextFormat.printer().printToString(this);
  }

Copy link
Copy Markdown
Member Author

@wu-sheng wu-sheng Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For safety, I run the demo codes locally

public static void main(String[] args) {
        RemoteMessage obj = RemoteMessage.newBuilder()
                                             .setNextWorkerName("dbc").build();
        System.out.println("Default toString(): " + obj.toString());
        System.out.println("TextFormat: " + TextFormat.printer().printToString(obj));

        try {
            System.out.println("JsonFormat: " + JsonFormat.printer().print(obj));
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
Default toString(): nextWorkerName: "dbc"

TextFormat: nextWorkerName: "dbc"

JsonFormat: {
  "nextWorkerName": "dbc"
}

);
}

} finally {
timer.finish();
}
Expand Down
Loading