Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: Add support for hierarchical and auto-generated keys for log filtering. #62

Merged
merged 25 commits into from
Feb 28, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
bcd3b69
Add auto-gen key support
LinZhihao-723 Feb 18, 2025
872b69d
Update...
LinZhihao-723 Feb 18, 2025
dd923c9
Apply code review comments
LinZhihao-723 Feb 19, 2025
a6b70c5
Update...
LinZhihao-723 Feb 20, 2025
bbf7340
Don't use .data()
LinZhihao-723 Feb 20, 2025
165a890
Implement hierarchical log-level and timestamp key search.
LinZhihao-723 Feb 20, 2025
4618e0c
Merge oss main
LinZhihao-723 Feb 21, 2025
2a02ff8
Latest clp
LinZhihao-723 Feb 21, 2025
483079a
Switching to my desktop
LinZhihao-723 Feb 22, 2025
4dc0ab8
Refactoring
LinZhihao-723 Feb 22, 2025
9d5395e
Done
LinZhihao-723 Feb 22, 2025
62d83bd
Apply code review comments
LinZhihao-723 Feb 25, 2025
4555da7
Fix unused header
LinZhihao-723 Feb 25, 2025
d5de853
Apply suggestions from code review
LinZhihao-723 Feb 25, 2025
2093972
Finalize js layer type.
LinZhihao-723 Feb 26, 2025
65837c9
Apply code review comments.
LinZhihao-723 Feb 26, 2025
b8c9cf6
Apply code reivew comments
LinZhihao-723 Feb 27, 2025
f12cf04
Remove header declaration of ReaderOption
LinZhihao-723 Feb 27, 2025
f46af70
Fixing...
LinZhihao-723 Feb 27, 2025
c364896
Apply suggestions from code review
LinZhihao-723 Feb 28, 2025
03bd5b9
Update src/clp_ffi_js/ir/StructuredIrStreamReader.cpp
LinZhihao-723 Feb 28, 2025
c0120ec
Linting
LinZhihao-723 Feb 28, 2025
973966d
Update src/clp_ffi_js/ir/StructuredIrStreamReader.cpp
LinZhihao-723 Feb 28, 2025
15211e2
Add log statements.
LinZhihao-723 Feb 28, 2025
eff990b
Merge branch 'key-search' of https://github.com/LinZhihao-723/clp-ffi…
LinZhihao-723 Feb 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/clp_ffi_js/ir/StreamReader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,8 @@ EMSCRIPTEN_BINDINGS(ClpStreamReader) {
emscripten::register_type<clp_ffi_js::ir::DataArrayTsType>("Uint8Array");
emscripten::register_type<clp_ffi_js::ir::LogLevelFilterTsType>("number[] | null");
emscripten::register_type<clp_ffi_js::ir::ReaderOptions>(
"{logLevelKey: string, timestampKey: string} | null"
"{logLevelKey: {isAutoGenerated: boolean; parts: string[];} | null;"
" timestampKey: {isAutoGenerated: boolean; parts: string[];} | null;}"
);

// JS types used as outputs
Expand Down
43 changes: 40 additions & 3 deletions src/clp_ffi_js/ir/StructuredIrStreamReader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
#include <cstddef>
#include <format>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <system_error>
Expand All @@ -11,6 +12,7 @@
#include <clp/Array.hpp>
#include <clp/ErrorCode.hpp>
#include <clp/ffi/ir_stream/Deserializer.hpp>
#include <clp/ffi/SchemaTree.hpp>
#include <clp/ir/types.hpp>
#include <clp/TraceableException.hpp>
#include <emscripten/bind.h>
Expand All @@ -27,6 +29,8 @@
namespace clp_ffi_js::ir {
namespace {
constexpr std::string_view cEmptyJsonStr{"{}"};
constexpr std::string_view cFilterOptionIsAutoGeneratedKey{"isAutoGenerated"};
constexpr std::string_view cFilterOptionPartsKey{"parts"};
constexpr std::string_view cReaderOptionsLogLevelKey{"logLevelKey"};
constexpr std::string_view cReaderOptionsTimestampKey{"timestampKey"};
constexpr std::string_view cMergedKvPairsAutoGeneratedKey{"auto-generated"};
Expand All @@ -39,12 +43,39 @@ constexpr std::string_view cMergedKvPairsUserGeneratedKey{"user-generated"};
* @param json
* @return Serialized JSON.
*/
auto dump_json_with_replace(nlohmann::json const& json) -> std::string;
[[nodiscard]] auto dump_json_with_replace(nlohmann::json const& json) -> std::string;

/**
* @param filter_option The JavaScript object representing a filter option.
* @param leaf_node_type The type of the leaf node for the schema tree full branch.
* @return A schema tree full branch constructed based on the provided filter option and leaf node
* type.
* @return A schema tree full branch constructed based on the provided reader option.
* @return std::nullopt if `option` is `null`.
*/
[[nodiscard]] auto get_schema_tree_full_branch_from_filter_option(
emscripten::val const& filter_option,
clp::ffi::SchemaTree::Node::Type leaf_node_type
) -> std::optional<StructuredIrUnitHandler::SchemaTreeFullBranch>;

auto dump_json_with_replace(nlohmann::json const& json) -> std::string {
return json.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace);
}

auto get_schema_tree_full_branch_from_filter_option(
emscripten::val const& filter_option,
clp::ffi::SchemaTree::Node::Type leaf_node_type
) -> std::optional<StructuredIrUnitHandler::SchemaTreeFullBranch> {
if (filter_option.isNull() || filter_option.isUndefined()) {
return std::nullopt;
}
return StructuredIrUnitHandler::SchemaTreeFullBranch{
filter_option[cFilterOptionIsAutoGeneratedKey.data()].as<bool>(),
emscripten::vecFromJSArray<std::string>(filter_option[cFilterOptionPartsKey.data()]),
leaf_node_type
};
}

EMSCRIPTEN_BINDINGS(ClpStructuredIrStreamReader) {
emscripten::constant(
"MERGED_KV_PAIRS_AUTO_GENERATED_KEY",
Expand All @@ -67,8 +98,14 @@ auto StructuredIrStreamReader::create(
*zstd_decompressor,
StructuredIrUnitHandler{
deserialized_log_events,
reader_options[cReaderOptionsLogLevelKey.data()].as<std::string>(),
reader_options[cReaderOptionsTimestampKey.data()].as<std::string>()
get_schema_tree_full_branch_from_filter_option(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ease of debugging -

can we print a log in deserialize_stream() after all IR units are decoded: if the m_optional_log_level_full_branch has value but m_optional_log_level_node_id is not populated, warn the user that the node cannot be found. Similarly for the timestamp we should also warn the user.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

reader_options[cReaderOptionsLogLevelKey.data()],
clp::ffi::SchemaTree::Node::Type::Str
),
get_schema_tree_full_branch_from_filter_option(
reader_options[cReaderOptionsTimestampKey.data()],
clp::ffi::SchemaTree::Node::Type::Int
)
}
)};
if (result.has_error()) {
Expand Down
Loading
Loading